The LDA Podcast: An Exploration of Evidence-Informed Approaches to Learning and Development

AI and L&D Insights-- Large Language Models: Hype vs. Reality

In this episode, Markus and Chris Pedder, Chief Data Scientist at OBRIZUM, discuss the rise and utility of large language models (LLMs) in L&D, including their limitations, as well as ethical concerns.

The ”Once Upon A Time” Episode

In this episode, we reboot the Truth in Learning podcast. After catching up with Will Thalheimer, and introducing Clark Quinn, we give a brief update on the direction of the show. We then explore the question, "What should every CEO know... care about... concerning learning and development?" In fact, Will will share some of his recent insights into the topic based on his new and upcoming book.Folks can find Will at https://www.worklearning.com/.Learn about his upcoming book here: https://www.ceosguide.net/Then, Kat Koppett, renowned expert on applied improvisational theater techniques and storytelling, joins the conversation with Matt and Clark. Together, they explore what is meant by storytelling in the context of learning, and learning and development. They discuss some of the academic and research disciplines that are foundational to applied storytelling, referencing psychologist Jerome Bruner, cultural theorist Mieke Bal, applied researcher Roger Shank, and others. You can find Kat at: https://www.koppett.com/.Finally, the team brings back the best and the worst-- where hosts and guests share their best and worst observations in the industry.

The ”Leadership Development is a Waste of Time and Money” Episode

In this episode (recorded on September 29, 2020— the day of the first US Presidential debate) we are joined by our friend, Gary DePaul, PhD. We explore:LeadershipLeadership development The current state of leadership development in L&DWhether we should even bother to invest in it.And a ton more.We start with a deep dive debating the definition of leadership and its Rorschach-like characteristics. Then, we bring up that old chestnut as to the differences between management and leadership. Gary proposes a unique slant on this conundrum.Matt links the discourse of leadership in business more concretely to history and politics.The discussion navigates through semantics, leadership semiotics,  and the effect definitions have on practical issues in leadership discourse. We argue the roles, responsibilities, and practices of leadership. And, of course, we debate the how, the why, and the what of leadership development. Gary referenced Mary Parker Follette. You can learn more about this American social worker and management consultant here.Matt references James MacGregor Burns. You can learn more about this historian here.Matt references Peter Northouse and his book, LEADERSHIP: THEORY AND PRACTICE. Will keeps bringing up the practical side of the issue (an excellent point in the discussion)… what do leaders do? Who leads and who follows? What are the roles of followers? And, can we develop toward those functions? We discuss several tropes in the field, such as, “Everyone can lead,” “Everyone should lead,” and “Everyone can be taught to lead.”The guys relate the issues of reliability and validity as they pertain to leader training.Matt references the book, LEADERSHIP BS, by Stanford professor, Jeffrey Pfeffer, sharing several findings that stipulate leadership development is hugely invested in, but either poorly measured or measured as ineffective. You can find the book here.Will shares a literature review, arguing against Matt’s premise. Leadership Training Design, Delivery, and Implementation: A Meta-Analysis by Christina N. Lacerenza, Denise L. Reyes, and Shannon L. Marlow, Dana L. Joseph, and Eduardo Salas. You can find it and purchase it here.Matt tells  of Ronald Reagan’s storytelling as a leadership technique. The reference for that is THE INVISIBLE BRIDGE: THE RISE AND FALL OF NIXON AND THE RISE OF REAGAN by Rick Perlstein.Matt’s book referenced is THE LEADERSHIP STORY: A NEW MODEL FOR LEADERSHIP (ePub version). Or, the Kindle Version here.We refer to several studies on the perception of women in leadership roles. A summary of those studies with embedded links is in this New York Times article.Of course, Will brings up the issues around measurement and evaluation.Finally, we end the episode briefly digging into what is ethical leadership, as well as courageous leadership.We have also put together a list of references and resources. CLICK HERE. The list is by no means exhaustive and complete. It is a set Matt uses in his program, The Leadership Development, Development Workshop.  Due to time constraints in the episode, we skipped The Best and the Worst this round.Our guest, Gary, is an author, speaker, and leadership consultant with Gary DePaul Consulting. With more than 20 years of practitioner and academic experiences in performance improvement, Gary helps elevate the performance of individuals, teams, and organizations. His areas of practice include leadership development, organizational evaluation, speaking events, workshop facilitation, and executive coaching. He can be found at: https://www.garyadepaul.comGary has written several books worth reading.WHAT THE HECK IS LEADERSHIP AND WHY SHOULD I CARE? NINE PRACTICES OF 21st CENTURY LEADERSHIP: A GUIDE FOR INSPIRING CREATIVITY, INNOVATION, AND ENGAGEMENTTHE MOST EFFECTIVE AND RESPONSIBLE CLINICAL TRAINING TECHNIQUES IN MEDICINEGary is also the moderator, along with our friend, Guy Wallace, of HPT Treasures, an blogging site with tons of HPT practitioners providing monthly articles. Both Will and Matt are contributors, as well. You can find it here. 

The ”Smiley Sheet” Episode

Will and Matt are back after a six month hiatus. For their return, they catch up the Truth in Learning community on what they have been doing--primarily focusing on the six-month Learning Development Conference 2020, or LDC. LDC was an inaugural global event with over 300 participants. An all-digital conference with over 60 live, synchronous programs and 20+ asynchronous workshops. In this episode our itinerant podcasters spend a few minutes talking about the passing of Roger Kaufman, a giant in the field of performance improvement. We were so lucky to have him on the show. The "I'm a Coach... Really... I Am" Episode where he and Will had a one-on-one conversation. In Will's eulogy, he mentions the recent creation of The Kaufman Center, created this year, as the custodian of Roger's legacy, a think tank dedicated to promoting Roger's Mega thinking in social value-adding projects and ideas.Then, we spend the majority of the episode talking about the issues surrounding Smiley Sheets, or “Learner Surveys” as Will now calls them. We explore the:The inherent issues with Smiley SheetsThe actual benefits of Smiley Sheets (According to Will, there are a few!)How to mitigate the negative effects of themHow to use Smiley Sheets to manage expectations among the participants, the facilitators, and the stakeholdersHow to construct a decent Smiley Sheet QuestionExamples of bad questionsExamples of how to translate a bad question into a good oneAnd, more…Will has recently been updating his book, Performance-Focused Smile Sheets: A Radical Rethinking of a Dangerous Art Form. DON'T GO BUY THIS VERSION OF THE BOOK. Will's second edition is a significant upgrade and he wants you to buy that one when it comes out.We also (BIG NEWS) announce our next venture—The Learning Development Accelerator (LDA), the new L&D professional organization that will have a plethora of services, programs, conferences, and resources for members. Website to come.And, of course, we end the episode with The Best and the Worst. Matt goes after the application of The Collective Intelligence. NOTE: Matt wants to be clear, his comments on voting were irreverent and mostly sarcastic. Will wants to be clear, he sometimes knows what Matt is thinking.

The ”Research-Informed” Episode

PLEASE NOTE: This episode was the last recorded episode before COVID-19 really exploded in the US. We were quite aware of, and sensitive to, the dangers world-wide, but weren’t aware yet of how the virus manifested, nor how badly the US, particularly the Northeast, would be hit. The magnitude of the crisis wasn’t fully felt at this point. __________________________________For this single-topic episode, we are thrilled to have the great Mirjam Neelen join us to discuss her new book, Evidence-Informed Learning Design: Creating Training to Improve Performance, co-authored with the also great Paul Kirschner. Throughout the episode we talk with Mirjam about many learning issues she and Paul raise in the book, including the following:The difference between evidence-based and evidence-informedWhy learning professionals sometimes default to poorly supported modelsThree-Star Learning ExperiencesWhether learning should be funThe difference between learning problems and performance problemsUsing authentic learning tasksDirected instruction versus discovery learningThe Top Five Effective Learning StrategiesAnd, the Flop Five Learning Strategies (well just a taste of these)Nuance and complexity in learningAnd, so much more!After the conversation, Will and Matt debrief and discuss many of the ramifications and applications from or chat with Mirjam. We go deeper into many of the issues raised, especially on the topics of fun, nuance, and complexity. We probably shouldn’t have, but we use President Trump’s reliance on his hunches with regard to the brewing (at the time of recording) COVID-19 crisis rather than a reliance on the scientific institutions he should rely on throughout all of this. We talk about Mirjam and Paul’s discussion in the book about the improper reliance on neuroscience. And, more…Finally, we bring Mirjam back for the Best and the Worst. Mirjam references Michelle Oker’s reading group on research. Click to go to the group.Will mentioned Truth in Learning friend, Emma Weber. Her company is LEVER: Transfer of Learning. Mirjam is a Learning Experience Design Lead with over 10 years of industry experience, working at companies such as Houghton Mifflin Harcourt, Google, the Learnovate Centre, and now Accenture.  In her current role, she leads the learning experience design processes across Accenture’s various business entities globally.  Mirjam is a proud advocate for evidence-informed approaches to learning design. She’s had a collaborative blog with Paul over 4 years on https://3starlearningexperiences.wordpress.com/.You can find Mirjam and Paul’s book wherever great books are sold. Click here to get it on Amazon.

Will’s Big Announcement Teaser

Will breaks into podcast studio, and shares some important announcements while Matt works tirelessly on the website for Will and Matt's big announcement. Will talks about the LearningDevelopmentReflections.com website response to the Coronavirus pandemic, recent episodes, cancellations of Matt and Will's live shows, and more... Will does a teaser for Matt and Will's BIG ANNOUNCEMENT--the thing that will change the L&D field forever. Matt's going to be really pissed when he learns about Will playing with the podcast while Matt works to exhaustion on the new website.

The ”Women in Learning” Episode

For our latest episode, we return to the single topic format. The entire episode focuses on WOMEN IN LEARNING. In other words, we explore what it’s like for women in the learning and development field. We are joined by three experts on this subject. Specifically, we explore:Issues women face delivering their work in L&DStruggles women face moving up in the field— overall and day-to-dayThe value proposition of inclusion and a diversityDynamics between companies, vendors, and employeesThe impact financials have on gender diversityThe role men have positively played in supporting and mentoring women in L&DCompanies that don’t focus on inclusive environments and what may happen to themThe Ernst & Young Learning ScandalSuggestions for what we can all do to make things betterAnd, so much more.Kate Graham is the head of content with analyst Fosway Group, She shapes the output from their research agenda, staying across technology trends and industry activity to drive practical, insightful analysis and resources for our corporate and vendor clients - and the industry as a whole. Along with our second guest, Sharon Claffey Kaliouby, Kate founded the WOMEN IN LEARNING initiative.  Fosway Group can be found at: https://www.fosway.com. And, Kate’s personal blog is: https://kategraham23.wordpress.com. Sharon Claffey Kaliouby is the VP, North America at Learning Pool. Learning Pool is a full-service e-learning provider, offering a range of courses, tools and content creation to over 750 organizations and 2 million learners in 21 countries. Lori-Niles Hofmann should already be familiar with Truth in Learning listeners, having joined us for our recent discussion about LMSs. Lori is a senior learning strategist with over 20 years of L&D experience across many industries, including international banking, management consulting, and marketing. Lori’s website is: https://lorinileshofmann.squarespace.com. You can find WOMEN IN LEARNING at #womeninlearning on LinkedIn. And, of course, we end with The Best and the Worst.Will mentioned our good friend, Julie Dirksen. You can find Julie at https://usablelearning.com. 

The ”Cheap Trick” Episode

For our latest episode, we’ve got mail! All good, and some very useful and constructive feedback. We share. And, somehow, the greatest working rock band in the world, Cheap Trick gets a plug. (Eye roll from Will) In the episode, Matt mentions his experience meeting the band and here’s the photo he shared with Will. Matt’s in the middle, next to his cousin, Mark.Matt with Cheap Trick before their Albany, NY concert on February 7, 2020.For Segment One, Will shares a discussion he had on LinkedIn with an individual who posted bad information as valid. This person made a case that eLearning is better than in-person training. The elearning-is-better argument wasn’t really what Will had issue with, but rather the evidence this person used to make the case. The individual is on the learning technology side and had a stake in what he/she espoused. The argument used a whole bunch of falsehoods. After Will called the piece out, the person went out in search of evidence to confirm the original thesis… confirmation bias. The point of this segment is not to litigate this individual’s view. No. Rather, it is fodder for Will and Matt to discuss what gets put out there as research. In other words, how does actual research get misconstrued, misunderstood, and then put out there as truth. We also mention the documentary about the Flat Earth Society called BEHIND THE CURVE. https://www.netflix.com/title/81015076 We also mention a LinkedIn discussion Matt started on ending the traditional Icebreaker. https://www.linkedin.com/posts/matthew-richter-0738b84_truthinlearning-thiagi-trainthetrainer-activity-6636243679377965057-tHgN Then, in Segment Two, there are two scams our friend Cara North identifies during her discussion with Will. The first involves teachers (and others) who are transitioning into our field and look for help from those who say they can support and facilitate them through the transition. Unqualified consultants and “experts” offer their services with the promise of making jobs happen. But, the outcomes don’t support the promise. Cara also highlights how universities often offer Masters’ programs in instructional design, but don’t then provide the skills and competencies to help people get jobs. Students often do not get what they need to be able to do the work. So, for her, these two “scams” are linked and related problems we have industry-wide. Cara offers advice and tips before jumping into a program or getting help from the so-called experts. You can find Cara’s website here: https://caranorth.com. Then, Matt and Will debrief and discuss some of the semantics related to the topic.Finally, in Segment Three, we discuss how one approaches learning. Why do we separate eLearning designers from in-person designers? Why do we put the onus on the platform first and not on the learning outcomes. We argue that logistics should be a compromise we make after we initially understand the desired outcomes and objectives. We look at Will’s eLearning research review from 2017, still a definitive comparison of elearning and classroom training. https://www.worklearning.com/2017/08/10/major-research-review-on-elearning-effectiveness/ Making the connections to our first segment, we explore that eLearning can be better than in-person training. Or, in-person can be better. In other words… as always, it depends!And, of course… we end up talking about what we mean by eLearning. Is it a webinar? A Live, Virtual, Online Training (LVOT), an asynchronous program? Etc. You can learn more about Will’s Presentation Science Program here: https://www.presentationscience.net Will mentions Pathwright for writing online courses. Here’s the link: https://www.pathwright.com And, we end the session with the Best and the Worst where we call out (in a good way) Jane Bozarth, Director of Research at the eLearning Guild. You’ll have to listen to the episode to learn why!

The ”What the LMS” Episode

In this episode, Will and Matt are joined by two renowned experts, Lori Niles-Hofmann and Steve Foreman. This extended, single segment show is all about the LMS. What is it? How did we get here? Where are we going? What are we trying to accomplish and how do we meet our LMS needs? Lori and Steve share their expertise on how to evaluate an LMS and how to evaluate LMS vendors. They provide tons and tons of tips and advice on everything about these monsters. Did you know there were tons of different types of LMSs? We didn’t. Do you even need an LMS? Everyone has them? Do you? Why do you?Lori is a senior learning strategist with over 20 years of L&D experience across many industries, including international banking, management consulting, and marketing. Her specialization is large-scale digital learning transformations and acts as a trusted adviser to CLOs around the world. Lori has developed data-based methodologies and frameworks that empower L&D teams to move from business support function to strategic business driver. She is currently based in Toronto, Canada with one foot in Europe. You can find Lori at https://lorinileshofmann.squarespace.com. Steve is a management consultant and software designer/developer in the eLearning space. He subscribes to a broader definition of eLearning that includes knowledge management, expertise location and management, communities of practice, and performance support solutions. You can find Steve at http://infomediadesigns.com. In our Best and Worst, Lori mentions during her best, Matt Ash from The Media Zoo sending her a game in the New York Times, called CAN YOU DEFEAT THE PRIVACY CHICKEN. Check it out here.

The ”Happy New Years’ Resolutions” Episode

DISCLAIMER: We apologize for the recording sound on Matthew’s side of things… First, he was sick. Second, his microphone broke and the sound quality was reduced. We will fix this before the next episode. Now… onto the show…In our first segment, we explore what is a learning objective. How are they used, should they be used, how are they constructed, how are they consumed, and what are alternatives to them. We mention Bob Mager and his book, Preparing Instructional Objectives: A Critical Tool in the Development of Effective Instruction. Will also refers to his mentor and doctoral advisor, Ernie Rothkopf. One of the research articles Will references is: Rothkopf, E. Z., & Kaplan, R. (1972). An exploration of the effect of density and specificity of instructional objectives on learning from text. Journal of Educational Psychology, 6, 295-302. From Will, here is the classic piece he wrote about rethinking instructional objectives. Click here. And, Will’s classic video on objectives that Matthew sends to everyone. Click here. Our second segment delves into the overall purpose of training. Is it just to ultimately yield business results? Shareholder value? Or, aligning more with the sustainability values of today, to provide outcomes aligned also with employee well-being, the enhancement of the community, and other broader benefits to the many. Click here for the LinkedIn Post that inspired this segment. We also brought up Self-Determination Theory again. The motivation model originally conceptualized by Edward Deci and Richard Ryan from the University of Rochester. While both of them have done quite a bit of writing about SDT within the context of work (and Rich is now offering a program on Coursera beginning February 10), the two researchers Matthew mentions are Marylène Gagné and Jacques Forest. The specific articles and book referenced are listed first along with other articles Jacques has shared with Matthew. That list is at the bottom of the episode notes. If you are interested generally in SDT, go here and/or take Rich’s Coursera program.In our third segment, we are joined by Kassy LaBorie. Together we all discuss how to bring interactivity to the virtual classroom. We discuss the differences and similarities between classroom and virtual classroom delivery. Kassy shares the misconceptions many have about virtual classrooms. And, so much more about virtual delivery. Kassy mentions the wonderful video, A CONFERENCE CALL IN REAL LIFE, embedded above. Or, click here to see it on YouTube. Kassy can be found at https://kassyconsulting.com. She is the co-author, along with Thomas Stone, of Interact and Engage!: 50+ Activities for Virtual Training, Meetings, and Webinars. Tom, also Kassy’s husband, wrote the very cool baseball book, Now Taking the Field: Baseball's All-Time Dream Teams for All 30 Franchises. Finally, we take a break from our Best and Worst closing and have several of our listeners share their personal New Years’ Resolutions. Will and Matthew share theirs, as well. Contributors are: the very same Kassy LaBorie you listened to in segment three, Phil Reynolds, Cara North, Clark Quinn, Ellen Burns-Johnson, Roger Kaufman, Allison Samon, Ulrich Boser, Julie Dirksen, Alex Salas, Mirjam Neelen, and Dave Barton.SELF-DETERMINATION THEORY REFERENCESForest, J., Gilbert, M.-H., Beaulieu, G., Le Brock, P., & Gagné, M. (2014). Translating research results in economic terms: An application of economic utility analysis using SDT-based interventions. In M. Gagné (Ed.), The Oxford Handbook of Work Engagement, Motivation, and Self-Determination Theory (pp. 335-346). Oxford:  Oxford University Press.Trepanier, S.-G., Forest, J., Fernet, C., & Austin, S. (2015). On the psychological and motivational processes linking job characteristics to employee functioning: Insights from Self-Determination Theory. Work and Stress, 29, 286-305. The Oxford Handbook of Work Engagement, Motivation, and Self-Determination Theory Güntert, S. T. (2015). The impact of work design, autonomy support, and strategy on employee outcomes: A differentiated perspective on self-determination at work. Motivation and Emotion, 39, 74-87.Trepanier, S.-G., Fernetet, C., & Austin, S. (2015). A longitudinal investigation of workplace bullying, basic need satisfaction, and employee functioning. Journal of Occupational Health Psychology, 20, 105-116.Van den Broeck, A., Schreurs, B., De Witte, H., Vansteenkiste, M., Germeys, F., & Schaufeli, W. (2011). Understanding workaholics’ motivations: A self-determination perspective. Applied Psychology: An International Review, 60 600-621.Ng, J. Y. Y., Ntoumanis, N., Thogersen-Ntoumani, C., Deci, E. L., Ryan, R. M., Duda, J. L., & Williams, G. C. (2012). Self-determination theory applied to health contexts: A meta-analysis. Perspectives on PsychologicalScience, 7, 325-340.