Invisible Learning Quotes

“The Google team at Project Aristotle concluded that there were five main traits of a successful team. Psychological safety was the most important, and their research suggested that it underpinned the other four[xxvi]. The five traits were as follows: (1)   Psychological safety: team members feel safe to take risks and be vulnerable in front of each other. (2)   Dependability: team members get things done on time, and meet Google’s high bar for excellence. (3)   Structure and clarity: team members have clear roles, plans, and goals. (4)   Meaning: work is personally important to team members. (5)   Impact: team members think their work matters and creates change.”
― David Franklin, Invisible Learning: The magic behind Dan Levy's legendary Harvard statistics course

“In Dan’s case, the interest in policymaking is so central to the students’ presence at Harvard that it would be foolish to run a statistics course that did not acknowledge it. He goes as far as to build this into the purpose of the course: as we heard earlier, his purpose is “not just to maximise learning about statistics, but also to maximise learning of the skills that will be useful to have out there in the world.”
― David Franklin, Invisible Learning: The magic behind Dan Levy's legendary Harvard statistics course

“The inclusive atmosphere of the class was the key driver of allowing the class to be eager to learn. The focus really shifted away from grades and points, and towards understanding concepts and learning.” The key link is between inclusivity and the shift of focus away from test scores. When students are given the responsibility to support each other, their own score falls down the list of priorities.”
― David Franklin, Invisible Learning: The magic behind Dan Levy's legendary Harvard statistics course

“Exams also give us a concrete example of Dan’s attempts to promote understanding over memorization. He encourages students to bring with them a ‘cheat sheet’, which can include anything they want. He does not want students being tested on whether they can remember the formula for the standard error of a distribution, and does not believe that it will be helpful for them to do so. Allowing cheat sheets means that students can bring more than just formulas with them: if they were struggling with a topic, they could bring all sorts of pre-cooked explanations to help them with the exam questions. In turn, this acts as an extra incentive to Dan to set questions that rely on a deeper understanding.”
― David Franklin, Invisible Learning: The magic behind Dan Levy's legendary Harvard statistics course

“Dan’s use of two-stage exams kills two birds with one stone. Firstly, he maximises learning by ensuring that the exam itself is a learning experience. Second, in doing so, he makes clear that the grade is less important than the learning. Two-stage exams have not yet ‘taken off’ around the world, and grades remain the key outcome of most exams for most students. Dan, though, has taken advantage of his position in a graduate university environment to push the idea forward.”
― David Franklin, Invisible Learning: The magic behind Dan Levy's legendary Harvard statistics course

“With these issues in mind, Dan has a unique way of tying exams into the learning process. After attending a workshop run by the Nobel Prize winner Carl Wieman, he was converted to the idea of two-stage exams. The first stage of the exam is the same as any other. The second stage brings the students together in groups of four, and gives them all a subset of the questions they just answered in the first stage. The students are encouraged to discuss their answers in their groups, and they all submit a second version. Groups who manage to improve upon the students’ individual scores get a small boost in the final outcome.”
― David Franklin, Invisible Learning: The magic behind Dan Levy's legendary Harvard statistics course

“Acknowledging that students have competing purposes is the first step to managing them.”
― David Franklin, Invisible Learning: The magic behind Dan Levy's legendary Harvard statistics course

“I remember he said something like ‘this should be completely obvious to you’, and to me that was crushing… we had a conversation after the mid-term exam, in which the class had averaged 20 out of 90. It was 25 years ago, and I still remember what he said: ‘Frustration is necessary for learning. This idea that you can enjoy learning is a very American idea’.” Dan pauses, the memory of that conversation etched on his face. “I felt so offended by that claim. My professor felt that to learn, you had to push yourself. One of my dreams was to go back to Venezuela[7] and start a university, and I vowed that I would write in the walls of the university that frustration was not necessary for learning.”
― David Franklin, Invisible Learning: The magic behind Dan Levy's legendary Harvard statistics course

“Maximising enjoyment All students want to enjoy their classes. This simple truth means that teachers who can make learning fun will, other things equal, be more successful. If the only way students can enjoy themselves is to ignore the class and play on their phones, then student enjoyment is a headwind that slows the boat’s progress. By making learning fun, teachers can adjust the sails to take advantage of the wind and speed the boat up. The progress it makes may not be directly towards our destination, but we can use the sails nonetheless to pick up speed and get closer to it than we would otherwise have done.”
― David Franklin, Invisible Learning: The magic behind Dan Levy's legendary Harvard statistics course

“Recognising competing purposes Although few would have much trouble identifying ‘maximising learning’ as the purpose of the class, there may be several competing purposes in play, all of which are invisible. For students, maximising enjoyment, minimising effort, maximising test scores, maximising learning about something else, and making friends are all likely to be found across the classroom. For the teacher, we can add to that list maximising evaluation scores, which may be closer linked to student satisfaction than to student learning.”
― David Franklin, Invisible Learning: The magic behind Dan Levy's legendary Harvard statistics course

“The failure here is an example of a common flaw in our approach to adaptive challenges: we try to solve them with technical solutions. Terrified of the unknown, we seek to replace problems we have no idea how to address with those we know how to solve. This is a form of ‘work avoidance’, a term coined by Heifetz[x]. If you have ever procrastinated by spending your whole afternoon making a brightly-coloured work plan, and then congratulated yourself on being productive, then you will be familiar with the concept. By focusing on what they know they can measure – the extent to which a lesson went as planned – some professors may avoid the harder work of trying to gauge the learning of their students.”
― David Franklin, Invisible Learning: The magic behind Dan Levy's legendary Harvard statistics course

“On the whiteboard in Dan’s office, he has written in bold black marker the phrase ‘TEACHING ≠ LEARNING’. He believes that most lecturers pay too much attention to what they are doing in a class. Planning is important, and professors are right to spend time in advance thinking about how the class will be structured. One downside, though, is that professors will often evaluate themselves against what they said they would do in the plan. If they covered the material without tripping up, and gave the right explanations at the right time, they will be happy about how the class went.”
― David Franklin, Invisible Learning: The magic behind Dan Levy's legendary Harvard statistics course

“Einstein is supposed to have remarked that “education is what remains when one has forgotten everything they learned at school”.”
― David Franklin, Invisible Learning: The magic behind Dan Levy's legendary Harvard statistics course

“The first key idea is that when you have a challenge for which measuring progress is hard, the ability to adapt is just as important as the ability to plan.”
― David Franklin, Invisible Learning: The magic behind Dan Levy's legendary Harvard statistics course

“But few of us really stop to ask ourselves what new information might change our minds, especially on questions of politics and identity. We fall into the trap of confirmation bias, in which rather than updating our prior beliefs, we mould new information into a form that will confirm them.”
― David Franklin, Invisible Learning: The magic behind Dan Levy's legendary Harvard statistics course

“The great economist John Maynard Keynes is supposed to have said “when the facts change, I change my mind. What do you do, sir?”
― David Franklin, Invisible Learning: The magic behind Dan Levy's legendary Harvard statistics course

“There is also something powerful in the concept of an ‘airport idea’. On the surface, it links with Dan’s plea for what students should do if they meet him in an airport in five years’ time. But there are hidden suggestions too: the airport carries an association of being high-level, and fits in with the course’s aim of empowering the students with the tools they need to effect change. Once the course is over, they will disperse to airports across the world and put these ideas into practice in their home countries.”
― David Franklin, Invisible Learning: The magic behind Dan Levy's legendary Harvard statistics course

“The phrases ‘certainty is an illusion’ and ‘think probabilistically about the world’, both introduced in the first class, are the first two examples of high-level concepts which Dan calls airport ideas.”
― David Franklin, Invisible Learning: The magic behind Dan Levy's legendary Harvard statistics course

“Dan’s goal, taken from his mentor Richard Zeckhauser, is to get students to think probabilistically about the world. This means engaging with the challenge of understanding and accepting what 29% means in the context of a Trump victory, and fighting the brain’s natural inclination to see things in binary terms.”
― David Franklin, Invisible Learning: The magic behind Dan Levy's legendary Harvard statistics course

“I’d like anyone who was born on March 22nd to stand up.” Two students stand up. They link eyes from across the room, and smile. The room laughs. “Now, anyone born on June 14th.” Another two stand up. The room laughs again. “Now, anyone born on August 2nd.” Two more: the class is laughing at each pair, and re-examining their original takes on how low the probability was likely to be. They are learning through experience about human fallibility with estimating probabilities, and at the same time learning something about their classmates. “How about December 21st?” This time, three students stand up. This gets a big laugh for two reasons: firstly, it is unexpected, and secondly, it helps to ram home the idea that in a class that size (there are about 3,000 pairs in a class of 80), there are likely to be “coincidences” everywhere.”
― David Franklin, Invisible Learning: The magic behind Dan Levy's legendary Harvard statistics course

“With 80 people, the size of this classroom, the probability that there will be no shared birthdays is so tiny as to be virtually impossible.”
― David Franklin, Invisible Learning: The magic behind Dan Levy's legendary Harvard statistics course

“It turns out that you only need 23 people for it to be more likely to find a shared birthday than not.”
― David Franklin, Invisible Learning: The magic behind Dan Levy's legendary Harvard statistics course

“But everyone who guessed in this way went low, rather than high. It suggests that there is some form of systemic bias in play.”
― David Franklin, Invisible Learning: The magic behind Dan Levy's legendary Harvard statistics course

“Now – those who got it right. On the count of three, I’d like you to sit down if you’d seen the question before. 1, 2, 3!”  Everyone who was standing sits down. This gets a huge laugh. The students who got the answer wrong suddenly feel much better about their mistake: in a Harvard class, not a single person got it right unless they had seen it before.”
― David Franklin, Invisible Learning: The magic behind Dan Levy's legendary Harvard statistics course

“He asks the 80 students to respond based on their ‘gut feeling’. Again, students are given five options, ranging from ‘less than 1%’ to ‘above 40%’. About half of them believe the true answer is less than 5%, of which plenty go for the ‘less than 1%’ option. Only 1 in 6 get it right, picking the highest option: it turns out that the true figure is 41%. He invites those people – 13 in total – to stand up.”
― David Franklin, Invisible Learning: The magic behind Dan Levy's legendary Harvard statistics course

“What is the probability that at least two people in a group of 20 people will have the same birthday?”
― David Franklin, Invisible Learning: The magic behind Dan Levy's legendary Harvard statistics course

“In advance of the class, he asked students to fill in a quick survey in which he asked them to describe their own experiences with one of these three uses of statistics. He tells the students that he has read their answers, as well as their student profiles, and asks for volunteers to talk about what they wrote. Several hands go up, and he picks Juliana, a student from Brazil, interested in education. She starts to talk about a program she helped to run in Brazil which assessed whether improvements in teacher training had a causal effect on test scores. As she is talking, her words appear on the big screen, in big quotation marks, along with her photo. The class laughs, and as she looks up she realises she has become a celebrity.”
― David Franklin, Invisible Learning: The magic behind Dan Levy's legendary Harvard statistics course

“After pausing, the student continues: “There’s also an element of believing something because you want it to happen.” This betrays a second bias in which we believe what we want to be true. This is a form of confirmation bias, which exacerbates the distorting effect of availability bias. We seek out information that confirms our existing beliefs or desires, and ignore information that refutes them.”
― David Franklin, Invisible Learning: The magic behind Dan Levy's legendary Harvard statistics course

“I guess… it was just a shocking event. We’re not used to someone like him being President.” Here, the student reveals an availability bias in which we expect future outcomes to look like what has gone before.”
― David Franklin, Invisible Learning: The magic behind Dan Levy's legendary Harvard statistics course

“It sounds like a lot of you are not too surprised if this 30% chance materialises,” Dan replies. “You take precautions against it. The rain, if it comes, doesn’t shock you, and you have your umbrella. So why was everyone so shocked about Trump?”
― David Franklin, Invisible Learning: The magic behind Dan Levy's legendary Harvard statistics course