How Might We Learn?

Talks about learning technology often center on technology. So instead I'd like to begin by asking what do you want learning to be like for yourself? If you could just snap your fingers and have the ideal perfect learning environment, what would that be like? One way to start getting at this question is to ask what were the most rewarding high growth periods of your life? I've noticed two patterns when I ask people this question. First, people will tell me about a period in their life where they learned a ton, but where learning wasn't the point. Instead, they were totally immersed in some project with a great deal of personal meaning, like a scrappy startup or or a research project, or an artistic urge, or just a fiery curiosity. And a lot of learning happened just along the way. They learned whatever they needed. They dove in. They got their hands dirty. Secondly, in these stories, learning really worked. People emerged feeling transformed, newly capable, filled with insights that remained with them years later. These stories are so vivid, in part because learning rarely feels this way. People are often telling me, somewhat wistfully, about an experience that occurred years or decades earlier. Learning rarely feels so subordinated to an authentic pursuit. Often if we try to just dive in, we dive into a brick wall, or we find ourselves uneasily cargo culting others without any real understanding of what's actually going on. So why can't we just dive in all the time instead? It often feels like we have to put our aims on hold while we go do some homework and learn properly. Or worse, learning so often just doesn't really work. We take the class, we read the book, and then when we try to put that knowledge into practice, we find that it's fragile, it doesn't transfer well. Worse, we'll often find that we've forgotten half of it by the time we try to use it. Why does learning so often fail to actually work? These questions connect to an age old conflict among educators and learning scientists between implicit learning, also known as discovery learning, inquiry learning, situated learning, and guided learning, which is often represented by cognitive scientists of the kind we find here at ucsd. Advocates of implicit learning methods argue that we should prioritize discovery, motivation, authentic involvement, and being situated in a community of practice in the opposing camp. Cognitive psychologists argue that you really do need to pay attention to the architecture of cognition, to long term memory, to procedural fluency, and to scaffold appropriately for cognitive load. In my view, each of these points of view has something a lot. Each of these points of view contains a lot of truth, and they also ignore each other to their detriment. Implicit learning aptly recognizes meaning and emotion, but ignores the often decisive constraints of cognition. What we need to actually make learning work Guided learning advocates are focused on making learning work, and they sometimes succeed, but usually by sacrificing that purposeful sense of immersion that we love about those rewarding high growth periods. One obvious approach is to try compromise. Project based learning is a good representation of that by creating a scaffolded series of projects. The suggestion is that we can get some of the benefits of implicit learning authenticity, motivation, transferability, while also getting the instructional control and the cognitive awareness that might be typical of traditional courses. The trouble is that usually we get the worst of both worlds. I remember when I I was in college I was interested in 3D game programming, so I signed up for a project based course on computer graphics. The trouble was that those projects weren't my projects. So a few weeks in I ended up implementing array marching for more efficient bump mapping. And unfortunately I was really just implementing some math that I was handed because this course was trying to take project based learning seriously. So there weren't long textbook readings, weren't long problem sets, but I didn't understand the math. What I ended up with was a project that I didn't care about implementing something I didn't understand. Instead, I suggest we should take both views seriously. Find a way to synthesize the two. You really do want to make doing the thing the primary activity, but the realities of cognitive psychology mean that in many cases you really do need explicit guidance, scaffolding, practice, intention, and memory support. Learning by immersion works naturalistically when the material has a low enough complexity relative to your prior knowledge that you can successfully process it on the fly, and when natural participation routinely reinforces everything important, giving you fluency. When those conditions aren't satisfied, which is most of the time, you will need some support. You want to just dive in and you want learning to actually work. To make that happen, we need to infuse your authentic projects with guided support where necessary, inspired by the best ideas from cognitive science. And if there's something that requires more focused explicit learning, then you want those experiences to be utterly in service to your actual aims. Now, I've been thinking about this synthesis for many years and honestly, I've mostly been pretty stuck. Recently though I've been thinking a lot about AI, which I know gets an eye roll. Certainly pretty much every mention of AI in education gets a eye roll from me. But I confess the possibility of AI has helped me finally get what feels like some traction on this particular problem. So I'd like to share some of those early concepts. Today, we'll explore this possible synthesis through a story in six parts. Meet studied computer science in university, and they're now working as a software engineer at a big tech company. But Sam is bored at their day job. Not everything is boring, though. Every time Sam sees a tweet announcing a new result in brain computer interfaces, they're absolutely captivated. These projects seem so much more interesting than what they're doing by day. Sam pulls up the papers looking for some way to contribute, but they hit a brick wall with so many unfamiliar topics all at once. What if Sam could ask for help finding some meaningful way to start participating? With Sam's permission, our AI, and let's assume it's a local AI, can build up a huge amount of context about their background from old documents on Sam's hard drive. Our AI knows all about their university coursework. It can see their current skills through work projects. It knows something about Sam's interests through their browsing history. So it suggests a few ideas. And Sam is excited about the idea of reproducing the paper's data analysis. That seems to play to their strengths. They notice that the authors use a custom Python package to do their analysis, but that code was never published, and that seems intriguing. Sam has built open source tools before. Maybe they could contribute here by building an open source version of this signal processing pipeline. So Sam dives in. They found an open access data set, and they've taken the first steps to start working with it. Tools like Copilot help Sam get started, but to follow some of these signal processing steps, what Sam really needs here is something like copilot, but with awareness of the paper in addition to the code, and with context about what Sam's actually trying to do. This AI system isn't trapped in its own chat box or in the sidebar of one application. It can see what's going on across multiple applications, and it can propose actions across multiple applications. Sam can click that button to view a change set with the potential implementation here, that is, and then they can continue the conversation smoothly, switching into the context of the code editor. Like what is this axis equals one parameter. The explanation depends on context from the code editor, from the paper being implemented, and also the documentation that came with the data set. Sam's working with the AI underlines assumptions made based on specific information and turns those things into links. So Sam can click on that in this data set link and our AI opens the readme to the relevant line. All this is to support our central aim, which is that SAM can immerse themselves as much as possible in what they're actually trying to do, but get the support they need to understand what they're doing. And that support doesn't have to just mean text. SAM next needs to implement a downsampling stage and this time guidance includes synthesized dynamic media so that SAM can understand what downsampling does through scaffolded immersion. SAM doesn't need to read an abstract explanation and try to imagine what that would do to different signals. Instead, as they try different sampling rates, real time feedback can help them internalize the effect on different signals. By playing with the dynamic media, SAM notices that some of the peaks are lost when the signal is down sampled. These dynamic media aren't trapped in the chat box. They're using the same input data and libraries that SAM is using in their notebook. So at any time SAM can just view source and to tinker with this figure or to use some of its code in their own notebook. Now SAM presses on. But as they dig into band pass filters, the high level explanations they can get from these short chat interactions really just don't feel like enough. What is a frequency domain? What is a Nyquist rate? SAM can copy and paste some AI generated code all day, but they don't understand what's going on at all. A chat interface is just not a great medium for long form conceptual explanation. It's time here for something deeper. Now our AI knows sam's background and aims here, so it suggests an appropriate undergraduate text with a practical focus. And more importantly, the AI reassures Sam that that they don't necessarily need to read this entire thousand page book right now. It focuses on sam's goal here and suggests a range of accessible paths that SAM can choose according to how deeply they would like to understand this material. The AI has made a personal map in the book's table of contents, so that if Sam, for instance, just wants to understand what these filters are doing and why, there's a 25 page path for that. But if they want to know the mathematical background, how these filters work, there's a deeper path. And if they want to actually be able to implement them themselves, there's an even deeper path. And choose a journey here. When Sam digs into the book, they'll find notes from the AI at the start of each section and scattered throughout, which ground the material in sam's context. Sam's project sam's purpose this section will help you understand how to think about signals in terms of frequency spectra, that's what low pass filters manipulate. Sam's spending some time away from their project in a more traditionally instructional setting. But that doesn't mean the experience has to lose its connection to their authentic purpose. Incidentally, I've heard some technologists suggest that we should just use AI to synthesize the whole book. We get per person bespoke textbooks. But I think that there's actually a huge amount of value in having shared canonical artifacts in any given field. There are key texts that we can all point to, and they form a common ground for the culture. I think we can preserve that by layering personalized context on top as a lens like this in my ideal future. Of course, our canonical shared artifacts are dynamic media, not digital representations of dead trees. But until all of our canonical works are rewritten as a transitional measure, we can at least wave our hands and imagine that our AI could synthesize dynamic media versions of figures like this one. Now, as Sam reads through the book, they can continue to engage with the text by asking questions as before, and our AI's responses will continue to be grounded in their project. As SAM highlights the text or makes comments about details which seem particularly important or surprising. Those annotations won't end up trapped inside the PDF. Instead, they will feed into future discussions and practice, as we'll see later. In addition to Sam asking questions of the AI, the AI can insert questions for SAM to consider again grounded in their project to promote deeper processing of the material. And just as our AI guided Sam to the right sections of this thousand page book, it can point out which exercises might be most valuable, considering both Sam's background and their aims. Better, it can connect the exercises to Sam's aims, so that aspirationally between those problems feels continuous with Sam's authentic practice. Even if the exercises do still feel somewhat decontextualized, SAM can at least feel more confident that the work is going to help them do what they want to do. So Sam ends the day with some rewarding progress on their project and a newfound understanding of quite a few topics. But this isn't yet robust knowledge. SAM has very little fluency. If they try to use this material seriously, they'll probably feel like they're standing on shaky ground. And more prosaically, they will probably forget much of what they just learned. So I'd like to focus on memory for a bit here. It's worth asking, why do we sometimes remember conceptual material and sometimes not? Often we take a class or read a book, or even just look something up and Find that a short time later. We have retained almost nothing, but sometimes things seem to stick. Why is that? There are some easier cases. If you're learning something new in a domain that you know well, each new fact connects to lots of prior knowledge, and that creates more cues for recall and more opportunities for reinforcement. Likewise, if you're in some setting where you need that knowledge every single day, you. You will find that your memory becomes reliable pretty quickly. Conceptual material, like what Sam just learned, doesn't usually get reinforced every day like that. But sometimes the world conspires to give those memories the reinforcement it needs. Sometimes you read about a topic, and then later in that evening, that topic comes up in conversation with the collaborator. So you have to retrieve what you learned. You. And that retrieval reinforces the memory. Then, maybe two days later, it comes up again. You need to recall that knowledge for a project. Each time you reinforce the memory this way, you forget it more slowly. Now, perhaps a week can go by and you're still likely to remember. Then maybe a few weeks, then a few months, and so on. With a surprisingly small number of retrievals, if they're placed close enough to avoid forgetting, you can retain that knowledge for months or years. By contrast, sometimes when you learn something, it doesn't come up again until, say, the next week. Then you try to retrieve the knowledge, but maybe it's already been forgotten, so you have to look it up. Looking it up doesn't reinforce your memory very much. And then if it doesn't come up again for a while longer, you may still not remember the next time, so you have to look it up again, and so on. The key insight here is that it's possible to arrange the top timeline for yourself. And of course, courses sometimes do, when each problem set consistently interleaves knowledge from the prior problem sets. But immersive learning, and for that matter, most learning, usually doesn't arrange this properly. So you usually forget a lot. What if this kind of reinforcement were woven into the grain of the learning medium? Collaborator Michael Nielsen and I created a quantum computing primer, Quantum country, to explore this idea. It's available for free online. If you head to Quantum country, you'll see what looks at first like a normal book. After a few minutes of reading, the text is interrupted with a small set of review questions. They're designed to take just a few seconds each. You think the answer to yourself, then mark whether or not you were able to answer correctly. So far, these look like simple flashcards. But as we've discussed, even if you can answer these questions. Now, that doesn't mean you'll be able to in a few weeks or even in a few days. So notice these markings at the bottom of each question. These represent intervals. So you practice the questions while you're reading the text. Then one week later, you'll get an email that says, hey, you have probably started to forget some of what you've learned. Do you want to take five minutes to quickly review that material again? Each time you answer successfully, the interval increases to a few weeks and a few months, and so on. If you begin to forget, then the intervals tighten up to provide more reinforcement. Now, you may have seen systems like this before. Language learners and medical students in particular often use tools called spaced repetition memory systems to remember vocabulary and basic facts. But the same cognitive mechanisms should work for more complex conceptual knowledge as well. There are 112 of these questions scattered throughout the first chapter of the book. On that basis, Quantum country is a kind of new medium, a mnemonic medium integrating a space repetition memory system with an explanatory text aspirationally to make it easier for people to absorb complex material reliably. We now have millions of practice data points so we can start to see how well it's working. This plot shows the amount of time spent practicing on the X axis versus the reader's demonstrated retention on the Y axis. That is how long a reader was able to go without practicing and still answer at least 90% of questions correctly. These five dots represent the median user's first five repetitions just for the first chapter. Notice that the Y axis is logarithmic. So this straight line plot we're seeing here represents a very nice exponential growth. Each extra repetition, which is a constant extra time input, yields increasing output, that is retention. And so, in exchange for about an hour and a half of total practice, the median reader was able to correctly answer over 100 detailed questions about the first chapter after more than two months without any practice. Now, the first chapter takes Most readers about 4 hours to read the first time. So this plot implies that an extra overhead of less than 50% in time commitment can yield months or years of detailed retention. It's also interesting to explore the counterfactual how much would people have forgotten if they didn't have this extra reinforcement? So, as an experiment, we removed nine questions from the first chapter for some readers and then covertly reinserted the questions one month later into their practice sessions. This graph shows what happened. These nine points represent those nine questions. The Y axis shows the percentage of readers who are able to Answer that question correctly after one month with no support at all, you can see that some questions are harder than others. All the way over here on the left. One month later, the majority of readers missed the hardest three questions, and about 30% missed the middle three. About 15% missed the easiest three. We can compare these to another group of users who got practice while reading the essay, like we saw in the video a moment ago. And for any questions they missed, they got a bonus round of practice the next day. Then these questions disappeared for a month, at which point we tested. These readers performed noticeably better, though a big chunk of them are still missing several of these questions. Now, here's one last group, like the previous one, except they got just one extra round of practice a week after reading the book. Then we tested them again at the one month mark, and that's what you're seeing here. Each question takes six seconds on average to answer. So this is less than a minute of extra practice in total for these nine questions. But now for all of these questions, at least 90% of readers were able to answer correctly. Of course, some readers have a much easier time than others. So the left plot here focuses on the bottom quartile of users, I.e. the readers who missed the most questions while they were first reading the essay. Notice that I've had to lengthen the Y axis downwards here, because we can see that without any practice at all, most of these people in the bottom Quartile forgot 2/3 of these held out questions. If they only had in essay practice, roughly half of them were left for getting roughly half of the questions. And here, with just one extra round of practice, that extra, slightly less than a minute of extra practice, even this bottom quartile of readers performs quite well, almost as well as the overall population. So this is the power of practice, efficient practice at least. And this mechanism is useful for more than just quantum computing. In my personal practice, I have accumulated thousands and thousands of questions. I write questions about scientific papers, about conversations, about lectures, about memorable meals. I will definitely be writing a bunch about meetings I've had here today. All of this makes my daily life more rewarding, because I know that if I invest my intention in something, I will internalize it indefinitely. Central to this is the idea of a daily ritual, a vessel for practice, like meditation or exercise. I spend about 10 minutes a day using this memory system. And because these exponential schedules are very efficient, those 10 minutes are enough to maintain my memory for thousands of these questions and to allow me to add about 40 new questions every day. But There are some problems, so I want to mention a few of these problems. One is pattern matching. Once a question comes up a few times, I may recognize the text of the question without really thinking about its meaning. This creates the unpleasant feeling of parroting, but more importantly, I suspect it often leaves my memory brittle. I'll remember the answer, but only when cued exactly as I've practiced it. I wish the questions had more variability. Likewise, the questions are necessarily somewhat abstract. When I face a real problem in this domain, I won't always recognize what knowledge I should use or how to adapt it to that medium. A cognitive scientist would say maybe that I need to acquire schemas. Now, unless I intervene, these questions stay the same over years. They're maintaining my memory, but ideally they would push for further processing, increasing depth over time. And finally returning to this talk's thesis. Memory systems are too often disconnected from my authentic practice. What I'm actually interested in. Say I'm studying a topic in signal processing for a creative project. Unless I'm very careful, the questions that I get from that probably won't feel very connected to my project. They will probably feel like generic textbook questions about signal processing. Let's return to SAM now and see if we can apply some of these ideas about practice and memory. So SAM did the work to study that signal processing material. They want to make sure it actually sticks. How might that work? Let's say they can install a home screen widget which ambiently exposes them to practice prompts drawn from highlights, questions asked, and any other activity that the AI can access. SAM can flip through these questions while waiting in line or on the bus. And notice that this isn't a generic textbook signal processing question. It's actually grounded in the details of sam's brain computer interface project so that at least aspirationally, practice feels somewhat more continuous with authentic doing. These synthesized prompts can vary each time they're asked so that SAM gets practice accessing the same idea from different angles. The prompts get deeper and more complex over time as SAM gets more confident with the material. Notice also that this question isn't so abstract. It's really about applying what SAM has learned in a bite sized form factor they can do anywhere. Now the widget can also include kind of more open ended discussion questions. Why do we think Metzger et al downsampled their signals to 200 hertz? Maybe it's for performance. Here SAM gets some elaborative feedback, an extra detail to consider in their answer. Now, when questions are synthesized like this, it's important that Sam can steer them with feedback, assuring that future questions are synthesized accordingly. Because again, what we're trying to do here is to make all of this not be homework, but to actually support what Sam's really trying to do. So far we've been looking at bite sized questions Sam can answer while they're out and about, but if they make time for a longer, dedicated session, we can suggest meteor tasks like this one. And what's more, we can move that work out of fake practice land and into Sam's real context here, a jupyter notebook. Notice that the task is still framed in terms of Sam's specific aims rather than some generic signal processing pipeline. Now, Sam got into this project not as a learning exercise, but as a way to start legitimately participating, to start working with BCIs while playing to existing strengths. So just as our AI can help Sam find a tractable way into this space, it can also facilitate connections to communities of practice here suggesting a local neurotech meetup so let's say Sam goes to this meetup, meets a local scientist, and sets up a coffee date with permission. Sam records the meeting, knowing the notes will probably be helpful later. And of course, Sam ends up surprised and intrigued quite a lot in this conversation. Our AI can notice these moments of surprise and help Sam metabolize them. Here that insight turns into a reflective practice prompt. Four big design principles are threaded through Sam's story. I'd like to review them now and for each point out the ways that AI has helped me think about them. First, we bring guided learning to authentic contexts rather than thinking about it as a separate activity. We're able to make that happen by imagining an AI which can perceive and act across applications on Sam's computer. And as the audio transcript at the end of the story here alluded to, that action can potentially extend to activities outside of the computer as well. The AI can give appropriate guidance in part because with permission and executing locally, it can learn from every piece of text that has ever crossed sam's screen, every action they have ever taken on the computer. It can synthesize scaffolded dynamic media so that SAM can learn by doing, but with guidance. And then when explicit learning activities are necessary, we suffuse them with authentic context. The AI in our story grounds all of the reading and practice Sam's doing in their actual aims. It helps Sam match the learning activities to their depth of interest, and it draws on important moments that happen while Sam is doing like insights from that coffee meeting at the end or questions asked while implementing parts of the project, and it brings those moments into study activities. Besides connecting these two domains, we can also strengthen each of them. So our AI suggests tractable ways for SAM to just dive in to a new interest, and it helps SAM build connections with a community of practice. On the other side, when we are spending time in explicit learning activities, let's make sure that they actually work. So our AI creates a dynamic vessel for ongoing reinforcement. It varies what's in that vessel over time so that knowledge transfers more effectively to real situations. And it doesn't just maintain memory, it increases depth of understanding over time. Now I'd like to give two cheers for chatbot tutors. Most discussion of AI and education at the moment revolves around the framing of chatbot tutors, and I think this framing correctly identifies something really wonderful about language models, which is that they are so good at answering long tail questions. If the user can articulate the question clearly enough, and if the user is trying to perform a routine task, chatbot tutors can often diagnose problems and find good ways to get the user unstuck. That's great. But when I look at others visions of chatbot tutors through the broader framing that we've been discussing, they're clearly missing a lot of what I want. I think these visions often fail to take seriously just how much a real tutor can really do. In large part, I think that's because the authors of these visions are usually thinking about educating something they want to do to others, rather than learning something they want for themselves. Now, a sad truth about the world is that postdocs and graduate students are incredibly underpaid, so it is actually surprisingly affordable to get an expert tutor for a technical topic I care about. But if I hire a real tutor as an adult to learn about signal processing, I will tell them about my interest in brain computer interfaces, and I will expect them to ground every conversation in that purpose. My goal here is not to learn signal processing. It is to participate in the creation of brain computer interfaces. Chatbot tutors are not interested in what I'm trying to do. There's a set of things they think I should know or should be able to do, and they view me as defective until I say the right things. If I hire a real tutor, I might ask them to sit beside me as I try to actually do something involving the material. They can see everything I'm doing, see what I'm pointing at, and if it's appropriate, I can scoot over and they can drive for a minute. By comparison the typical conception of a chatbot tutor lives in a windowless box, and it can only see whatever is provided on scraps of paper passed under the door. It can have no effect on the outside world. My goal here is to dive in, to immerse myself, to start doing the thing. But these chatbot tutors can't join me where the action is. So interactions with them create distance and pull me away from immersion. If I hire a real tutor, we'll build a relationship with every session. They'll learn more about me, my interests, my strengths, my confusions. Chatbot tutors, on the other hand, as typically conceived, are transactional amnesiac. Now we can fix that as context windows get longer. But that relationship is also important to my emotional connection. If I view conversation with my tutor as a kind of peripheral participation in the community, I'm hoping to enter an interaction between a novice in the discipline and an expert in the discipline. Then tutoring just becomes part of doing the thing. But if my interaction with the tutor is transactional, that will tend to make my tutoring sessions feel like learning time separate from doing the thing. Finally, people talk about how Aristotle was a tutor for Alexander the Great. But you know, what's most valuable about having Aristotle as your tutor is not that he can diagnose misconceptions, but rather that he's modeling the practices and values of an earnest, intellectually engaged adult, is demonstrating how and why he thinks about problems, his taste in the discipline. The high growth periods that we love transform the way that we see the world. They reshape our identity. In my demo earlier, I showed a chatbot, but it didn't really work. Like most chatbot tutors I describe, it focused all its actions on the user's interest rather than bringing its own agenda. It wasn't trapped in a little text box. It could see and take action in the context of authentic use. It can communicate through dynamic media. It had a deep memory, drawing on everything I'd ever seen and written. So in some ways, the system I've shown is more like a real tutor. But in my ideal world, I don't want a tutor. I want to legitimately participate in some new discipline and to learn what I need as much as possible from interaction with real practitioners. So I view the role of the augmented learning system as helping me act on my creative interests, ideally by letting me just dive in and start doing as much as possible. That will often mean scaffolding connections to and interactions with communities of practice. One theme for this Design at Large series is the ethics of AI and its likely enormous social impacts. So. So let me say I am tremendously worried about those impacts. In the general case. I am worried about despots locking in their powers. I'm worried about lowering the bar to bioweapons. I'm worried about economic chaos. I would not feel comfortable ethically with researching more powerful frontier models myself. But within the narrower domain of learning that we've been discussing, my main moral concern is that we will end up trapped in a sad, narrow future. A condescending authoritarian frame dominates the narrative. In the future of learning. I'll caricature it to make the point. With AI, we can finally take all of these defective kids that don't know the stuff they're supposed to know and get them to know it. You know, that's personalized learning. The AI lets us precisely identify where the kids are wrong or where they're ignorant and fix them. Then we can fill their heads to the brim with what's good for them. By contrast, the famous bicycle for the mind metaphor has no agenda other than the one that you bring. It just lets you reach a wider range of destinations than you could on foot. And it makes the journey more fun, maybe particularly if you're biking along with some friends. The bicycle asks, where do you want to go? Of course, that question assumes your destination is well known and charted on some map. But those most rewarding high growth experiences are often centered on a creative project. You're trying to get somewhere no one's ever gone before to reach the frontier and then start charting links into the unknown. Learning in service of creation. It's a dynamic, context laden kind of learning. It's about more than just efficiency and correctness, more than just faster gears on a bike. And that's the kind of learning that I feel an almost moral imperative to help create. Thank you. So I think we maybe have time for questions. We have some time for questions. Love the talk. Thank you. I really like a lot of what you said. One thing that I'm curious about is you mentioned that you add questions to your own. Like. Like for your space repetition routine, you add questions every day. And you said you add 40 questions a day. So I guess what I'm wondering is in your example you took one topic. Yeah, but many people are often learning multiple things and very often we don't know that we want to learn something until like we've built some momentum. Right. So my question is probably how do you decide what questions you want to add? And how do you plan for your future self in some way. Right, right. First I should clarify. So I don't in fact add 40 questions a day. That is the carrying capacity of my practice time. 10 minutes will support 40 a day added in practice. It ends up like some kind of Poisson distribution or something. But how do I do this? How do I plan appropriately? You can't know what's going to be important to you in advance. Sometimes you can, but in general you can't. So the system needs to be resilient to that. The way that that works right now is very coarse. Like you can delete questions that you add if you decide that you don't like them. I view this as an interface problem of sorts. I think ideally you don't need to plan for what's going to be important. You can just do stuff and stuff will get reinforced and you can steer more of this, less of that. And I think that's an interesting challenge in interface design to create systems that behave more like that and less like these kind of discrete destructive actions. Destroy this question, Andy. I love that vision over here for that. Especially liked how you talked about how the AI could reinforce things you've learned and challenge you. It could integrate you with communities. Love all those ideas. What I didn't get, and maybe it's because I walked in two minutes late, was that an actual demo that you've created or is that just a vision for how you want it to be in the future? Yeah, I'm sorry for not making that clear. If it wasn't, I thought it looked sufficiently fake. No, this is concept art. Thank you for asking. There is, I suppose, a grand tradition in our discipline of smoke and mirror concept art. I have been using this talk as an excuse to figure out what I think I want to do with respect to AI and learning. And so these drawings are part of that process. I was curious, I wanted to follow up on the question about sometimes I don't know what I want. And I think if you look at some of the more Montessori or Papert esque learning experiences when, you know, when you have like, you know, if you just said school is going to be whatever you're interested in forever. Yeah, it has lots of benefits. A challenge is that I. There are topics that I might benefit from. And I loved your paternalistic caricature as being. That's what we don't want. There is a. Like the psychologist Dan Gilbert defines the happiness challenge as the delta between our current self and what our future self would have wanted us to do. And I can like A nice thing about a good tutor, your talk, for example, is I'm learning things that will be valuable to me in the future that at 4 o' clock today I wouldn't have thought to ask for. Right? Yeah, yeah. I think this is totally right. I think the way that I think about this is a kind of unbundling. Like normally when we think about schooling, we ask schooling to do two jobs. One, to decide what is it that I should know and then two, to cause me to know it. I think there are all kinds of cultural institutions that help us figure out what is it that we want to know. In the example that I showed, Sam figures out that they may be interested in brain computer interfaces through Twitter. If you are in San Diego, you can decide to attend this talk and ideally it doesn't feel like schooling. I hope it doesn't feel like schooling and you can decide, oh, maybe I'm interested in studying some of the cognitive science that Andy mentioned. So I think there's a variety of cultural institutions and venues and channels we can use to help deliver that first part and then possibly use separate tools for the other part. Good afternoon. Should we be planning for the day where we won't have to work due to artificial intelligence, automation, computers and robotics? Your thoughts? Thank you. I'm so grateful that you asked that question because I spent a month preparing for this talk, writing another version of this talk that I didn't give that was called what's Worth Learning in the Age of Strong AI. And it didn't end up aligning, so I didn't end up getting to present any of it. So I'll give you a piece of that. Say that you are a composer and you have some very vague idea about a new cello concerto that you would like to write. I claim that you can't just ask the AI to write it for you because you don't know what it is that you want. In fact, you discover what the cello concerto is supposed to be and through the process of composing it. Likewise, when we talk about software, an interesting thing about software is that it really, it feels so messy and unpleasant so much of the time. We want a particular piece of software, we ask some people to go start making it for us and then they come back 3x over time, 3x over budget, 1/3x quality. Why is software so hard? Why can't we just tell the system what we want and get the thing out the other side? And it turns out like we actually do have that kind of technology. It's called formal modeling. We've had it for a long time. And I think the reason why it's not more widely used is that that's not how we think about designing software. Namely, we figure out what software we want in the process of making it. It's continuously negotiated, it's contingent. So I think there are many activities in human life that have that characteristic where we can't actually appropriately specify what it is that we want to the model. We can't externalize it and make it legible. We have to participate in the creation of the thing. Then we can ask what do we need to know in order to participate in that creation? What are the dynamics of that participation? How much injection of involvement is necessary to steer appropriately? And I have some thoughts on that, but I should probably move on to the next question. So yes, I hope that's somewhat helpful. Hi. So with my question, I was thinking about how with the AI, what you have is you like say that AI advancement will lead to some dystopian feature where we find people the knowledge and then we force them to know it. But who's to say that that is still already happening to some extent with curriculum right now, without AI, that isn't necessarily a concern as much as other things, I think. But beyond that, I think it's kind of like. Because that problem can happen like when we don't look at the root problem. It can happen in the most immersive learning, like in ethnic studies class. You can still end up restricting someone to that specific topic instead of connecting identity as much as you would in say a math class or something like that. So what if you could implement that like sort of, I guess to give another example of how in the other hand you can do something really well even in a math class, for example, if you look at the higher concepts or if someone explains it really well, or in like say neuroscience and those sort of fields, there might be content that is above the class that is interesting, that would make someone more interested and able to learn the class themselves, especially if it connects to their own prior knowledge. So what would you think if like you had, I guess, how do you think you would implement this into like a learning system to take advantage of it, like say canvas? Do you think that you could do it in a way that you could customize in part assignments to synthesize the students interests and the teacher's intent to teach people necessary content and teach people the content that they want to know in a way that they might be doing more but ultimately less because it's easier for them if they know the context and can connect to it. Cool. I think I followed so to your first question. Yeah, I think you're right. That part of the talk was part of a complaint about schooling in general. And AI doesn't necessarily make that worse, though it may continue trends. I don't like to your second point or your second question. I have deliberately avoided the question of schooling in this talk and I did that for a reason. I think it makes everything very complicated. So in terms of integrating into canvas, I think it's a very difficult position to start. I have done a bunch of collaborations these past few years with professors in higher ed teaching large classes and what I experience again and again is just an enormous fraction of the student body that is just fundamentally not engaged with the class. And I don't think any amount of UI chicanery or AI involvement is going to change that. And I think I basically don't want to put myself into that problem solving situation. In some sense that's actually why I left Khan Academy. Hey Andy, my name is Taylor. I'm a software engineer@repl.it so I would say in some sense my salary depends on some of the ideas in your talk, which is to say that AI can be an effective tool for teaching difficult concepts like computer programming. So my question to you is, have you seen any inklings of this being true? The ideas in your talk of the delta between something like a GPT being a teacher and what you're outlining, have you seen any of those ideas actually start to play out? Or do you think this is purely speculative with what we have today? Yeah, I think we're seeing a lot of hints at what I described already. So lots of people already use GPT to just dive into stuff. It's missing that universal IO, it's missing the billion token context window, and yet it's still already able to deliver some value, which is great. I think missing also that the bridge to textbooks. Again and again I've been talking to people, not that textbooks are the answer here. I'm using textbook as synecdoche for some kind of like deeper, more focused learning experience. I talk to a lot of people who like get a really good start with GPT because they can get the couple sentence answers and that lets them make a certain amount of progress and that's really motivating. You can kick up some momentum and momentum is very powerful, but they hit a wall. And I had a section that I cut talking about programming. I think it's really interesting that in Programming. A lot of people can manage to self teach programming. In part, I think that's because of things like syntax for instance, is reinforced every time you sit down to program and obscure facets of syntax which are not reinforced regularly, like maybe your macro library or whatever people do tend to forget. But again and again you meet people who are like, I don't understand how pointers work though. Like that asterisk can see, it's like it's a bridge too far. So there's a certain amount of conceptual understanding that seems acquirable with, I don't know, like the frontier of self learning tools that are widely available. And then like as soon as you run into like difficult conceptual territory, it's just a cliff. So I think we're seeing glimmers of progress in this vein and perhaps this talk will inspire people to try some more stuff. Hello. Hi. So my question is, to what extent do you think learning, the desire to learn comes from the innate pleasure of learning or specific to whatever the material the person learning is? And to what extent do you think the desire to learn comes from some sort of use of the material the person learns? And do you think this distinguish or making these distinctions important to the design of systems for learning? Yeah, probably. This is a good question and it's something that I don't emphasize perhaps quite clearly enough in the example story that I gave. Everything is motivated by this Persona, Sam wanting to do a project. It's very concrete, it's out in the world. And that doesn't mean that curiosity based learning is any less legitimate or that I think any of the claims don't apply. It's just that what authentic practice and legitimate participation looks like is different. So in mathematics, for instance, which may be very abstract, there may not be like a project I'm trying to do as I learn about algebraic topology or something like that. What legitimate practice looks like is engaging with problems and questions in algebraic topology that I find authentically interesting. And as soon as you make that move, rather than thinking about, well, I need to learn algebraic topology in order to do this like 4D renderer that I was working on for a game project. If you think about it from the perspective of authentic curiosity, then I think much of what I'm saying applies. However, there are issues with rendering legible the nature of the curiosity and the interest to the AI. And like a lot of what's happening is internal and that may make a lot of what I'm describing somewhat more difficult. The nature of relation to community of practices changes somewhat Although not completely so. Partial answer, I suppose. Is there someone with the microphone? Oh, hi, Jonathan. Hi, Andy. This is great. I just. I love the deepness of the reflective practice. Your demonstrating for us here, I think, is it Aristotle that said, give me a child till the age of seven and I'll show you the man? And when you bring up the ethical issues, this is what scares me the most, is for an AI tutor. Have you thought about adversarial tutoring? Given that we're going to have, if we do, if we hypothesize AI tutors are out there and that we have the schooling industrial complex producing them, how do we counteract them? I love that. It's like manufacturing the mythical. And often in the teaching community, maligned Robin Williams character, the teacher from the film whose name I'm forgetting, I haven't thought about that. It's a lovely provocation. I don't. I feel uncomfortable with like an activist framing where it's like, I need to get out there because I know what's best for the people or for the kids. So, like, they're doing it wrong. I'm going to make a thing that's going to like, put fluoride in the drinking water so that they end up in the right spot. I'm uncomfortable with that. I feel much more comfortable saying, like, well, I can make a thing that lets you pursue the things you're interested in if you would like that. That feels less complicated. So I will need to think about your question. I find it very provocative. Thank you. Hey, Andy, I have a question for you about this idea of that AI tutor. I'm wondering, is there, like a limit to what kind of disciplines that that course kind of technology could be applied to? I've always deemed myself as being book smarts, but not street smarts. That's a great question. One of the inspirations for this talk was trying to exorcise myself of the Young Ladies Illustrated Primer. I don't know if that phrase means anything to you. That's a shame. Okay, so an inspiration for many educational technologists is this book called the Diamond Age by Neal Stephenson, which depicts a sort of far future utopian learning environment. It's like a magical book that transforms a young girl's life. And one of the things that's interesting about this book that I think is underappreciated is that it is almost exclusively focused on street smarts. So, like, this book teaches her, like, martial arts and like, getting out of sticky situations and like persuading people and so on and I chose not to deal with any of that. I think the way that I would handle much of that looks very different. And I'm afraid I haven't really thought about how I would handle that. It's difficult to think about authentic practice of street smarts. What does that look like? Is it like I'm going to go to a bad neighborhood and look purposeful as I walk and then. But that's not exactly authentic practice because like, you went to this neighborhood just for that purpose. So I find myself confused thinking about this and I find myself most comfortable when I think about people who enjoy practicing martial arts not as a kind of preparatory measure for imagined conflict in the future, but rather because they like the way it makes their body feel, they like their community at the dojo. And. And so if we start thinking about that, like, oh, it would be fun to do some exercise with others, then I think some of the same techniques can help. So you could probably use tools to help you find an appropriate community and perhaps to help you practice appropriately. I play the piano, and that's not exactly book smart, but one can use practice systems, vessels for practice like I've described, to orchestrate piano practice. You can imagine extending that to martial arts. I don't know about other kinds of street smarts, but perhaps. Thanks for the question. Let's thank Andy and I'm sure he'll stay around if he will. Thanks everybody. I appreciate you.