Interface x Quick Coffee - AI-First User Interfaces - Linus Lee, AI at Notion

Thanks everyone for coming. My name is Linus, I work at Notion on AI. So I work on Notion as AI team. I've been there for about a year now and I'll show a little bit of what Notion's done in the last year. But most of the rest of this talk is going to be a little more speculative because I think that there's a lot more interesting things to talk about than the things that we've already public announced. But if you have any questions about things that Notion has done or how we're thinking about things, I'm also happy to answer them towards the end. So Notion's been working on AI products for about the last year. The first thing that we've sort of shipped three distinct categories of things in the last year. The first thing that we shipped, which we call AI Writer, which is language model. Inside your documents you can have a conversational experience of writing and drafting and iterating through ideas as you're working on them. You can generate tables, you can generate databases and summarize content and translate them and so on. It's also, I hear, quite an effective editing coach and so you can ask it for feedback and revisions. We've also put that same kind of technology inside Notion databases, which is a more structured kind of format for information. And so you can have a database of like meetings or customers or companies and summarize that kind of content or automatically tag them and so on and so forth. And then the last thing that we worked on, which we announced in beta last November, is called Q and A, which is, if you're familiar with retrieval, augmented generation. We equip a language model with a knowledge of everything else that is in your Notion, in your documents and your tables, and it's able to help you answer questions about them. Everything from like what's the WI fi password in the office? To what is XYZ team working on this week? So that is Notion. But for the rest of my time, I'm going to talk about something a little bit more speculative and I'm going to start out by talking about representations that we represent information. This is an IBM 1401. This was at one point the most popular computer. I think sometime in the mid-1960s. This was something close to half of all computers that were in usage. And obviously this is like an industrial scale computer. Individual people didn't own this, companies owned this. And the reason that it was so popular is because of its generality. So instead of having specific computer, specific machines for particular tasks, you could load up this computer with stored programs that the computer could just keep in memory and do arbitrary operations on them. One thing that's very unique about this computer and some of the computers in this kind of family, excuse me, compared to modern computers, is that this computer is a decimal computer, which means that instead of using zeros and ones to represent numbers in binary format and doing additional multiplication in that format, this computer uses zeros through nines. And so it has a much more complicated circuit for operating on numbers that makes it suitable sometimes for specific kinds of operations like finance, where you want that extra precision and you really want to make sure that you're operating the same terms of numbers that humans are operating on. But obviously, these days, computers mostly don't operate this way. We mostly have moved on to binary numbers, zeros and ones. And one big reason for that is simplicity, right? If you have a binary representation, and unless you. You can write simpler circuits, simpler operations that operate on that, and if you have that uniform representation across all the computers in the world, then we're all working together on the same kind of simple representation. And that simple representation allows us to have simpler algorithms, allows us to have simpler hardware. And so this representation shift from something more complex, albeit more human interpretable, to something that's a little more kind of tailored for computing, I think is interesting. There are a couple of other examples of this. If you look at color pickers and how design tools allow people to work with color, there is like the human interpretable representation of color, which is just obviously the things that you look at on the screen. But then if you're a designer in modern design tools, there's a lot of other ways for people to represent colors. There's rgb, which, like web design uses often. There's hsl, which frequently corresponds best to how humans perceive color. So HSL is hue saturation lightness, or hue saturation. Brightness, which is hue is what color of the rainbow the color is. Saturation is how strong or how vibrant the color is. And the lightness, obviously, is how dark or light the color is. And this is often a more reasonable way to talk about color because it corresponds more to how humans intuitively talk about color. We don't really talk about how green or blue something is. We just talk about how colorful something is or how dark something is. And so there are all these representations of color that if you're a professional designer, you're more likely to work with because it's a superior representation for the. The kinds of operations that you want to do on color, like Making something lighter or blending two colors together, and it corresponds better to how humans perceive color. But it wasn't always the case. We have to develop a decent amount of color theory and an understanding of how displays work and how human eyes work and how humans perceive colors coming out of these displays for us to get to this point. The last example that I have is my favorite, which is Mel spectrograms. Spectrograms are a way of representing sound instead of. So normally a sound is just like a wave over time, right? And it's like up and down a single number. But spectrograms split that wave, that vibration, into the different frequencies that exist. If you're playing an instrument, each kind of row in this diagram is like the occurrence of that particular frequency or that particular pitch in the music. And looking at sound in this frequency space instead of just the wave space allows us to do a lot of. A lot more powerful and interesting things relatively simply. So if you're a music producer, you work frequently with compressors and equalizers that operate on music not in terms of just what the waves are, but in terms of what frequencies the music contains. And so you might tone down the low end, or you might boost the high end, or you might do certain kinds of operations for ranges that human voices occur at versus instrumentals occur at. And you can visualize this, too. So if you visualize music this way, which I borrowed this beautiful graphic from my friend Greg Crawford, you can see in this kind of list of tracks exactly where the bass comes in strong, exactly where the repeated parts are, exactly where the high end jumps up versus where the low end drops out in this first track. And so it's a much more expressive way to write down music in some ways than just a track list or just the sound waves and the waveforms that you see in tools like SoundCloud. So all these representations, kind of alternative representations, have two things in common. One is it allows us to look at these underlying kind of information in a new angle that lets us glean new insight about them. And two is they're all built on a deep kind of technical understanding of the thing that we're trying to represent. If we didn't have a science of sound and we didn't have mathematics like Fourier transforms, we would not be able to produce this alternate representation of music. So useful representations, useful representations are designed from a deep understanding of what they're trying to represent and evolved through use and context, through people trying to use this to accomplish and improve their craft. And so one thing that I've been thinking about a lot for the last few years is whether we can discover new useful representations, useful ways to look at things and describe things more mechanically. Can we automate this process of inventing better representations? And then came language models. I have a hypothesis about language models, which is that I think that neural language models work as well as they do because they learn a more useful representation of language in the model's latent space than humans have. We work with words and tokens and letters, and it's very useful. We've gotten a lot of things done. But words and letters are not actually a really useful way to represent knowledge. One metaphor that I jump to often is imagine if you had to look at a photo, and the way you understand what's in a photo is you start out with a top left pixel and you read the image pixel by pixel, row by row, until you get to the bottom. That would be an awful way to try to understand the picture. Instead you can just look at it. But that's kind of what we do with writing. When you're reading a PDF or reading a textbook, you start out with the first word and you read all the words. And you have to keep all of this context and state in your head about exactly what you're wr, read what the topic is, where in the sentence you are. And I think there's perhaps better representations of meaning, better representations of thought that these models are constructing internally as they operate on the language. And I think it's quite likely that there's no crazy magic algorithm that these models implement. But it's more just that within the model, they use an alternate way of thinking about things, of writing things down. And in that better way, in that better representation, complex thought becomes kind of simpler geometric operations. So if that doesn't quite make sense, I have a bunch of demos and we'll get to it. But first, I'm going to lead with a couple of historical examples, because there's a lot of great work in this space that have led up to kind of where we are today. So these are all examples of different ways that neural models represent information that we found in the wild. So this is from a work called An Overview of Early vision in Inception v1. Inception v1 is like an early computer vision model that learned to recognize images. And if you look inside this model, you can find specific neurons, specific parts of specific layers of this deep learning model that seem to correspond to detecting specific patterns that humans can recognize. So you can see here is a pattern for a neuron for detecting texture patterns. There's one for some positional information, high frequency information, low frequency information, some color information. And if you look at, as you look deeper and deeper into the model, the model learns to combine these different patterns to form higher level concepts like a dog ear or a wheel of a vehicle. Another example that we see is this is from one of my favorite human computer interaction AI works of all time, which is called Using Artificial Intelligence to Augment Human Intelligence. Awesome name. And they train a small neural network to be able to understand and reconstruct fonts. And in this example, they find that the model has specific directions in the latent space of the model. And if you move in a certain direction, it corresponds to the model recognizing a bolder letter or a thinner letter or perhaps a more italic letter versus a non italicized letter or a more condensed form. And sometimes these are things that are hard to articulate in words, right? Like how do you define exactly what a condensed letter is? Or how do you define exactly how serif a letter is? But models learn these things kind of implicitly, and they structure the space inside of this model so that the geometry of the space represents all the different attributes that the model has learned to recognize about the impact. And this analogy carries over into language as well. And so a lot of the work that I've done, language and images as well. So a lot of the work that I've done in the last year was about building interfaces on top of latent spaces of language models so that you can, for example, take a paragraph, push that paragraph in a specific direction in the latent space and pull it back out again. And now it's a longer paragraph, or now it's a more positively sentimented paragraph, or maybe it's about a different topic. The techniques didn't quite work as well last year, and a lot of the work that I've done in the last year is to improve the technique and understand better how these models work so that the techniques can be improved and you can do something similar with images as well. So this is from a model called Clip and Unclip. And you can see you can start with this image and push it in a specific direction towards that other image, and you can sample every point in between, and you see a pretty smooth transition of the style of the content of the topic, of the composition of the image. In a lot of this, I work with embeddings. I think embeddings are interesting because they're a really compact numerical representation of what the model sees when it sees a particular paragraph or particular image. And embeddings technically are supposed to show us what the model sees in a sample. Does the model see what? What does the model see in terms of style? What about the text? Does the model think is interesting to recognize and remember in its neural network versus not? And so a key question I've been studying for the last six months or so in my research is, can we read out what the model sees in human interpretable terms, not just numbers? And if we can read them out, maybe we can also control what the model outputs by controlling internally the structure of what the model is understanding. And recently there's been kind of an explosion of really, really interesting and encouraging work in this space. I won't talk specifically about these papers, but they're sort of the foundations on which I've done a lot of my work. And I'm also happy to talk about them afterwards, if you're curious. But the basic intuition behind the recent breakthroughs is that are built on two assumptions. First, we assume that over the course of training, an AI model learns many, many different features that it learns to recognize in a particular input. Features can be in a language model. Features might be this input contains the name of a character, or this input is about religious text, or this input has bullet points in it. It can be anything, structural, topical, anything about the input, anything about the sentiment or the style or the tone. But the model learns billions and billions of these features. But for each particular input, the input only activates a few of these features. So an intuitive way to think about that is a single paragraph of text can't simultaneously be a Python program and also a Bible verse. It has to be like one or the other. And most features are kind of mutually exclusive in this way, where there are only a few features that are relevant for a particular input. And so maybe we can kind of train a secondary model to understand what our original big model is thinking by training this model to look at all of the ways that model processes a particular input and say, okay, what is the minimum number of features that can fully describe what is the minimum number of states in the neural network that can fully describe a particular input? And so this technique is called sparse autoencoders. And what I've done in the last couple of months is train a bunch of sparse autoencoders for an embedding model that I have. So here's what these features look like. So this is based on one particular embedding model that I've been working with, and Here are a bunch of things that the embedding model has. Can you see the text back there? Is it too small? Okay, here's a bunch of features that the embedding model has learned to recognize about the input. And these are all labeled by GPT4. So these are human interpretable. So if we click on one, for example, this feature is a particular feature inside the neural network that has learned to recognize legal text. And if you look at where this feature is turned on for which inputs this feature is turned on inside the model. All of these are kind of about judgments, court cases, decisions of courts, legal contracts, and so on. A lot of the sort of highest confidence, most frequently occurring features about topics. For example, this one's about biology and medicine. There are a bunch of other interesting ones as well. So there's one that's about questions. And so this feature detects if the input is a question. There's a feature for detecting specific numbers. So this one detects the presence of the word. Also this one detects the presence of numbers 11, 12 and 13. There's other ones that detect the presence of specific years. This one's about the general duration of time. So a week, a year, a few months. Some of these are specific dates, like 2pm or 2008. So we can, using this list of features, we can take a particular input. So let's say I'm going to take this paragraph from our launch blog post and I can put it into this little text box and I can run this input through my model and see what are the features that the model sees in this particular input. So another way to think about this is when the model reads this input, what are the terms by which the model is thinking about the input? What are the terms by which the model recognizes what's relevant about this input? And you can see use of superlative language because we say some, some superlative text. So there's historical, cultural, event, specific context, because you referenced Doug Engelbart, presence of specific keywords. Maybe it's like formal text and so on. So we can first rec. We can first see what features the model recognizes. And then once we have these features that the model sees, we can also try to edit this text by editing the internal state of this model. And so, for example, this text is about obviously human computer interaction. But we can look at this particular feature about medical subject matter. And we can see what happens when we run this input through the model. But we force the medical subject matter feature to be turned on to its max value. And now this output, maybe if I turn it up a little bit more now, the output will keep the general structure and character of the input, but it'll be about medicine because we kind of did lobotomy on the model and turned up that particular value. We can go to the question. Sorry, go to the question feature and look at what happens when we turn the question feature on in the model. And then now the output under this max heading will contain questions. You can also do this with the 11, 12, 13 feature I found earlier. So for this one, this one I think I'll have to probably boost up a little more than the default value. But if we run this input through the model and turn this feature up, there we go. So there's 11 square meter calculations. And then if we turn it up a little more, it mentions 12 bits. And so these are still like fairly fluent sentences. They're also kind of related to the original input, but now we've forced the model to see something that wasn't originally there. And so all of this is, I think, really interesting because we're now starting to understand the terms by which a neural network is processed in the input. And there's a lot of really interesting interface possibilities you can explore. On top of this, I have just one as a demo. So once we have this list of features, we can go. And here's the first couple of paragraphs of Harry Potter. And what I've done here is I've made a kind of a heat map where I have a bunch of features that the model recognized in this particular passage of text, and I can click on any one of them and see which sentences most strongly show that feature. So an interesting one is mention of children or kids, because these are the sentences that kind of talk about Harry and other characters in the story. There's references to the United Kingdom, which is interesting. There's no actual references to uk, the country, but it activates really strongly in any sentence that mentions locations that sound like they could be in the uk, which I found fascinating. Here's a longer passage from a novel called Neuromancer. And my favorite example from this passage is this feature that turns on anytime the model sees a dialogue. So anytime there's a quote or there's someone talking, there's a kind of strong prominence of that particular feature. So all of this gives us the ability to understand how the model sees a particular input and then potentially edit what's going on inside the model so that we can elicit outputs that we want. And the bottom line here is latent spaces inside these models appear to encode human interpretable and human controllable representations of models, inputs and outputs. And, and importantly, we did not have to prompt or handcraft these representations and features. They're present in the model and we're kind of like reading them out through this kind of unsupervised mechanical process. And I think that's something that's really interesting about this process, where this technique can surprise us about the internal structure of the models. They can teach us things that we did not proactively ask about the model. And that's quite different from how we normally work with models, which is around prompting and eliciting specific behaviors that we know we want. And I think this is a really subtle and very, very strongly motivating point where we can start to see surprising things about models that we didn't specifically know to ask, but can discover how the model sees the world. So there's a lot of extra room for very exciting future work. There's. On the technical side, there's just general improvements to this technique that can be had that lots of people are working on for how to discover better features, how to discover more of them more efficiently, faster. It's also in my plans to try to apply this technique to image models, for example, because I think there's a lot more features in images that are hard for people to describe with words than there are in text. The one that I'm most interested in wrapping up this talk with today is interface possibilities for what we can do with this foundation. So let's talk about interfaces. This is going to be a bunch of sketches because I'm not a designer, but I want to start with the familiar one that we just looked at. So this idea is I'm calling kind of heat maps. The right one is one that I just showed you. The left one is drawn from Andrej Karpathy's blog post on RNNs. But in both of these, we're starting to go from what I alluded to earlier around just tokens on paper that we can't really recognize. We're starting to overlay on top of that text, semantic layers where you can be like, you can take a step back from, from the page and still kind of recognize what's going on. It's like syntax highlighting, but for any kind of language. And maybe you want to highlight conversations happening in the text. Maybe you want to highlight when there's like, tension in the plot. Maybe you want to highlight when there's opinions versus facts. All of these are things that you can detect with features inside a model. In this case, on the left here, I think this is detecting occurrence of quotes or line length. And on the right it's detecting presence of dialogue. And this is one way to go from this very token based representation of text to something that's maybe a little higher level, a little more visual. Something else that I've thought about around spectrograms, which I alluded to earlier, is imagine if you're scrolling down a news story and you swipe in from the right and you see the scroll bar that's kind of like it's annotated with what kind of tone or what kind of topics are being talked about in different parts of the story. So it's kind of like an extra powerful command F Find sidebar where maybe one of these colors represents which parts of a news article are facts versus opinions. Maybe another represents whether there's some kind of a conflict being talked about or politics being talked about. And this is another way of overlaying on top of text representations other other ways of describing meaning that you that help people navigate information, understand information. We can also let people work directly with the these features and representations. Instead of just letting people view them, we can let people edit with them. And so imagine if you highlight some pieces of text and instead of just seeing bold and italic and underlined, you also see like what tone do you want this text to be? Or what structure do you want this text to have? And you drag it down and you see a bunch of sliders for do you want this to have a more assertive tone versus a more muted kind of tone? Or do you want it to be really opinionated or more evidence based on These are things that you can start to do with language models today with models like GPT3 and GPT4. But the level of precision that you can get by editing the internal states and model directly I think are a lot more intuitive because they give you smoother controls and don't require prompting. And I think they can also be a lot more precise than just prompting based methods. And lastly, I think there's a lot of metaphors that we can borrow from existing creative tools and other places. So this is Logic Pro, which is a tool for editing music. And I think it would be really interesting to think about a way of a writing tool where instead of the human choosing all of the words manually and putting them next to each other, you just kind of dump a general idea in there and then you can look at the different tracks of like, what are the topics being talked about, what are the perspectives being used, what are the tones being used and kind of decompose your writing into all these different layers of meaning and edit them independently. And then later the tool can compile it back into like an essay. And I think there's a lot of room for carrying over these other kinds of creative tools, metaphors into writing now that we have a representation of text that we understand a lot better. Oh, this is the true last one. This view is from a tool called OP1Fun, which is a kind of a repository of different samples of sounds that people can use to add to their music. And one way you could carry this over into writing tools might be if you really like someone's writing style, or if you really like the way that someone structures their article or something like that, you should be able to just like command C, copy that style and paste it over on top of your writing. And now it's. Now it's in their style or now it's about a particular topic. Like maybe you have a particular way of describing something, but you want to use that tone to describe another topic. And you're just like, lift the topic part of topic layer of that text and put it in your writing. And so once you decompose language into all these different layers, there's a lot of direct manipulation that you can let people do at the semantic layer. The ultimate vision that I have and the motivation that I have for working on stuff like this is not actually just to make text editing better, although that would be really cool. I think the motivation that I have for this actually is that writing is text is kind of an inhumane way to interact with information. For most of humanity, humans built things by like using our hands and interacting with real materials. And real materials are full of this rich texture that communicate how we can use them. Hard things are hard and like soft things are soft. And you can see, look at two materials or two objects and see whether they fit together or stack on top of each other. They have this amazing property of like, they're very intuitive when you interact with them. Exactly what you can do with it. But text is not like that. If you have two different sentences that disagree with each other, they look the same, right? Wouldn't it be cool if you could put two sentences on the table and if they disagreed with each other, you physically couldn't put them together. So that's kind of fanciful and far fetched, I think. But that direction of making interacting with information more tactile and more visual, I think is a really interesting direction to try to pursue in evolving the way that we deal with information forward. So physical materials show us what they're made of and they show us how they can be composed together to make bigger structures. And we can't do that with ideas and thoughts and arguments yet, but maybe we'll have a find a better way for humans to interact with text. I always like to end with this thought of. I think the way that Brett Victor encapsulated this idea, which I always like to come back to, is the point always is to let people bring thought outside the head to represent concepts in a form that can be seen with the senses and manipulated with the body, because that's how humans are most creative and work best. And I think that's a great aspiration to think about when we're building tools and notations and interfaces for how people work with knowledge. Thanks. Thank you so much. You had a question. Thank you for brilliant talk, Linus. This was great. So I'm going to be greedy. I have two questions. First one is what's the weirdest kind of latent space vibe or concept that you found? Oh, that's a good question. It's a bit hard to answer because the techniques that currently work are. They sort of find the most popular features first, and so they tend to be the most boring ones. We could scroll. We could scroll around and see. One that I like is there's. Oh, also the other thing that I learned while building this interface is there's just a lot of, like, trash in these data sets. So one feature that I found in this process, it's actually kind of a bug in the process, is license text. Just like software licenses that just get wrapped up in these data sets that don't actually teach the model anything, but they're repeated a lot. My favorite feature that I found that's kind of strange and kind of quirky is I don't know if it's in this particular model, but there's a feature for like, bodies of water that's kind of interesting. It's like anytime someone mentions bodies of water or like maritime activities, boats, this thing is active. There's also, I mean, the 11, 12, 13 feature I think is actually quite interesting too, because it's so specific. And one of the things that you find when you train a bunch of these kind of interpreters, secondary interpreter models, is that the bigger. The bigger the original model that you're trying to interpret is, the more granular the features are that it learns. And so in a smaller model, this would probably be a part of a generic presence of number feature. But in this larger model there are specific for commonly occurring numbers like 11, 12 and 13 or 10 or 5 or 3. There are specific features that respond to the presence of those. That was my second question. So in larger models you're saying that they become more specific versus weirder because like my sort of, I'm thinking like, how does that look like inside of GPT4? Do you get like these weird neurons that are like, I don't know, taking a walk in winter and thinking about Heidegger is one? Yeah. I mean, at the scale of so. So this model is 1 1/2 billion parameters, encoder and decoder each. And so it's on a totally different scale than any of the modern language models. This is also an embedding model. So an embedding model just produces like a fixed representation, a static representation. My mental model for autoregressive language models are a little bit different. And the intuition that I have there is these models are basically the models spend some of its capacity translating human words tokens into these representations, these features. And then they have a lot of circuitry that they've learned internally that are kind of mini ad hoc algorithms to operate on these features to predict something about what should come next. And so an example of a circuit like this that's very common is like one thing that's very common in a lot of these training data sets is that the same name or the same proper noun or phrase will get repeated over and over and over. And so if in this data set there's a token that's like Linus, a lot of things can go after that. Usually it's probably like Torvalds or something. But if the data set mentions like Linus Li and then later it mentions Linus, the thing that follows is probably Lee. And so there's a little circuit that implements that like copying process that's called an induction head. And there are a lot of these circuits in larger models. I think there are, especially in the like, earlier layers. A lot of these circuits are for just like operating at the like linguistic level of predicting words. I'm sure there are also circuits that exist that operate at a much higher, more interesting level of like even something as weird as is this sentence grounded in reality? Or is the sentence supposed to be a silly joke or fiction? But I don't think anyone has done that kind of phenomenology on these features. Models that are large enough to get that? Interesting yet? Thank you. Actually, let me ask one quick follow up that I thought about just now. I mean, what you basically showed is like it could be a kind of garage band or editor for text, right? And what I ask myself is, you know, if you now have a much powerful tool to be a writer, and you can produce much more text that has different styling, and you can tune this text in a way that you maybe wouldn't have been able without that kind of tool, what will make a good writer? What, what's going to really? What's going to. What's. What kind of human interaction will create good writing if you have better tools than just writing out the words that flow from your brain? One way to think about writing is like one way to think about an embedding space is that it's a very large. It's like a very large geometric space and you can pick any arbitrary point in that space and that corresponds to some output text. And if the embedding space is structured well enough, every. Almost every conceivable piece of writing that could exist is somewhere in that space. And so like all the paragraphs that have been written before, all the paragraphs that will be written after, as long as it is like under 250 tokens is like in the embedding space of this model, which is kind of weird to think about. If you've read there's a great short story called the Library of Babel from this Argentine sci fi writer, Borges, and he has this concept of like an infinite library that contains every possible text that could ever exist with a fixed number of characters. And an embedding space is kind of like that, where you can pick any point in the space and for any given output that you want to elicit out of a model, there exists an embedding that corresponds to it. And so if you hear that, one immediate reaction could be like, well, if every piece of writing that could possibly exist exists in this space, and what's the point of writing? But the space is so inconceivably vast that you still have to know where to look. And I think the skill in writing, perhaps it gets a little easier to navigate around the space, but you still have to want to go somewhere, you still have to want to say something to evoke a particular reaction from a reader. So I think the number of choices that you still have to make are, there's still so many choices you have to make when you write. And I think this just makes the space of those choices easier to Navigate. But the importance of those choices and the taste that drives it, I think are still very important. Interesting. Thanks. Thanks for sharing. When I see this as kind of a thing from a user perspective or what I would use this for, what is very interesting is that I think by kind of exposing this, you're showing us a lot of like, what I would think of myself, like, subconscious context that I use in my writing that I might not like consciously be like, okay, my writing has all these clusters of features that a model recognize. And it would be interesting to unearth these features. On the one hand they would help with creativity. On the other hand you could say like, what's the opposite feature set? So like, oh, I wrote a certain text and it's very similar to another text because the model interprets these features having, you know, these weights or this importance in this clustering that can also help you synthesize your thinking. Like, oh, there's maybe some subconscious things that I'm doing. Maybe it's in my tone, maybe it's in the context. It's very interesting to see how your, what you've, what you've built is how it interprets what you've written and might unearth. This sounds a bit woo woo, but some of the subconscious context that you've used in your writing in a way because it's being interpreted, the relationships in that are being interpreted in your text. Yeah, that wasn't really a question, but I can, I can react to it. Yeah. Something that that reminds me of is I don't know how many people here use the Gmail web site, but if you do, and if you have like the Smart Auto suggestion enabled as you write, the model will try to like write a few words in front of you and you can hit tap to autocomplete it. And that feature is customized per user. So depending on all the emails that you've written before to people in Gmail, it'll try to predict the word that come after based on your own writing style. And because Google has to serve billions of people with this, they can't train a fully custom model for every user. But the clever trick that they use is that it turns out they can express almost all of the variants in different people's writing styles by just having one good underlying language model and then keeping track of some bigram statistics. So biography statistics are like, for me, maybe when I say have a good, I'm more likely to say day, whereas someone else might be more likely to say weak. And so they keep track of these word Pairs that occur frequently for specific people. And even just that layer of extra customization is enough to determine, at least in the limited context of email, a lot of the variance in writing style between people. And so I think that it reminds me of that because something you alluded to is the. What are the. What are the actual things that determine the difference in writing style between you and me? I think sometimes it can be as simple as, like, something that looks just like statistics. And obviously there might be more interesting things as well in larger models, but that's a. That's an interesting starting point, and we've got maybe even one level deeper than that. Like, what is even more deeper than beyond style? What are some other subconscious contexts that you, as a writer might not kind of think of? But what this model does is it unearths some of these things that you didn't consider but were there in a way. Yeah, that's also certainly possible. Take one more question and then take a quick break. Hey, Lus. So if you think about, on the one side, you have, like, technology changes, like the printing press of the worldwide web brings kind of media that's qualitatively different into people's lives. And do you think some of this stuff with language models, I know it's hard from looking into the future, but it will bring, like, by the same token, something like a webpage, but for the language model into people's everyday lives. And like, what might that look like or what may some of the features of that be given? You're kind of exploring this. One interesting way to think about that might be like, what is like a new noun that this gives people? So what I mean by that is like, The early Internet gave people a new noun, which is like a website, which is a concept that didn't exist before. And you can send people, or like a URL, and you can send people URLs and you can simply websites and then after that. But like, the websites are like a static thing that exists at a particular URL. And that is, I think, these days, somewhat of an antiquated concept, because now the same URL can look different for different people, but you have some other noun instead, which is like a post or like a profile. And those are like new abstractions that we have with new technology. So one way to extend that into kind of what I just talked about may be like, I think a really interesting feature would be one where people can refer to specific styles or specific tones or specific kind of ineffable features in writing or in images as like nouns. You see some of this with things like textual inversion that Fagan talked about where at least in the world of image generation because images are so hard to describe with words, people often will like train a specific fine tuned image generation network and say this is a particular style and that's a noun that's like an abstraction that we can grab and be like okay, I want to make this image with this style, with this character. And before models we couldn't do that because models give us a way to concretely talk about a particular style or a particular composition or character. We can do that now. And I think as these techniques get better it'll get easier to discover those styles and it'll get easier to share those styles with people. And so I think that's, that's one like a new concept that I think might, might be brought in. Thank you very much Linus, thank you very much for making the trip.