Transcript
Use with ChatGPT / Claude Auto-scroll A
The speaker introduces the video’s goal: building Netflix-style continue watching as a focused system design feature. He explains why the problem is hard at large scale and outlines the tools and code walkthrough that will follow.
Speaker A
Hey there everyone and welcome to another awesome video.This video is a package of learning about system design.Some of the concepts that are used at Netflix as well as we are going to jump into the code base as well.The whole video is just around this one feature which is continue watching and building this feature.Continue watching at Netflix scale is really the challenge.Anybody can build a continue watching but when it comes to the Netflix scale, this is where problem comes into the picture.For example, if I just show you how many people are there on Netflix, There are around 325 million paid subscriber and I'm pretty sure this is a bit outdated information, but let's just say we here and there assume 300 million users at a given time, even if 10 million users are watching.And let me remind you, usually people watch on multiple screens as well.
They want a seamless experience on mobile as well as on tv.And if all of them sends a request onto a database, it is pretty easy to overwhelm your database.
So this video is going to walk you through about the system design concept, the continue watching we will understand the theory part of it.We will understand that where redis fails and what you can do about it.Not only that, I will also walk you through with the code base of this entire thing that how this can be coded so that you can practice as well.And and don't you worry, I'll give you the repo link as well so that you can also understand the code in case you want to or you can use any coding agent to make sure the code is easy and understandable for you.So that's going to be this one.
By the way, I asked on my WhatsApp channel that hey, would you like to jump into the features of system design where I help you to understand just one feature, not building the whole Netflix, but one feature which is really, really difficult to implement.And a lot of you said yes and I was super happy about it.So in case before we proceed with the video, you also like these kinds of video where we don't do the entire Netflix thing or Amazon thing, but just one feature which is really difficult to implement and a whole lot of engineering team works behind that.Then let me know in the comment section.Your comment is really, really important for me.
It gives me idea, it gives me motivation that yes, you do care about these videos as well.By the way, this is the time of AI, so we won't be writing the code manually.I will walk you through, I will give you a tour of the code.Feel free to use agents to write the code once you understand the concept.And that's exactly what we'll be doing.And that's the best part.Gone are the days of like 8 hours, 20 hours of video where I implement this.If I had to show you the implementation by hand, surely this would have taken some time.Anyways, now just go ahead and let me know in the comment section.Would you like to see more such video?
I'm waiting.Yeah, still waiting.Come on.Just small asking.Please do that.
Now let me take you on to the journey of what we are building here and how it is going on.By the way, I do have this one as well.I will walk you through this portal as well.It doesn't look really nice, but it has some of the things at the bottom which will help us to understand a lot of stuff.Anyways, okay, so let's start this one by saying that how does Netflix remember where you stopped watching?
It feels like very obvious feature.Yes, Netflix knows it might be saving some things in the database to let them know that what I was watching.But it's a seamless continuity.Whatever you're watching, you can just immediately go on mobile and start watching it.It still remembers.
And building this Continue watching is really an interesting feature.Whether you're building Netflix clone, maybe a YouTube clone, any EdTech website, JioHotStar, Amazon prime, all of them implement this.And knowing this gives you gives you an extra hand in the system design interviews as well.
This section explains the core problem of frequent heartbeat writes from many devices and users. It shows why direct writes to Postgres create locking, vacuum, and throughput issues, motivating a smarter architecture.
Speaker A
So we're going to build Continue watching.We are going to use Dragonfly for this, we are going to use Postgres for this and we are going to use ImageKit Player for this one.
Why Dragonfly?The one example that I'm going to give you about this Dragonfly is just simply here.If you just look at the stats, it's 25x more QPS than Redis.Yes, it is much more performant than the Redis.Why so?Because of this simple diagram and architecture.This is what Redis does and this is what the Dragonfly does.Dragonfly shared nothing thread per core architecture eliminates the scaling bottleneck of traditional real time infrastructure.So the idea behind the whole Dragonfly is that when you reach onto a certain scale, the architecture of Redis is not that sustainable.And Rediscover and Redis has done this again.Again they don't.They cannot obviously change their architecture now, but what might cost you at the Redis might be a fraction of the cost at Dragonfly and the service here.By the way, they also are on GitHub we are going to learn and practice them through the Docker container.
So you don't need to spend that much of amount of money.But yes, a lot of people use them for enterprise.
This is like a pocket change for them compared to what the Redis bill they are paying.They also allows you to use a lot of things for free as well.If you log in and sign up they have this credit system as well.You can just cash their credits hundred dollar they gave you which is pretty generous to use all of this.So you can start with free to play with the infrastructure as well.
And this is the main reason why we are using Dragonfly.Apart from this, what we are also using is Image Kit.I had to host videos somewhere to show you the examples in demo and and I happen to choose Image kit for that especially.I'm using the video player which is again open source, freely available, no charges for this one.So you can just go ahead and use this one.I will walk you through that.What are the important features of it and which you should be looking in every single player if you are building this kind of application.And Postgres, of course, everybody's favorite.That's what we'll be using.And I will also walk you through what is the point where postgres becomes a bottleneck in this scenario.
Hope you're all ready.All set.And of course we do have a code base.I will give you the entire walkthrough of the code base so that you can understand it shouldn't be just like a black box.Like usually it happens with the Claude and codecs.But you will understand fairly enough once I'll give you a walkthrough.So let's start with that.So it looks like a tiny row of cards whenever you're watching Netflix movies, this is how it looks like.I happen to have these three videos.Database design, Twitter Timeline and Agent Tick overflow.
And wherever you have stopped this video, this is exactly what it looks like.In what order am I watching right now?All these things.In fact, Netflix also gives you that whoever is watching this from this account, it gives you this is live right now.Or at least somebody is playing with that.
So there are three questions.Every design choice in this talk answers one of them for millions of people at once.Where did I stop?In what order?And am I watching it right now?These are the three questions that we have to build in our design.And we have to design it for millions of people.This is where things get interesting.Why?Because if you look at that Scale.
I'm not asking for 300 millions of scale, I'm just asking 10 million, which is very low compared to the Netflix.If you look at this Netflix, if somebody is watching on that player, that player sends some heartbeat or a signal to the database after every 5 second, known as heartbeat in the concept.And, and this heartbeat allows them to write something in the database.Now, if every five second you're writing something in the database, this is the number that you write every second.And no matter how good your postgres is or how scalable your postgres is, this is going to choke up eventually.
And even if you can handle this, there are some better techniques in system design through which you can actually reduce it to frame frictions.Fractions, not frictions.Okay, so almost every two of these queries is overwritten five second later.We don't need history, only the latest position.And this is where the smart architecture comes into the picture.
And again, this is a round number.This is not the real figure in Netflix, but this is usually the assumption that we take.I hope that makes it interesting that it is kind of really interesting to see when it is just 10 million 5 second.Oh man, that is a lot of writes for queries.Now scale this number to just 20 million, 50 million, 100 million.
And you get the idea that what happens.So I hope that is very clear to everyone, that the video player keeps telling the server that I am at this position, I'm still watching it, and this heartbeat is captured and is saved to the database.Now, foundationally, there is nothing different.We are still going to use Herdbeat because this is the only mechanism in the system design that helps us to track the user.But we are going to do this smartly.
So if you look at this, if you just save the Herdbeat to Postgres, this is how it's going to happen.The same account might be sending some data from tv, from the phone, from the laptop, or maybe one account has multiple users.And Netflix allows that officially.And if all of them are writing to the postgres, the amount of write that you are going to do is a lot.What one Postgres server can roughly take is 10 to 50,000 per second.But you are giving it this many.And Postgres is built to never lose data.So every single update, what you do is a real disk work.Yes, I know that Postgres uses something known as Wal write ahead log.So if anything goes wrong, first of all, Postgres writes it in the logs and that is saved to the disk.And from this disk, Postgres picks that data and store in the database.
And although it is very crash proof because of this wall, but there is also a concept in the postgres which is vacuum.That means the rows are being just appended.They just keep on adding the rows and old rows that are no longer in use are.The reference is just removed from them.
So a process known as vacuum or a technique known as vacuum comes behind the scene.And it's a technique which does the cleanup job.It removes all the rows which are not being used.Of course there is an index as well in the database and there is a row lock as well.So database has to acquire a lock in order to modify that particular row of the table.
And if the same user is watching from phone or somewhere else, you cannot do anything because you have acquired a lock.So as you can see, that postgres is nice, but it comes with its own challenges as well.And by the way, these all challenges that I walked you through, they are system design interview rounds as well.People ask them a lot.Anyways, let's move on.
The speaker introduces the split between hot in-memory data and durable database data. He explains the write-behind caching approach using Dragonfly as the fast layer and Postgres as the durable store.
Speaker A
So this is where the key insight comes up.So we are going to split the hot data from the durable data.Now we have two types of data.One is hot, which is keep on coming in.And another one is durable, where I want to save the user.Maybe last hot data was very important and user comes back after two days, maybe after a trip and he wants to continue watch the movie.So in this case the hot data, which is hurtby data that goes to the Dragonfly.And because the scale is so high, I think Dragonfly only makes sense Redis.You might have to spin up a lot of clusters of Redis.Surely you can do that with that.But the bill is going to be enormous.So instead of spinning up the cluster, people prefer to go with the Dragonfly services because it has a very optimized architecture.
Now reading and writing in the memory takes under a millisecond.So surely we can handle the hot data and overwriting a value cost almost nothing in the Redis ecosystem or Redis like architecture.Now it serves the homepage row directly, but whenever we want to write this, a copy lands here every minute, not every second.
And we're going to store only the latest position.And it obviously survives the restart.This one doesn't.But there are techniques how you can actually make it survive the restart as well.I'll not go into that.But yes, the techniques exist now.And we are going to refill the memory if everything is Swiped.Now here's the interesting thing.The hot data changes constantly and is read constantly.So what we need is something which is instant, which is in memory.And this is where the data, the Dragonfly, comes into the picture.Now the durable data must never be lost.And hence we are writing this.Now here's the interesting thing.We are not always writing that in the database itself.We are first keeping it as a cache.But there are techniques as well in the cache.We are keeping it in the cache, I would say Dragonfly in the memory.
And then once after certain amount of time, let's just say after one minute or two minutes, you're writing that in the db.Here's the interesting thing.This concept is known as write behind caching.And a lot of time interviewers ask what is write behind caching?Can you give me in one example?This is the classic example that.Continue watching.
This is a classic implementation of write behind the caching because you write behind the caching, not in front of the caching.First the caching is being done and once caching is all good or there's some timeline, then we write in the db.So very interesting concept.I hope it helps you to clear some interviews as well.Okay.
And in case you don't know, cache, a cache is a fast temporary copy of the data kept in the memory.Pretty obvious.You know this.Anyways, let's go there.So this is the whole system that we are about to build in this one.And this is a pretty interesting one as well.If I go ahead and zoom this a little bit, you can see we have this ImageKit CDN which is serving our videos.We are taking these videos in our video player which is in the browser.And our video player is constantly sending the message to the API server.Now API server could have directly talked to the postgres, but we are not doing this.
We are using another technique to save some of the time.And this you can see we are writing directly to the Dragonfly db.Once Dragonfly DB is writing, it is writing almost every millisecond, it can actually handle that.And then we use a simple flush worker which every 60 second takes the data from the Dragonfly DB, which is pretty easy operation as well.And then just go ahead, write at its own speed without overwhelming the postgres in this.
Now if any time user comes back, he doesn't read from the Dragonfly db, he actually reads from the postgres which is your loading time in the Netflix.So this is all what we are doing.So again, this 60 second is an arbitrary number that I have taken.It could have been two minutes, five minutes, whatever you wish to have.I hope now you understand the architecture that we are trying to have in this ecosystem pretty good, Pretty easy.
This chapter covers how grouped writes reduce network trips and database load, including the flush worker that periodically persists cached progress. It also discusses the acceptable tradeoff of losing a small amount of recent progress if the cache restarts.
Speaker A
Okay, now that you understand the architecture, here are some interesting behind the scene system design concept which are asked and I'm pretty sure if you don't explain them or understand them in this sequence, it is difficult.So there are three data structure do all the work.For example, the Dragonfly store keys to point to the value.The value can be simply longer or string.So here's the three thing that we have to do.
So we have a hash, we have sorted set and we have a string.Which one would you like to pick to actually store inside the dp?That's a very interesting question itself and I'll give you the answer as well.Don't worry about that.So let me go ahead and get started with that.
Okay, Write in one trip, read through, read without touching the database.The idea behind the architecture of Redis or using Dragonfly is that Dragonfly is sort of a not sort, it is a drop in replacement of Redis.So whatever Redis can actually do.Dragonfly also support that.And one of the major problem with us is we want to to do all the things at one time.
If you remember.Let me take you on to a couple of steps back.There are three things that we are supposed to do.Where did I stop, in what order and am I watching it right now?If I go ahead and update this, that means I'm acquiring a lock in the postgres, then I'm done.
Then I'm doing the second step, then I'm acquiring a lock and then I'm done.So instead of acquiring the lock three times, what I can do is, is I can just go ahead and acquire entire lock at once and I can just go ahead and write all these things.For example, start a group run all together.That means I'm going to go ahead and write in my memory database Dragonfly that hey, my position is at 120 seconds in the movie.Then I will move this to the front that hey, this is what I'm watching right now.
And it's live for 30 seconds.So for the next 30 second I'll make it as live.By the term live, I mean somebody's watching it.And then I'll simply copy to this in the postgres later on and then I'll run the entire group.By running this in the group, what will happen is that I don't have to do all these entries one by one by one.I can do all this in one stamp and go and that's it.It's saved in the memory much faster.One network trip and the API just response saved in one millisecond.If you don't perform all these operation at once, that means you are writing them and then getting a feedback, then writing them.So make sure all of these steps because memory is faster, should be done there.
Now if you remember, we also have a worker here which is pulling this flush worker from the Dragonfly.So this worker simply just run for this Zrev range, whatever you call that in your.It's very hard.People have opinions on this.Some call this as Z rev range, some people call this one as a Z range.
Lot of names anyways.So what you can do is you can Simply get the 10 most recent videos, the newest first and then you can simply work on that.Now this part I will walk you through in the code base as well is little bit dependent on the database as well.So let me just shrink that.Don't worry, I will walk you through with the DB part as well.
So where is my database?Here it is.So how you are actually designing your database that is also dependent on that.And here's the schema, I will walk you through with the schema that why we are doing it, how we are doing it, all of this.Okay, just for a second, leave that.
Okay, so the architecture is pretty simple that we will get 12 beats, her beats in 12 hertbeats and then for that we are going to write just one database.Right.If this is still overwhelming, I can also go ahead and get 24 beats and write for that one write.So again, it is all configurable now.One minute of watching a heartbeat every five second.That means at 65 seconds, 72nd and all of that, once I know that, okay, this many heartbeats I have reached, then my worker can just go ahead and write in the database so that I don't lose it.And again, since my Dragonfly DB instance is also pretty large enough, I don't have to immediately do that.I can afford a few good seconds of delay in this case as well.All right.And this will be 12x fewer database writes.
So already you're optimizing 12 times, you're writing less 12 times.Good.And again, even if I miss like few seconds, it is fine in this case, system design is not about making everything so much accurate, it's about what kind of trade offs you are accepting.So in this case, even if I miss one or two, a few seconds here and there are totally okay.So I hope you got it.
Now comes the interesting part.This is the most interesting part.So what if the Dragonfly loses everything?This is an honest trade off.Dragonfly can restart and its memory could be empty.Although you can save it.There are features for it, but let's just say we are not using that feature.What happens in that case?Nothing.Hardly.You'll be losing like what, one minute?Not big deal.A user opens the homepage and their watch is missing.No worries.I can pull this off from the postgres.
Load that user's unfinished video from postgres.That's all I have to do.Put them back in the Dragonfly and serve the page.It's fast again.So this is an honest trade off that yeah, anything not yet copied, that is mostly 60 second could be lost.And we are fine with that.Although we can do multiple dragonflies as well.But anyways, this kind of thing is okay.
The speaker explains why Dragonfly is used as a Redis replacement and how its performance and persistence features help. He then dives into hashing, Redis cluster slots, and the key design choices behind storing watch state efficiently.
Speaker A
Now here's the interesting thing.Okay, so this is all about Dragonfly, why we are using it.First of all, this is drop in for Redis.All I have to do is change the URL of the redis.That's it.Nothing has to change.Our code is plain IO Redis, nothing special.Whatever the code you're writing, you don't have to touch that.It uses every CPU core as I mentioned in the diagram as well.This is what I'm talking about.It uses all the CPU core.The architecture is different and that's why it is able to serve much more efficiently.It uses your machine efficiently and there are safer snapshot as well.So it can actually save the memory to the disk.As I said, even if the Dragonfly restart, there are ways how you can protect your memory.So it can write to the disk after some X amount of time and whenever it restart, it automatically pulls that data from the database.
As people say that hey, Redis is pure memory.Once it's gone, it's gone.Not with the Dragonfly.And of course it is memory efficient.You can see we can how many this can be saved.By the way in this repo.I have also given you NPM load test which can.Which is actually limited to two cores only, but still it can do a lot.So.And another part of this one is the video player that I'm using.The video player that I'm using is of the image kit and the reason why I'm using it is if you look at the website, it is very adaptable.First of all it is video transformation.So all the transformation and changes in the video can be there.This is my favorite adaptive bitrate streaming.So all the HLS and Dash protocol.In case you're not aware of it, let me know in the comment section.
I will walk you through with that.This is really important because you don't want to load the entire 5gb or even 800mb or a 900mb video all at once.You want the video to come in chunks.And this technology is responsible for taking the videos in the chunks so that you can serve that.Apart from this, you can have the subtitles, chapters, player enhancements, custom branding, shoppable videos and analytics.You can make your video shoppable in between.The video user can click and just buy something.And this is exactly what we are using.I can upload just one MP4 in their system.It automatically does the transcoding all the changes.I can use my videos in hls.That means adaptive bitrate streaming.I can stream my videos in 360 and automatically it changes based on the network.I also have the seek previews which is really hard to implement.
It takes a lot of tech and there's a player events as well.So this is the code.That's it.And anyways you are not going to be writing the code.You will come here at the top and there is.There we go.
You will just simply say copy for LLM all view markdown.You'll provide this and that's it.Your agent can implement it end to end.Okay, so from one laptop to hundreds of millions of users.But there's still one more big picture that probably you have missed.So let me walk you through.First of all, what is hashing?This is another system design concept that we have to learn in here.I know lots of concepts, but hey, that's fun.Okay, so imagine we have a 16348 number shelves.
Okay, first of all, I'm pretty sure you are looking for this.What is this 16384?This is.This is a common thing.So if I go ahead and look for this 16348 Redis, you're going to see this number pops up every single time.
Why Redis Cluster only have 16384 slots.It is by the development, when the development was happening, this is what they used.It is actually a 2 to the power as well.But anyways, this is the magic number everybody uses.There's a story behind that.Let me know in the comments if you want to see that story as well.But hashing is required because if you don't do hashing, this is half the knowledge so far.And I'm pretty sure a lot of people have watched just the half the video so far.We have understood the technique, but how do we implement in the code?This is where another system design concept of hashing comes into the picture.
So let me know in the comment section if you're still watching.Because if you are still not watching, that means that you have learned a little bit half of it.Let me show you the demo of it.So as you can see, I can just go ahead and change the user.Let's just say I go ahead and move into a user.Two, I go ahead and search for that.Hey, I want to watch scalable Twitter timeline.I go ahead and play it a little bit.Maybe I move a little bit.Notice this how good the player is.
I can just seek through it.That's why I said that it is so good of a player that I can just seek into it.Now that is it.I want to pause it and I'll go with this.Now notice here the scalable timeline, it is not writing directly in the db.It is just sitting here writing into the Dragonfly and eventually it will write in the postgres as well.But here's the interesting thing, I can just close it and now notice it.This is still watching 32nd.And if I go back to different user, I see their profile.If I come back, hey, I still see my profile, I can resume this.
It will start exactly there.This is exactly what we're building.And another missing component of building this is this hashing.So here's example.Imagine a library with 16,384 number of shelves.
By the way, this split this room into.Split this across three rooms.So this is number of shelves that we have and we are splitting that into three rooms.Now if we have to place a book, where you are going to place the book in the first 1/2 room or the third room where you are going to put this.Now the obvious thing is that, hey, I'll put it by doing a hash.
This is where interviewer is going to catch you.Okay, you do the hashing.This is the hash function that I have used which is a CRC 16.In case you don't know, CRC is cyclic redundancy cycle check.We could have used here more complex algorithm maybe SHA algorithms or something.But those are designed for attack vector surface.They consume CPUs as well.This one is good enough here.So let's just say you have now got your hash with the watch user one.Okay, good enough.
This is the value and hash.All these are being calculated.But what's going to happen with this hash is if you use this one a same user, the three points, the recent, the playing and the watch user one they may end up into different server.This requires more attention.Okay, I thought hashing solves all the problem in system design.
This is where interviewer catch you.No, it doesn't.So in case you do the traditional hashing for any of these, for this one user that hey, this watch user is the key and I want to save the data.User might end up in any of this.So user 1 three keys landed on three different servers.
Our save steps spend all the time in one group multi.A user can only run on the worst server.So while retrieving the data or saving the data, we are saving them on the one server itself.But there could be a thing.This is very strange, isn't it?
The thing is, it's not just you don't want to just want to hash it.You want to hash it into a very special way.So put the braces around the part of the key like watch and then the user one.Notice this very carefully.Only the part inside this parenthesis goes into the hash function.
So all of the user keys get the same slot.Notice the difference here?This is how we were encrypting or hashing, not encrypting.And this is how we are hashing here.So these small steps can actually change it.And there's a big difference.In case you didn't understood this part, I don't want to get into too much of just the hashing part.In case you want a detailed video on it, let me know in the comment section.I'll do a deep dive thorough into it along with the code example to understand that hey, what's the difference between playing user one as well as this one watch then the user one.So if you got it, that's fine.I was expecting that intermediate.But in case not, let me know in the comment section.I'll do that one in detail as well.Okay, so why slots and why 164 of them?
This is where you want to understand.
So the obvious is I can also go ahead and hash it with the 3, but as soon as I change from 3 server to 4 server the answer gets changed.So that is why we always go ahead and check it for this magic number of 16348.Even if you put up something more it is fine.So why is 364 not this one?This is fixed forever.This is something that everybody uses in the production grade.So a key slot never changes.And even if the server goes up to 1000, it's still same.And this was given us this number.Magic number was given us given to us by the Redis team or Redis founder itself.
And pretty far enough it's a pretty good one as well.And in case you have ever worked with Redis or Dragonfly, you might have seen this number as well.Anyways, so five ideas you can reuse anywhere.First of all, separate hot from durable.This is 101 system design.
Not every write deserves a disk.Not everything needs to go in postgres, at least not immediately.You can do it later.Also, merge your writes.Okay, I understand that you are pulling up of your writes from the Redis or the Dragonfly.
But again it should never be done like this.Merge, write, collect them and write them.Pick the right data structure.Hash for the state sorted set for the order TTL for is it live?You need to pick up the correct data structure.In case you are not aware of the hash or the sorted set or the ttl, let me know in the comment section.I will do a dedicated video on these data structure.This is not going to be a lead code data structure.This is more of the production side.Okay, if you are planning for losing the cache, know what you lose and how to refill it.
Always have the cache eviction policy.There should not be any surprises ever in system design.You should exactly be aware of when the cache is going to be evicted and how it is going to be split.The user by hashing as well.Make sure the slot that you are adding to the server, it doesn't move just like anywhere.
This section walks through the project setup, Dragonfly deployment, and the database schema for videos and watch progress. It also covers the web player integration and a strict mode bug encountered during implementation.
Speaker A
So let's go ahead and now talk about the code base.I'll give you the code base as well.Don't you worry about that.Let me first get you that how you can actually make the Dragonfly up and running.For this all you need is.First of all, by the way, let me walk you through.This is a monorepo configuration.Pretty simple and pretty easy in that you can see there is a docker compose file.In the docker compose file.I'm just using a Dragonfly directly.
I can actually, in case I'm on production or something, I can just go ahead and log in.There are data stores here as well.I can just go ahead and create my own data store as well.Just click on it and I can Just go ahead and use it on aws.This is a very production grade and professional machines.
Or I can use GCP or Azure whichever is your favorite.And then I can choose my region.Mumbai should be good for us.And then we can go go ahead into cluster form, emulated or swarm in case you know about it.I think single shard is good enough.
And then we can choose the memory again.It uses memories and core very efficiently so this should be good.There is no eviction policy as of now.This is good enough.Then the security is good enough.
We can go ahead and check out some specialization as well in case you are using it with BullMQ or something.And you can actually go with the cache eviction or no eviction.These are the two common one asked in interviews as well.But I guess that's it.And then once you click on Create it is going to create the machine for you.
They give you this hundred dollar free credits as well on creating your first data store.But one interesting thing, you don't have to pay anything to claim this hundred dollar free credit.But since this is a very professional grade production service, they are going to ask you for a card information.So you just put the card.They don't charge you anything, just like $1.That also gets reverted back to your account once they verify otherwise.Giving $100 to each one who sign up is not a good idea.At least by putting the card they can know that hey, that's a unique user all right.And by the way, there's a NET network backup and all these things.But in case you want to just try this out, make sure you use a configuration similar to this one.Why this one?Because image you can put a pull up from the docker DragonflyDB but also make sure you mention this command because Dragonfly needs 256 MB per thread and how many threads you want to use.
You have to provide that.So just multiply it if you want.Two threads, four threads, depends on that.And just go ahead and change this one.That's it.And also make sure you mention the thread here.So that's the most important line.And there is a U limit as well.I'm pretty sure you might be aware of it in case not let me know in the comments as well.And then there is a port.This is all it takes to spin up a Dragonfly DB and it's a pretty rock solid one.In case you haven't noticed this one on the Dragonfly there is a GitHub icon as well.Make sure you start the repo support them as well.
We need more such things to be available for everyone.By the way, this and these things both are available on GitHub.Go ahead and check out their source code, contribute meaningfully and help them so they actually gives you a lot of stuff.Market is full of free stuff and GitHub is one such awesome place.Anyways then this is our trusty postgres.
I don't think I need to explain it much.This is just postgres password.That's it and that's it.This is all what you need as an infrastructure.Once this is done, the next piece of step is the database and you'll be surprised I'm using by the way drizzle in this one.
You'll be surprised to know how clean and quick our database is.This is the index pretty basic.This is the client pretty basic.The schema is pretty interesting one and very basic.There are two parts of it just to implement this one.I'm not implementing any authentication or anything like that.So if you look at the videos we have the ID title, URL and the duration.That is all the video we are saving id, you know that it's a primary key title for each video we need.That URL is coming from the image kit itself and then we have the duration in seconds.The watch progress in itself is a table which actually stores the user ID and the video id.These are the two things which I am interested in with the watch progress that hey, what video I'm watching and who is the user.
Because if I come back in here, notice that I can just close this one and I can switch the user.So I'm simulating this via the user and which user is is watching what video.I can just come and resume this and you are going to notice it resumes directly there.How beautiful this.
I wish that this library would have existed even when I was building the first version of my products.Anyways, I'm going to switch after this video right away.I'm going to use this.It is freely available, it is on GitHub so I'm going to use this like anything.And by the way there are a lot of settings changes, adaptive bitstream and it is configurable.
I haven't much configured it, just a bit of a glass look, but I can configure that anyways.So this whole design is what we are doing and the three things that we are writing in the DB of the Dragonfly now they come into the postgres here so positions in second is where you are in that video the last watched at and then there is a completed as well.So all these things can be there and you can store more as well.But I think that is enough.And if you are live watching I can keep it in just the Dragonfly and that's it.
I hope the database part is clear because there is nothing more in it.That is all.And anyways your agent is going to write it.You just need to watch it that hey, this is all going good.Yeah, it is going good.
Okay, now comes the part of the apps.So this is a monorepo.We have two apps in it.The first one is API, the second one is web.There is an interesting thing in the web.
Let me just show you in index HTML and we have this main.No, don't want to do that.So there's the source and there's this main tsx.What you might have noticed here, there is no strict mode in here.When I was using the strict mode and my agent also gave me a couple of headaches on this one.
So when I actually have this strict mode the strict mode loads this player somehow.It's a weird bug.So it disposes the videojs player on the first mount and it removes the video element somehow which crashes and nothing plays.So I have written this for you as well.I'm pretty sure the English is not as correct because agent didn't wrote it anyways.
And this was a weird bug that I saw.But again just remove in the production when it will be going.It will not be a strict mode anyway so it doesn't show up in the production.But I'll definitely be reporting them on the repo.The good thing is in the repo I can just go ahead and quickly report the issue.
There is only one issue right now which is good and I'll be just sharing this one as well.Anyways, this is the only thing you need to worry about it Apart from this.You can see that there are all the hooks available in the player itself and you can just work with that whatever you want.It can do all the polling, all the things that you need in the API.I'm doing all the polling listeners and requests that's coming in anyways.No need to worry too much about it.Agent is going to write it, Claude codecs, somebody.You are probably not going to be writing that.You just need to aware where the problems are and how the application architecture is front end anyways is simplified now.
The final chapter shows the flush worker, Redis/Dragonfly interaction, and the end-to-end demo of resume playback. The speaker closes with practical advice, lessons learned, and encouragement to build the feature using the architecture discussed.
Speaker A
So in the backend part what you're going to notice is we have all the things pretty, pretty basic.
First of all, this is the worker, if you remember the worker that I talked you through just a couple of videos back.
So the architecture that we have, this is the architecture.So this is the flush worker that we have.So as you can see, this is the flush worker.This is the interval that I'm using.And the flush worker simply goes ahead, pull the data from all the keys and just flush it.
It's a flush batch.So it pulls up all the data and just goes through the Redis pipeline and kind of dumps it into the postgres.That's all the DB transaction and all of it.So this whole part is just pulling all the data.And if you look at this part, this is where we are doing the DB transaction.
It feels really good that I don't have to write this manually and I can just use even auto suggestion.Even if I cut this out, you will notice that this is so good of agents now that even if I try to write it, await.There we go.I can just hit the tab key.It's going to do exactly same things and that's it.That's why I'm saying that you just need to know the basics of it.Anyways, I'll put this back and you'll notice.Same.Exact same.All right, Good context.And by the way, I am not on any codecs paid plan or something.It's just Pre one doing so good now.Yeah.Anyway, so this is our worker, this is our Redis.And again, as you can notice, this is Redis from IO Redis and Dragonfly speaks the Redis protocol.
So no changes in code at all.It is just this tiny URL that I changed.By the way, I can change it with Redis as well.It will still work.You can notice I have a backup of this one.
Again, I don't need a backup.This one works absolutely fine.So what I'm trying to implement is that this is a drop in replacement with higher performance.So this is what we are doing.We have the watch recent playing and hydrated as well.
So that's all.That's your Redis and then simply progress and all of this, these are typical transaction and all the Redis related stuff.So in case you have ever worked on Redis, I do have a Redis playlist, but it's not in English, sadly.If you want that in English I will rerecord that.We write a lot of code together manually in that it's on my other channel.
So this is Classic redis work.And that is it.As you can see, there is nothing much going on.We're just logging progress and continue watching which gets the data continue watching and just pulls the videos from the redis itself and just sends it.That's all.
It looks little complex, but now that you know underline behind the architecture and all of it, you realize that what this is all and you can understand it better.As you can see, this is the hash, this is the Z set, this is the string and this is the set.So we are using correct database for each of the one.And I have this one.So I can just go ahead and flush all.
So there we go.So we have flushed all the progress here and I can just keep on adding it flush to postgres.It writes in the postgres this alone project.It's not very complex but it will help you to understand all of this.The only request I would make you is since I will not be keeping these videos.These were just for the demo purpose for really long.Pull up download any of my YouTube video.Upload that on this one image kit here.Why one video less.There we go.
Now the new video came in.I don't know, it is showing up on our player.So there we go.Now it is here.It will do all the transcoding part and everything for you and works nicely.
And even if I shows you the environment example, It's a database URL.Dragonfly database is our Postgres by the way ImageKit your endpoint URL and that's all you need.You can also add ImageKit, ImageKit endpoint URL and these URL.That's all you need.So pretty fun stuff that we have built and gone through with it.
I hope.Now if one take that I would recommend you to do is try to build this.Even though you built through the agent just go through with the code base.No need to go line by line.Gone are the days.
But go ahead and try to build this and at least understand the architecture what we are trying to do.In case.In case you have any confusion, any single point that hey, I couldn't understand this or that.I am more than happy to make the video on that and cover that.This video was specifically for just one feature which is continue watching.So I didn't explicitly went into the depth of the hashings and all of it.If you want we can do a title with the hashing itself and can go more into depth of it.All right.So quite a lot.And yes, these videos takes a lot of time for preparing the explanation, the code material and a whole bunch of things.So how can I make it easy?But yes, this is one of my favorite feature and I actually manually wrote one such similar feature in the early days when the AI was not there.
And that's why I understand this architecture quite a lot.Because yes, I agree that when the first time I wrote it, I just dumped everything into the database MySQL at that time, not a good decision.But very quickly, within two days I realized oh man, this is a bad implementation.My database is getting overwhelmed, shouldn't be doing that.And that's when I realized that hey, this technique could be used.But happy that I was able to share this with directly with you so that you don't make mistake that I did in the production.And your life remains much much easier.Your life is anyways easier.The agent is writing the code.You tell them to use better architecture and better structure, better data structures and go ahead and build even better products.So that's all I'm saying.So pretty fun, pretty cool video.That is it for this one.
Let's go ahead and catch up in next such video.Don't forget to drop the comments.I'm collecting all of the comments my agent as well me to figure out what next video to record.That is it.And I'm going to catch you up in the next video.