# AWS Masterclass for Data Engineers with Project **Channel:** Darshil Parmar **Source:** https://www.youtube.com/watch?v=yvAWbbQa8eE **Transcript page:** https://www.withtranscript.ai/video/yvAWbbQa8eE ## Chapters - 0:00 — Project Overview and Learning Goals - 2:16 — Revisiting the Earlier AWS Project - 5:40 — Expanded Architecture for the New Pipeline - 8:47 — Business Problem and Project Requirements - 10:54 — Cloud Computing and AWS Market Context - 15:16 — Cloud Service Models Explained - 18:48 — AWS Fundamentals and Global Reach - 23:17 — Regions, Availability Zones, Edge Locations - 30:30 — AWS Accounts and Access Management - 34:32 — Core Data Engineering Services Overview - 38:24 — Preparing the Local Project Data - 43:33 — Creating Bronze, Silver, and Gold Buckets - 51:20 — S3 Storage Classes and Lifecycle Rules - 57:22 — Uploading Raw Data to S3 - 1:14:32 — IAM Roles for Service Access - 1:21:12 — Lambda for JSON Transformation - 1:31:42 — Glue, SNS, and Access Setup - 1:40:58 — Glue Catalog and Athena on Raw Data - 1:55:00 — Querying Raw Data with Athena - 2:00:40 — YouTube API Ingestion Lambda - 2:16:46 — Lambda JSON to Parquet Pipeline - 2:27:17 — Glue ETL from Bronze to Silver - 2:45:26 — Data Quality Check Lambda - 2:52:22 — Building Gold Analytics with Glue - 2:59:35 — Analytics Tools and AWS Data Services - 3:07:56 — Step Functions Orchestration Setup - 3:23:06 — Debugging Permissions and Pipeline Execution - 3:49:22 — Project Recap and Next Steps ## Transcript **[0:00] Speaker A:** Welcome to AWS masterclass for data engineers with project. In this video, you will learn about major services required for data engineers on AWS. Not only that, you will also get hands-on practice and you will build this amazing project. This project is about YouTube trending data pipeline. We will extract the data from real-time YouTube data API. We will have a static data set that will be ingested onto Amazon S3. We will be using medallion architecture, bronze layer, silver layer and the gold layer. Once we have our raw data added, we will explore that data using glue crawler and put the life cycle so that it goes back to glacier after some time. After that, we will clean our data into the silver layer. So we will use AWS Lambda to process our raw JSON data that is coming from YouTube data API and the static data set from the kegel. **[0:51] Speaker A:** We will also have a AWS glue running on Apache Spark that will process our CSV and JSON data. Both of this will transform our data into the park format that will be added onto the silver bucket on AWS S3 and we can explore that using glue catalog and Amazon Athena. Before putting our data onto the gold layer, we will do a data quality check. So we will write our data quality check logic into AWS Lambda and we will attach AWS SNS simple notification service to get the real-time notification of our data validation. So in case something wrong happens we will get the email about our data. After that we will transform our data from silver layer to the gold layer if everything is fine into three different categories or three different tables. tending analytics, channel analytics, category analytics that will also be added onto the Amazon S3 goal bucket and catalog so that you can explore it. **[1:51] Speaker A:** Now you can use Amazon Athena and quicksite to visualize this. Once everything is done, we will use AWS step function to orchestrate the entire thing from injection to transformation, data quality check, aggregation and final success notification. This video is your AWS masterclass for data engineering with industry grade project. So like this video if you find this helpful and without wasting time let's get started. **[2:16] Speaker A:** So let's start by understanding what we are going to do in this project. Now this is the project about YouTube data analytics and this project we have done if you have been following me from the last four to 5 years. You'll already know we have done this project previously uh around 4 years back and that project was pretty simple right? So if I show you this is also available on my YouTube channel. It's like uh 1 hour 2 hour of the project. There are two parts of it. So you can see the entire project available here. Now this project is built using old concept and built using the old AWS UI. A lot of things have changed and uh people are finding it very difficult to execute this project. And the goal of this video is not to just do this project is to give you the entire AWS masterclass. **[3:04] Speaker A:** So what we will do? we will learn about AWS concept okay while executing the project. So it usually what happens is that let's say if you want to learn AWS you will learn individual services one by one and you'll keep learning and uh you'll learn about theories you might do some hands-on practice but it will be very difficult for you to understand the entire picture and how all of these services work together. You might do one project at the end but we are going to take a different approach here right what we will do uh we will learn AWS while executing the project and there are few places where I will act like I don't know anything right and I will try to learn with you okay so it's sort of like you are learning something with someone it's like a peer-to-peer learning while I will also learn and I will **[3:51] Speaker A:** teach you whatever things I know if I let's say if I face some errors or if I don't know everything. We will try to understand all of these things by reading documentation. We will use AI to write code. We will do a lot of things. So this will be like the entire masterass where you will be able to learn and understand and also how to learn new things with me. Okay. So fully value packed video. I hope that you watch till the end so that you get most out of this video and do let me know how you like it. Okay. So let's start moving forward with our video and understand what we are going to do. Now if you know okay this was the old architecture we executed uh around 4 years back it was very simple uh we extracted data from the kegel okay about YouTube data analytics we loaded that data onto the Amazon **[4:42] Speaker A:** S3 now if you don't know about any of these services that is all okay we have notes available over here we have ppts also where I will explain you all of these different things from the basics so totally okay if you don't understand just stay with me let's go with the flow okay and everything will start making sense as we move forward. So don't worry about it. Okay. So we used to upload our data onto the this uh Amazon S3. Once that is done, we had three landing zones. So one is our landing zone where we put the raw data. Then we clean our data a little bit and put it onto the S3 and then we do the analytics on top of it. Okay, this was the architecture we created. We use AWS glue to write the transformation job for our uh in Apache Spark. **[5:27] Speaker A:** We use a AWS lambda to do the basic uh cleaning of the data uh data catalog on AWS glue to create a table on top of the files Athena to start writing SQL queries. So this was the overall overview. **[5:40] Speaker A:** This project is still available on my GitHub. If you want you can go and see that. Okay. So this is the this is what we did like four years back. Now we are going to let me just uh put this over here. Okay. Now what we are going to do this okay here we have added lot of things. Now this project was very basic. This project covers lot of different things. We will be using uh here we were just using simple uh static data available on uh kegel but now we will integrate YouTube data API onto our project and then we will load that data onto the x3. we will use same data set that we use as a kegel data set to put that data onto the S3. Okay, we will have a life cycle created. So anytime we don't need any data, we will put that back to the Amazon glacier again. **[6:34] Speaker A:** You will understand all of this. Just look at the overview. We will execute each and everything one by one. I will and I will explain you in detail what we are doing. Okay. Then we will run a blue crawler on top of it. That is basically if you want to run SQL queries on top of the existing files, you can easily do that. Okay. Then uh this is a bronze layer. So we'll use the medallion architecture to create our entire project. Bronze is our raw data. After that we have the silver layer. Here in the silver layer we will do the basic cleaning of our data. So we will have a raw JSON and then we will transform that raw JSON into the park file. uh we will read our CSV data or the JSON data from here and we will transform that into park file and then we will put that onto the silver layer of our bucket. **[7:17] Speaker A:** Okay. Then we will do the glue catalog so that we can start running SQL queries. After that we will do the data quality check to make sure all of the data that is uh coming to us okay are in the right format that there are no nulls based on the validation we will put all of these data into the right quality. So we will also understand about the data quality. Okay. We will also have a uh notification system created so that let's say our uh data quality does not match or there are any issues or it is successful. We will get the email notification on our uh mail that okay this uh particular pipeline has failed. This is the issue and we'll get all of the notification. After that we will do the Apache Spark ETL where we will uh build build the aggregation layer where you can analyze all of these different things and we'll create a catalog for that also and then we can analyze **[8:08] Speaker A:** this data either on Athena or Quicksite. Now if you don't understand any of this service totally okay don't worry I will explain you one by one. This is the video about AWS masterclass. If you want to learn about AWS from the scratch this is that video okay but we will use advanced level industriate project to understand that. So you learn both of the things in just one video. Okay. And we will also use the AWS step function. It is like Apache Airflow. If you want to orchestrate all of these entire thing uh into a simple pipeline and run it together, we will do that. Okay. So this is the uh entire architecture that we will be executing. **[8:47] Speaker A:** Now this is the project overview. And usually uh when you do any project you need to have a problem statement like why are you doing this particular problem on the first place right uh every data project has a goal uh and the pro uh project's goals usually include solving a business problem it can be like you know finding the bottlenecks uh generating revenue for the businesses there are a lot of different things available so we will start with a assumption or hypothetical scenario that there is a client okay who has a problem okay so So the requirements are like this. Okay. They are launching a datadriven campaign. Okay. And they have chose YouTube. Let's say they have a product that they want to sell and run a marketing campaign online. And they chose that YouTube is the best platform for right uh right now. Okay. **[9:35] Speaker A:** And they want to answer these questions such as how to categorize videos based on the comments and statistics. What are the factors affect how YouTube videos will be popular or not? Okay. So these are the some initial questions they have that okay I want to run a campaign on YouTube and these are the things that I want to find before running the campaign. Why YouTube? Again YouTube uh there are like top three most visited website. One is the Google, YouTube and the Facebook. Okay YouTube comes second. Uh it owns lot of video traffic also. So YouTube is the best platform if you want to run some ads and based on the client requirement they are choosing YouTube. Okay. Now these are some of the goals of the project. Again we have discussed this previously also but let's revise it. We want the data injection automatically happening. **[10:19] Speaker A:** We want a data lake right now so that we can build architecture. Again in this scenario we will build the medallion architecture. Uh in previous one we just build a simple data lake. This will be the medallion architecture. I'll explain you what medallion architecture is. Don't worry. Okay. Then we will be using AWS cloud. Uh we need ETL scalability and reporting. So these are the basic criteria. Okay. So this is the overview of what we'll be doing from the project side and while we do the project we will also understand about everything around AWS. So what are the services available? what is important for data engineers and all of the other things. Okay. **[10:54] Speaker A:** So now let's start that right let's start by understanding AWS masterclass. Now if you want to take notes take out pen and paper if you don't we also have notes available here okay uh onto datavia.com/arn on the cloud and infrastructure you can go here and we'll be uh reading some of the things from here also we will also refer uh PPT that I've created so you can go back and forth of the different things. Okay. Now uh let's discuss this also right this is the AWS studio so we'll be talking about AWS but usually uh people have confusion about should I learn AWS GCP or ashure and this is the common uh question everyone has let's say you decided to learn cloud okay and you have three uh options available one is AWS second is GCP and azure now within few minutes okay if you start exploring you will understand there are hundreds of **[11:48] Speaker A:** services available on each of the cloud computing platform and you start getting confused that okay which one should I learn first should I learn Azure should I learn AWS should I learn GCP this is the common confusion everyone has right you'll be thinking okay if I learn Azure and uh if if I need to you know for the job interview if I need to prepare for AWS then I will waste lot of my time learning Azure this is the common problem everyone faces but here's the thing okay this is the cloud market okay in 2025 AWS owns 32% % of the market share. Azure 23% and GCP 12%. Okay. Most of the job listings are around AWS okay 60%. Worldwide most of the startups starts with the AWS uh AWS has been market leader from last I don't know 5 to 10 years. Uh if you check the Gartner quadrant uh Gartner uh cloud magic quadrant okay the this is the company that ranks different cloud providers. **[12:45] Speaker A:** And if you see uh let's say in 2025 uh if I zoom this okay AWS, Google or Microsoft are the leaders and Oracle is coming. Okay. So this is the uh top cloud platforms. Now what I mean is if you learn any of the cloud from the three options you are in the safe zone. Okay. Because there are more cloud available like Alibaba, IBM, Tens, Huawei cloud. These are based on the company use case. So if you're applying for a company and if they are using this cloud platform then it makes sense to learn them but usually if you learn all any of the cloud from here you are in a uh safe zone. Now the thing is you don't have to learn all of the cloud platform. That's the number one confusion people have like do I have to learn three of them? No, you just start with one and master one cloud first. **[13:35] Speaker A:** Once you have mastered it will be much more easier to transition into any other cloud because all of these cloud platform works on the same foundation. For example, object storage, right? Object storage is a way to store any of the data onto the cloud. On AWS, we have S3. On GCP, we have GCS. And on Azure, we have ADLS Gen 2. Okay. So, this is similar services. They work on a similar concept or the foundation. The service name, UI, all of will be different. Maybe the feature pricing will be different. But the core concept that this is an object storage. Okay. Is the same same goes with the data warehouse, right? AWS has red shift, GCP has bigquery, Azure has synapse analytics does the same work. Okay, this might be a little bit better from the cost wise, UI wise, the feature wise, but they do the similar job. **[14:29] Speaker A:** Same goes with a serverless ETL, manage spark, streaming, orchestration, all of these different categories, right? Capabilities or categories you want to call. Each of the cloud computing platform has a service available for that. And based on the service okay if you learn this it will be very easy for you to transfer here and here because you understand okay object storage a available on AWS S3 is same as GCS right on GCP so you can always have a understanding the foundation understanding about the service and it will be very easy for you to transition your career okay so simple answer if you're learning as right now if you're watching this video that means you want to learn about AWS so this is a good place to start if you're just learning Okay. So we'll go forward and start executing and understand about the AWS cloud platform. **[15:16] Speaker A:** Okay. So AWS masterclass. Now again if you already know some of this you can revise it. You can skip it also. But we will start with the introduction of the cloud computing before you learn about AWS or GCP or Azure. You need to understand what cloud computing is. So we will quickly skim over you. Uh so we will quickly look at it. We will not go in depth about all of these things because you want to learn about AWS and execute our project but we will start by understanding all of this. So cloud computing is on demand delivery of IT resources over the internet but with pay as you go pricing instead of buying and maintaining physical servers data centers. You access the technology on as need basis from the cloud provider. simple okay uh let's say if I want to host a website I need servers I need RAM storage capability networking power supply all of these different things I will need either I can build it or I can rent **[16:07] Speaker A:** it so cloud is basically someone has already built all of these for you they give you this UI you can always go on the UI and access those things differently so for example if I have to show you example here itself okay let's say we have EC2 EC2 is a elastic compute cloud Okay, it is simple like a virtual machine. So a machine is running somewhere. It's like my laptop. Okay, you can do a lot of things on your laptop. You can you know uh play games, you can host website, you can run something. There are browse internet. All of these different things you can do. Same as if you create a EC2 machine, it's like a virtual computer you are creating. Okay. And you can host different things there. If you can even if you want to play games, you can run a window machine and you can do that. **[16:52] Speaker A:** Okay. So these are this is where our data with the website. So this website that you see is running onto one of the EC2 machine here. So this is our staging server. This is our prod server. Okay. So this is our production servers are running and this is our terraform and labs that we are building. Okay. So this is where our machine is running. So it's pretty simple. Okay. We create a machine. We host all of the our things and put it over here. So this was one example of cloud computing that we are renting server from them so that we can run something. Okay. Uh let me do this. Go here. Now cloud computing platforms provide services to us in three different forms. One is infrastructure as a services. They give you infrastructure as a rent. Second is a platform as a services. They give you some kind of platform so that you can do your work and third is a software as a services where you can uh get access to different software and do the work. **[17:45] Speaker A:** For example, infrastructure instead of building again the uh your own computer or own machine like uh by buying RAM, storage, power supply, all of the things you can rent infrastructure from them. Okay. And the renting basically means if I show you uh cloud data centers you will see all of these are the data centers uh create uh created or developed by all of these cloud platform and let's say if you uh hypothetically if you let's say uh run one EC2 machine so from this rack there will be a small place assigned to you automatically via software they have written all of the logic and you will be able to access the small machine based on the configuration and do your work. So basically you are accessing their infrastructure via uh internet and then uh doing your work. So instead of building all of these you know systems by yourself you are uh getting the infrastructure as a service okay from them. **[18:48] Speaker A:** Uh let me close this. Let's show. Yeah. So this is infrastructure like EC2, S3, VPC, Azure VM. All all of this comes into the infrastructure like hardware things. Then we have the platform as a services. So platform for building and deploying application. Now there are a lot of services where I don't want to worry about the infrastructure. I just want to care about my code and running it. So by in the case of EC2 you have to manage everything by yourself like uh hosting your website making sure the number of uh amount of storage that you need ramp you need you have to put all of this into into the configuration but there are also cases that you don't want to worry about you know uh uh all of these like management of the infrastructure side you can also go with the platform as a services **[19:34] Speaker A:** so there are like examples is Google app engine elastic beanto AWS lambda is also also a platform as a service We will understand about that. So that you don't worry about the compute or resources. You just worry about your code. And then we have the software as a services. This is this is basically a software product provided by the company. So you can access it via web browser. You manage your data and preferences. For example, you use Gmail, Slack, Salesforce, Dropbox, all of the softwarees that you use online. This is called as a software as a services. People call it SAS. Like if you are into tech or in the if you are quite active on Twitter you'll know like oh I want to build a SAS startup right SAS is basically a software as a service you want to give a service via software to any client okay so this is the basic of cloud computing so you can again uh what you manage uh in the there are different uh responsibility for example on premise you have to manage **[20:28] Speaker A:** everything from application data runtime middleware all of these different things you have to manage but on infrastructure you will be managing application to OS and the main hardware level such as networking, storage, server, virtualization will be managed by the provider. In the platform, you will be managing your application and the data and all of the other things such as runtime, operating system, server, storage will be managed by the system. And the SAS, they will manage everything. You don't have to wor like you don't manage uh like Gmail by yourself, right? All you do is use Gmail or you don't manage let's say Dropbox or Slack you just use that right so in the software as a service you just use the application you don't manage anything so again three different services provided by the cloud again the foundation if you understand all of this things will start clearing up okay so infrastructure platform as a service software as a service now let's talk about AWS like what does AWS provides **[21:26] Speaker A:** us now AWS is the world's most comprehensive and broadly adopted cloud cloud platform. Most of the startup that you see are on AWS. So if you are someone who want to work in startup, AWS is a good place to learn. If you are someone who wants to maybe work in a banks, there are a lot of banks who use Azure also. If you are someone who wants to work in corporate, people use different so clouds, people use different cloud based on the uh companies use different cloud, right? Uh based on their requirements. So whenever you applying for a job the best is you research about the company. Okay. Let's say I want to work in tread or zomato or let's say any of the company that you are interested in. First look at what cloud computing platform they are using on the first place. Let's say zomato is using AWS. **[22:13] Speaker A:** So it makes sense to learn AWS because it you will have a higher chance of getting hired. Okay. So uh AWS was launched in uh 2006 started with S3 and EC2. Okay. It has now 200 plus fully managed services available. Uh serves millions of customer like from a startups to fortune 500. Market leader with 31% crowd market share. Okay. Ahead of Azure and GCP. It also has a pay as you go. So you only use or only pay for the things that you use rather than paying for entire thing. And it has a global infrastructure. We will understand about this part right now. Okay. Uh so it works on the main scalability. You can easily scale up scale down reliable. It uh runs across multiple availability zone cost effective and global reach. Now this we already saw okay this is the Gartner quadrant where AWS is a market leader but again Google and Microsoft are also there in the same quadrant of the leaders. **[23:10] Speaker A:** So of course if you want to stay safe in the cloud market you can learn any of one and you are good you're good. Okay. **[23:17] Speaker A:** Now let's understand about the AWS global infrastructure. How does the AW again I told you right this video is about the AWS so I want to teach you AWS but I also want to teach you how to execute project via hands-on so this is not just just a theory we will also learn theory but also uh hands-on thing okay so what is Amazon web services we already know okay world's most comprehensive cloud platform has 200 plus uh services available pay as you go model market leader we already know this. Okay. Other than that, it has 200 plus services available, 32 regions, 102 availability zones. Again, it keeps changing and 450 plus edge location. If you want to get the latest data, you can always go to the doc. Now, let's talk about what is AWS region. Okay, there are three things. One is the region, second is the availability zones and third is the edge location to understand how AWS is built. **[24:16] Speaker A:** Now the region okay AWS region is a physical geographical location for example US east AP south okay this is like a physical area like you know in India there might be inside the Mumbai region or they have given a this sort of name like AP south one US east one so these are different region located so each region is fully isolated from the other region so there is a one data center here there's one data center here these are different from each other like there is there will be one in India maybe into some Indonesia I don't know okay there will be one there will be one in India Africa maybe uh in US Canada all of these different region okay in one country also there can be multiple regions so in India it can be like in Mumbai or in Bangalore okay it can be a different regions available so based on that they **[25:10] Speaker A:** create their all of the physical resources there so choose region based on the compliance latency service available and pricing. Not all services are available in every region. So if I go to AWS console and if I click here, okay, you will see there are different region available uh to me. Okay, there's United States. So in US we have like these many regions available. Then inside the Asia we have these many region available. Inside the Canada we have these many region available. So these are the different region. This is just the country name but these are the regions. Okay, US East one, two, this is their way of writing it. But you can also look at the by by the state wise. Okay, so these are the different regions available on AWS. Okay, got the region. What the regions are basically where they actually create the data center. **[26:01] Speaker A:** Within the region, there are different availability zones. Okay, so one or more discrete data centers within the region. So uh there will be one availability zone, second availability zone, third availability zone. So each a has independent power cooling and network connected via high bandwidth low latency private links deploy across multiple a for high availability and fault tolerance. So what this means is that let's say uh uh I am inside here. Okay. So this is my let's say uh entire AWS uh AWS and I am inside let's say uh Mumbai region. Okay, this is my Mumbai region. Now in the Mumbai region, I will have different availability zones. Okay, this is A1, A2, A3, A1, A2, A3. This basically means these are the different data center. Okay. And these are very separate from each other. Right? So inside the Mumbai region the all of these AES will be hosted nearby location right so this will be one data center this will be one sec data center this will be one data center and these are separate **[27:23] Speaker A:** but within one region so if you want to deploy let's say high available application onto Mumbai region what you will do you will deploy let's say on a1 it deploys on a1 you can replicate the same data over here also so in case if this a goes down because of fire uh earthquake or whatever happens this will be always available to you. So within the same region I can have my backup available. So this is like a three different data center inside one region. This is what we call as a availability zone on AWS. Okay understood we have region which is basically a different places available inside that region we have multiple data centers right availability zone one this is one data center second data center third data center data center again I showed you uh this looks like this. So this will be one place and then like this there will be a second place at different location within that region only and the third place okay like that this is your availability zone available on AWS okay and **[28:21] Speaker A:** the third is edge location and now let's say you are watching Netflix okay and the Netflix is uh releasing one uh new web series and it is hosting on their server okay let's say on the US server Netflix is hosted now if you're trying to access that uh video from India it will take very long for that it to travel and give you the uh entire stream data on your network. Okay, you will you will face delays or it will keep buffering if you are accessing data from here. So that is to overcome that okay all of the content delivery network you can you call it as a CDN okay used by cloudfront there are different edge location okay we might not know where the edge locations are on AWS you can find it on the website but there are different edge location near to your location so let's say you are staying in India and there is a edge location near Mumbai so what will happen AWS **[29:13] Speaker A:** in the back end okay will cache all of the data of the streaming onto the nearest edge location available to you and whenever you are visiting Netflix okay when you are trying to access any data it will directly route traffic to the nearest edge location rather than going to the main server. So this is the concept of edge location that to give you the content very faster. Okay. So that uh the experience is much better. Okay. So cache content closer to end user for the low latency 450 plus point of presence available worldwide used by cloud fund route 53 the DNS AWS global etc also uses edge location. So this is basically a for the caching mechanism. So if you have a software background it will be quite easy for you to understand but in the simple way it is the way you can get the content faster because it is stored near your location. **[30:04] Speaker A:** you don't have to get data from the US or wherever the actual servers are hosted. Okay. So three things uh the three things we understood we have AWS region inside the region we have availability zone and inside the zone uh okay inside the so we have AWS region inside that we have the availability zone basically different servers and then we have edge location spread across the entire world that gives you content faster. **[30:30] Speaker A:** Okay, now this was mainly about how the AWS is managing the data center and everything. Now, how does the AWS account work, right? When you create AWS account here on the this is what is the structure of this account because each cloud provider has their own structure. Okay, so AWS has its own structure also. Let's look at that. Okay, so AWS account. Okay, this is your root user. Usually when you create a new account, you are creating account with the root user that has full access or you can use M MFA u that has the full access to all of the services from billing to anything that you want to do. You can use MFA for this and you should avoid using root access for your daily task. Okay, so this is for the root user. Then we have IM users and role. Again, we will understand about all of these things. **[31:23] Speaker A:** Just stay with me, okay? I'm just giving you overview of everything. So when we start understanding these things one by one, it will be quite easier for you. Okay. So then we have IM users and role. It is used for the access control. If you want to give access to some user or some service, you can do that. We have organization AWS organization is it is one service used to manage account across the AWS. So let's say you are running a big company of let's say 500 people and if you want to give access to uh AWS account access to all of these 500 people you can use AWS organization to manage all of these account then billing and cost management and free tier you usually get 12 month free and there are services that are always free. Okay. So this is the architecture what it looks like. **[32:09] Speaker A:** Okay. You have AWS cloud. Inside the cloud you have AWS organization. Then you have the management account where you can you admin can grant access to user policies groups and manage billing. And then there can be multiple accounts such as student account 1 2 3 from like that it can go to the multiple things. So pretty simple. AWS cloud inside the organization we have multiple one is your admin which is your root account. Okay. And then there are sub accounts that you can create if you want. Okay. So this was all about the core of AWS. Now what we will do, we will start understanding about the services. Now before we understand about services, I will explain you or I will show you this from overview. So again if you don't have AWS account, it is pretty simple to create. All you have to do is just go to aws.mazon.com create account. **[32:56] Speaker A:** Uh you can just fill the information here. Okay. And create your account. **[33:05] Speaker A:** uh this will give you $100 in credits and uh if you complete various activities I think uh this will give you more $100. So uh they might ask you for credit card initially for the verification but they will not charge you. In fact they will give you credits so that you can use AWS and various services. So anything we do in this project if you have a free trial or the credits available you will not be charged. So don't worry about it. Okay. So uh creating account is pretty simple. I don't want to spend time explaining you because it is some basic stuff. You just have to provide your uh details and you'll be able to create your account. Once you create your account, you will be created with a fresh AWS. Right now, I have visited a lot of services. My uh cost are running. **[33:46] Speaker A:** So, you will be able to see all of the uh cost that we have here. But this is what the AWS console looks like. And if I go to AWS organization or you can also do that with me if you want but I don't think uh you will have organization available but it looks something like this. So inside the organization this is my root account. Okay. And inside the root account I have like let's say students account available here that I created for the test. I have the security and the sandbox is available. So this is where you can manage accounts. Then we have IM again uh identity access management. Again, we have a proper section for this right now uh uh when we do our hands-on practice, but I'll just show you the overview. This is where you can manage users, roles and all of the other things. **[34:32] Speaker A:** Okay, let's continue with our video and let's go to the next thing so that uh we move forward. Now, till now I think you understood the core of cloud computing. We understood the core of AWS, how it is built. Now we will start understanding about the different services required for uh data engineering task. Okay. We will be mainly focused on data engineering services only. Uh on AWS if you go you will click here you will see there are hundreds of services available into each of the category for compute containers customer support databases like these are the databases supported by AWS. These are developer tools. H end user computing available. Front end mobile you also have services for there. Uh game development you have service for that. Internet of things. Uh machine learning you have services for machine learning. Uh content delivery that we looked about the cloudfront uh route 53. **[35:27] Speaker A:** All of these different services are available based on the use case. Now as a data engineer we don't need to learn all of these services. Okay. We just need to learn few services that will help you to do most of the task. And these are the services that we will start with. We will start with AWS S3. Okay. Simple storage service. This is a scalable object storage. So, Amazon S3. Okay. Core concept. Let's understand. It is an object storage with 99. It is an object storage with 99.999 119. Okay. Durability. Basically anything you store on AWS the chances of it being available or it is being stored there is this much high okay 99.119 so that uh it gives you very the chances of you losing your data is very less store unlimited data okay object you can uh store unlimited data at one time of the upload you can maximum upload up to the 5 TB each of that objects okay so if you Let's say video file of **[36:32] Speaker A:** 6TB it will not go directly okay but if you have up to 5 TB very very less that you will have video file of 5TP or any of the data okay because most of the data are divided into the different segment and then you upload it buckets are region specific names are globally unique we will understand that when we do the hands-on key value store is equal to uh it is a key value store so key is a object name and value is a metadata okay I'll also explain you this part okay just right Now understand the concept. It has versioning, encryption, life cycle, replication. It also has a different storage classes. For example, standard storage class. Uh you usually use if you access the frequently accessed data. Infrequent access. So there are data that you might not be accessing. You can move it to inf infrequent access so that uh the storage cost is less here. **[37:25] Speaker A:** One zone access you have single a lower cost. If you just want to put your data onto the one zone, glacier instant, glacier deep. Okay, this is basically if you want to archive your data. Instant you can have a instant retrieval. Uh glacier deep lowest cost uh for storage but the retrieval cost lot of money and intelligent it will do based on its understanding. Okay, this was the core concept. Again if you want to go deep dive uh I have created the detailed blog here also. So you can go deep here. Uh you can read about it. But I will explain you this concept onto the uh this by via hands. Okay. And via hands basically means we will do our project. So now we will going to we are going to start with our AWS project that we have simple uh technique. We will understand about the basics of services. **[38:13] Speaker A:** We will understand what all of these mean. Okay. Why? Why doing the hands-on practice? So that you also understand what it is while doing the hands-on practice. And we also have a project available. **[38:24] Speaker A:** Okay, before we move forward, uh let's just set up our project first. So if you have any code editor available, uh you can use that. Uh if you don't have then uh you can install the VS code from online and what I'll do, I'll open a folder here and I have a folder created into my documents projects. Okay, I'll open this folder and inside this folder uh I will create a new folder. I will put all of my projects here. So uh let's say the project name will be YouTube data pipeline uh 2026. Okay, I'll just keep the project name. Uh you can do the same with me and uh work together. So let me close this. Okay. Now inside this uh we will create now in Okay. Now inside this we will create one folder. We will uh not inside here we will create one folder as a data. **[39:20] Speaker A:** Okay. Uh and the data folder we will store our data from the kegle. So again let me go back to the architecture here. We know that our data sources are YouTube API and the kegle data set that we have available. These are the two data sources. One is our static data set and second is our actual YouTube API that we will be integrating. So first we will start with the static data set. So for that uh let me just get the data. Uh what you can do uh you can go to uh I'll put the link below also about this. Uh you can go here on kegel.com and you can go to the where is that? Yeah over here you can data explorer not here well yeah over here only you can click onto the download and download the data set as a zip file and it will download the entire data. **[40:14] Speaker A:** Okay, you can do it programmatically also but this is uh much easier. You can just download the data as it is and it will download the data something like this. Uh let me just uh see if I have the data downloaded. You'll have to register it. Just create your account. Once you have data register uh I'll show you how the data actually looks. Let me just paste the data itself uh onto here because where is that? Yeah. Data. I'll do paste. Okay. So this is uh what the data looks uh I'll just show you inside uh where is that project data. Okay. So this is what the data looks like. Okay. It has JSON data and the CSV data available and we'll explore this data but it has uh different countries of categories and the videos available. Let's look at this data. Okay. Uh you can download this and again move forward. **[41:10] Speaker A:** So uh let's see if uh okay h so we have two data available and also it is available on kegel if you want to have a context here. So uh it has content okay data set includes several months and counting of data on the daily trending videos on YouTube. Data is included for US uh GBDCA. These are different regions available such as USA, Great Britain, Germany, Canada, France respectively with up to 200 listing trending videos per day. Okay. So these are the data that includes. So uh they added the uh new version and added the Russia, MX, Mexico, South Korea, Japan and India also in this data set over the same time period. So there is one CSV file available. Okay. Uh here uh this CSV file includes all of the trending data such as you have the video ID, trending date, title, channel ID, category ID, publish, time, tags, views, like, dislike. **[42:09] Speaker A:** All of these are available. If you want to see, you can also open it onto the Excel or if you are using Mac, you can open it onto the numbers to explore this data and how it looks. It's always the best practices if you are doing a data engineering project, you should know what the data looks like. Then only you can write the transformation code or the cleaning code uh by understanding the data. Okay. So let me just wait for this to open. **[42:47] Speaker A:** Okay. Still it is opening. Uh let's move forward and understand the JSON data also. So the JSON data is also available. So this is what the JSON data looks like. Over here you will see we have different categories available such as film, animation, autos, vehicle, music, pets, animal, sports and all of the other things are available. You can use uh this uh category ID okay to connect with the ID of this and retrieve the category of that particular video. So that you have the JSON data available with the category and you have this uh different uh information available about the description uh rating, comments, thumbnail links, likes, dislike, views, all of these information are available. Okay, so we have the trending videos available and the JSON data available. This is our static data. **[43:33] Speaker A:** So make sure you have your data available over here. So we have our data available on the local storage and now we want to load this data onto the Amazon S3. Okay, this will be our bronze layer and again as we discussed we will be using a medallion architecture and this is what it looks like. Okay, pretty simple uh this is just a format of how you want to organize your data sort of like a standard practice used in the real world. So you have a data available okay and you want to serve this data. So what you do is you divide your data into three different categories. One is your bronze layer, second is a silver layer and third is a gold layer. Bronze is your raw data as it is ingested onto the platform. Okay? You don't do anything whenever you take your data from any source. **[44:16] Speaker A:** You just put that data as it is and you do not touch this data because anytime you want to roll back or want to get access to a specific value, you can always get the raw data. Okay. Then comes the silver data. Okay. The silver layer you can filter, clean and augment this data. So let's say if you want to remove null values, you want to rename something of the column, you want to drop something, you do all of these into the silver. So you read data from the bronze layer and put that onto the silver layer. And then we have the gold level. So you use the clean data to aggregate data onto some kind of level, right? So if I want to aggregate my data based on date or the region, whatever it is, right? I can based on the business uh uh use case or the goals I can aggregate my data onto the some level and then give the final metrics for the use cases. **[45:10] Speaker A:** So this is simple medallion architecture. Okay. Uh you can uh refer again you can just search on Google also we also have a notes available here. So if you just do just uh medallion architecture okay you will find about the medallion architecture. I also have a video on in my course available for that. You can get the details about it if you want to go in Apache Spark. Okay. So, uh this is about that uh everything about the middle architecture. So, what I want to do, I have my data available and I want to put that onto Amazon S3. For that, I will have to create S3 bucket. So, let's start doing that. Okay. And I also keep this open so that uh we'll know. Now AWS just go to the console and write AWS S3 or you can just write S3. If you click here in your case if it is a new AWS account uh it might not show anything uh anything. **[46:05] Speaker A:** I mean this will be uh completely uh empty. All you have to do first look at the UI. Okay. If you're looking at looking at for the first time just see what are the things available here. Okay. Just get used to the UI. You have buckets, you have access management, something available, some storage things available, general purpose directory, all of these things are available. Totally fine. I don't want to know about it. What I want to do, I want to create a bucket. Okay, again uh to create a bucket, it is a region specific and names are globally unique. What I mean by that is that I click on the create bucket general purpose bucket. Uh global name space basically uh they added account regional name space also. Now but we'll go with the global name space. So let's say if I do the test bucket okay and if I try to create the bucket with the test chances that this bucket name already exists that means somewhere in the AWS universe okay someone created the exactly **[47:02] Speaker A:** same bucket test and now I can't use it. So let's say if I do test til there are high chances that I might be able to create because the combination of this is quite right. Let's try it. Okay. Again fail to create bucket is already taken. Okay. So this bucket name is also taken. So in our case we will just create a unique bucket as bronze for YT data pipeline bronze US. Uh let me see uh I am into AP South one. Okay. I we will use a AP south one dev. Okay. So this is my bucket name. Now if I use this bucket name, you won't be able to use this bucket name. Uh right now we are working in the Mumbai region. As per your location, you can change the region here itself right now only. So that you only work on let's say if you're working from US or any region that you're working on that is nearest to your location, use that. **[48:03] Speaker A:** Okay. So this is my bucket name. I'll use that choose bucket. Uh I don't want to choose any settings. This is fine. This is fine. Everything is fine here. Okay. And I will click onto the create bucket. Okay. So my bucket YT data pip and bronze AP South1 dev is created. What I will do? I will copy this and I will paste it into uh let's say **[48:30] Speaker A:** uh I'll create a new folder inside the this one. dot this inside this new folder and I will give the name as scripts. Okay. And inside the script uh I'll create a new file and I will give the file name as uh let's say information md. I'll give the markdown file and I'll give uh let's say bronze bucket name and I'll store it so that anytime that I want to copy the data okay I can just come here and copy it rather than going back and forth from the this so bronze bucket name is done okay now just like this bronze bucket that we created okay we will create a silver bucket and the gold bucket right now only so that we don't have to come again and create it. So, you can copy this as it is. Okay. Uh to make it unique, again, make sure to add your name or some kind of flavor into it for a pattern. **[49:32] Speaker A:** Okay? And you'll be able to do that. Create the bucket. Click on this. I'll give uh instead of bronze, we will keep silver. Okay? Copy this and create the bucket. Uh then, okay. I think this is silver. This is silver. Did I make a typo onto this bronze? B R O N bronze. Okay, I think it's fine. This is fine. **[50:09] Speaker A:** No, this is my bucket. Wait. Okay. And then we have the gold bucket name as just copy this create a new bucket put it update the gold and create it. Okay. So now we have three bucket created for us for based on the architecture we have here. Okay. bronze layer, silver layer bucket, this one. Okay. And gold layer bucket that is available over here. So we have the three bucket created for our use case. We can create one more bucket. Last bucket that I want to create is to store all of the scripts. Okay. All of the glue script that we'll be writing will be here. So we will do uh YT data pipeline script. Okay. Uh and create the bucket. again add your own unique flavor behind this so that uh it can create a script bucket. Okay, this is done. Uh we have all of these things are available. **[51:20] Speaker A:** Now when we were learning about this which was huh AWS S3 we understood about the storage classes okay and the storage classes and the this versioning encryption replication and life cycle. So versioning is basically if you want to keep the multiple version of the particular data. So uh if I want to if I you know keep updating the data and if I also want to store the different versions of that you can enable the versioning when you go here uh uh bucket versioning there's an option available okay right now it is disabled but if I enable inside the properties you will be able to track the different version of the same file whatever the changes that you make okay this is versioning then we have the encryption uh encryption is also available Uh I think inside the where is that I think permission **[52:21] Speaker A:** uh where is that access control ownership bucket policy bucket access management replication properties. Yeah over here only inside the properties itself. So we use the default uh AWS manage S3 keys. Okay. S SS S3. But there are different ways you can encrypt your message. Again uh in your case we might not might don't want to do again in this case we don't want to worry about the encryption but when you are going onto the you know higher and higher level of the uh corporate and if they have the sensitive data they might use their own S3 uh encryption key or own manage key in order to encrypt the data but right now we can use the AWS S3 then we have the life cycle okay and the life cycle I want to understand this one or make you understand this one. Okay. So on the bronze layer, okay, we will be storing a lot of raw data as it is. **[53:17] Speaker A:** What we want to do, we don't want to store this data forever. Okay. Because raw data uh uh we use it for the silver and then we do the gold and after sometimes we might not need old data like 2024 data we might not need right now. Okay. So what we want to do? We want to transfer this data onto the Amazon S3 Glacier. What is S3 Glacier? Uh let me do this. Okay, we share. Okay, now we already understood about the uh different classes of S3 over here which was standard, infrequent. Uh we have one zone, glacier, glacier, deep. Okay. So there are different classes of storage available onto AWS. One is a standard. Okay. And the cost for that per month is 0.023 uh cents. It is already used for hot data or frequently query data. standard whatever data that you store you can always use it uh based on your demand. **[54:15] Speaker A:** Then we have the infrequent access. So there are some data that you might not want to use uh after a few months. Okay. So data access monthly or less. So maybe I want to use data for the last 6 months but I might not use it right now. So I can put this onto this standard. Then we have the glacier instant retrieval. The cost is very less okay but you can retrieve it very faster. and the glacier deep archive the cost is very very very less for the storage this is used for the long compliance so let's say you can store last 50 years of data but tomorrow the audit comes to the company like oh we want to review all of your data from the last 50 years because you have you might have created some fraud or you might be involved into some malicious activity we want to review all of **[55:00] Speaker A:** your data that is the reason you have to store your data into deep archive for the compliance reasons okay usually banks and all of the other things do this. So in our case we have put that into our architecture also. Over here we have the life cycle that puts our data onto the Amazon S3 Glacier. So we'll do that also. Okay. So all you have to do is go to the management uh of your bronze. So this is my script. Okay. go to the bronze uh let me yeah over here uh AP South one uh inside management and click onto the create life cycle this is what we were looking at here the life cycle auto transition between the storage classes I want to create a life cycle enter the rule name so rule name let's say archive old uh archive old raw of data. **[55:58] Speaker A:** Okay. Uh prefix uh let's say we will do YouTube slash. Okay. Anything uh data related to YouTube I want to archive it. Uh then there are different rules such as transition current version to the uh different object classes transition non-current version expire current version per delete something delete expire object. I will click on to this. So uh so this is the standard intelligent. So what I want to do I want to click onto the glacial flexible retrieval. Okay for formula glacier and after 90 days okay I'll create this rule I can add one more if I want to do deep archive but I'll remove it. So what I will do I want to transition my current data okay over here inside glacier flexible retrieval after 90 days. Okay and I will create this rule. Okay let me just acknowledge this. I'll create the rule. So after 90 days okay any raw data that I upload here inside the / YouTube folder all of these will go to the uh this one which is this instant retrieval so that I save the cost okay and **[57:11] Speaker A:** I can uh retrieve the data for the longer without incurring lot of cost. So this was the quick masterass of uh Amazon S3 that we understood. **[57:22] Speaker A:** Okay. Now what is the next step for the project? Okay, we understood about the S3 and most of the things we looked here also the core concept how what this is and storage classes and how to actually use that the life cycle to uh go between the storage classes. So now we have our data available onto the data. We will write a small script for this a simple bash script uh AWS_copy.sh. Okay, inside this we will have something like this aw3 copy. Okay. And the file name, let's say the file name is uh CA videos I **[58:05] Speaker A:** videos dot CSV. Okay. Then S3 / SL uh I want to see my bucket name which is this one. Uh this is where I want to store slash YouTube okay slash raw statistics uh this is **[58:30] Speaker A:** raw statistics slash region is equal to this is ca okay and slash so this is where I want to store this now now let me explain you this also uh if you are very new so uh on AWS We have AWS CLA. AWS CLA. This you might have to install. Not here. Okay. AWS CLA. It's pretty simple. Uh you will find a download option somewhere here. Uh where is that? AWS CLI. Just do the download maybe on inst uh onto Mac or Windows. This is the CLI version. Uh this is you can uh do on Linux like this I think. Wait uh let me do Windows uh you can download it from here the installation or if you're on Mac you can download uh from here uh this is the package available also GUI installer if you want to do that okay basically if you install this uh if you go to terminal I can do AWS help and it will give me the entire command uh for this okay let me I'll have to show you the setup also **[59:38] Speaker A:** wait let me uh so if you want to understand how to install AW W CLI on your local PC. Just go to chart JP and write how to install AWS CLI in my local uh Mac. Okay. And this will give you the entire instructions on how you can do it. You can run this script. You can install this entire thing you can do. And the main thing after you install, you can just write AWS. And if it gives you something like this means AWS is there. You have to do AWS configure command. Uh if you click on as configure you have to provide your access key, secret key, region name and simple okay you have to provide this where do you get this? Uh for that we will have to look at IM. Okay, let me open the IM. Now you can also do this with me. **[1:00:26] Speaker A:** So what we are doing is that uh simple right now our data okay data is available on data is on local PC okay now I have multiple options if I want to upload this data onto the S3 one is write Python script okay one option is **[1:00:53] Speaker A:** one option is I can write I can directly upload upload this data upload data directly okay or I can write bash script / cli okay I can do this these are three option and then I want to upload this data onto the S3 bucket okay this will go from here these are the options available here here it and go here here like this sort of like this okay so I can upload my data three different ways again there are multiple ways but these are the Python script direct upload data if I want to do I can just do click on to upload and dump all of this data as it is but I want to store my data into proper structure that is inside the YouTube folder I have created raw data and for each region I want to store my data for the partition basis okay And the concept of key value. **[1:01:57] Speaker A:** Okay, the this we understood the key is my object name and the value is the actual data available. So when you say AWS object S3, okay, object basically means this is quite tricky to understand but when you upload any data over here, okay, it is considered as a object. An object consists of two things. One is a key and one is a value. uh that we understood here. One is a key is object name and one is a value. What is key? Key is this entire thing. Okay. To reach it might look like a file format which is like SL slash but is not of actual file format. Is a object. Object consist of two things. Inside the object you have two things available. This is the key. Okay. Where the objects are actually stored and this is the actual data that is a value. **[1:02:48] Speaker A:** Okay. So when we do that when we do upload the data you will understand this. So for uploading data by your local PC to AWS, what you have to do is you have to go onto the user on IM and click onto the create user and give the username as AWS uh CLI local something like this. Okay. And I I don't want to provide the programmatic access. I can click on the next. Okay. I can attach the policy and I can give the policy. Wait, this is loading. I can directly give the admin policy for this right now. And I can just click on the create user. Simple. Now I can go uh go to AWS CLI. Go to security credential. Okay. And click go here access keys. Uh access keys are available. Okay. So you can go to the access keys and you can click onto the create keys. **[1:03:45] Speaker A:** Now it will ask you what is the use case that you're using. But I want to go for this CLI. I understand. And I want to go next. Create access key. This will give you the access key and the secret key. Click onto the download CSV. This will have your uh keys available over here. And click onto done. Now go to your terminal. Let me clear this. Write AWS configure. Click enter. You can paste your access key here from the file. Secret key from the file. Write the region name here. AP South one output format none. Enter. And you have configured AWS in your local PC. If I do AWS S3 LS, it will show all of my bucket that is available onto my S3. So you will see AWS glue AP South one that we created. Then uh there might be bronze also available. **[1:04:38] Speaker A:** Where is that? Yeah, YT data pipeline available here. Bronze gold is available here. Script is available here. uh silver is available here. So all of these are available to us. Now once what we did we just made a connection of our uh this local PC to AWS using CLI. Now using CLI now we can do anything onto AWS. You can also create bucket, you can delete bucket, you can do a lot of things. So instead of going to AWS console and doing things you can do it on the CLI level. I can list bucket. I can go inside the bucket and I can also upload the data. So this is what we will be doing. What I will do I will go to the CD documents projects. Okay. Inside I have YouTube data python. Inside that I will go CD data. Let me just clear this. **[1:05:32] Speaker A:** I will do ls. Okay. Now all of these data are available. What I want to do I want to run this. Okay, I want to upload this CA video CSV. So I will paste this AWS S3 copy this particular file to this particular location. I will hit enter. Okay, now it starts uploading the data. Uh completed 70 MB of 61. This is my speed one file. Okay, you can also go to the console here. Okay, uh once this is complete, you can just refresh this and you will see the data available. Okay, let's just wait for this to complete. **[1:06:11] Speaker A:** Okay, just drink the water. It's already 1 hour. I think you can also drink the water. **[1:06:23] Speaker A:** Okay. So, our data is uploaded. If I do refresh, you will see YouTube folder, you will see raw statistics, you will see region and there is a file created. Okay. Now, this entire thing is called as a key. This is object. Okay, when you go back, okay, this is object which is basically what we talked about. It is key value store. So key is object name. Value is our data and metadata. So this entire thing I think the even the name of this can be considered as a key. Okay. And the object is the actual content content that we talked about this is this uh available onto the uh uh inside the file. Okay, all of these are the content. Okay, now we have to repeat this for all of the uh things. So I have already written that. If you want what you can do uh let me just get this uh for this I'll do paste here. **[1:07:28] Speaker A:** Okay. So for even JSON I have uh this uh bucket name. Make sure the bucket name is fine. YouTube and this is raw statistics reference data region CA same format. Okay, I can also do this and run it onto the terminal. Uh CA JSON ID upload. Let me just uh go back YouTube. Okay, reference data is available here also. Okay, so both of the JSON file and the CSC file is available now. But we want to run this entire query okay for both of the category of the data available. And for that what what we will do we will go to the chat GPT. I will copy this okay and I will ask uh can you uh give me this command for all of the files that is available. Okay. I'll paste the command available in the same format for CSV and JSON. And uh what I what we need to do is that let's say we can either uh get the screenshot of this. **[1:08:52] Speaker A:** Okay, these are all of our files available. So what I can do I can just share the screenshot to chat GPD copy this paste it here. So one way is you can write all of these things manually one by one or other way you can actually just use shar GPT to do the same thing. Okay. So it will do entire thing. So it will give me the entire command rather than me writing all of these things one by one. Simple. I'll cop copy this and I'll paste this directly onto my terminal and it will start uploading the files again and again. Now it is also it also have this same uh CA videos. So what it will do it will uh just rewrite the file. Okay, instead of duplicating it. So this will run in the back end. Same goes for the this. So what I will do I will just put that onto you can just put that over here itself on AWS copy uh here and the same for this. **[1:09:55] Speaker A:** Okay. So both of the copy commands are available here. So I'll just wait for this to finish. Once this finishes uh we'll move forward with the video. uh once all of the files are uploaded you will see them uh inside your bucket bronze bucket inside YouTube bron statistics all of the data will be divided based on the region itself so that uh we can easily access the data okay so till this is happening okay uh let this this happen in the back end we can move forward right uh we don't want to wait for this uh right now I have some internet issue that is the reason it is uploading little bit slower uh so I'll wait for that uh till this happens if so so our data is being uploaded what we can do we can learn about all of the other data storage available onto AWS okay so after S3 okay we have EBS and EFS which is one is a block storage and one is a file storage okay EBS elastic block storage and elastic file storage these are the **[1:10:54] Speaker A:** two storage mechanism available we might not be using this uh in this video but uh again this is a masterclass video so it's good that you understand these two storage classes also available. So AWS EBS considered as a block level storage volume for EC2 machine. Okay. Attached to single EC2 instance available onto the same A persistent independent from the instance life cycle. These are different types of EBS available such as GP3, GP2, IO2, ST1, SC1. snapshot for backup store uh snapshots for backup stores in AWS S3. So S3 is an object storage used for objects like we understood the key value pair it stores that way. EBS usually used with the AWS EC2 right when you create a machine with the laptop we also have a storage available which is the hard disk that is attached okay like that onto the EC2 if you want to **[1:11:52] Speaker A:** attach a storage or a block storage you can usually use AWS EPL then we have EFS okay this is NFS file system what is network file storage I think NFS full let me see NFS on AWS OS I think network it is network level file storage something uh yeah network file system okay uh basically uh disk that is accessible via network okay fully managed by AWS uh it is a file storage like on our uh PC right we have different file system available where you go inside the folder then inside the folder you can store the different folders and then store the file this is available via the network so you can connect or you can attach sort of like a external hard drive. You can think of it that way that instead of connecting it via wires you can connect it via network in the back end it might be doing it on the physical layer but via the network it looks like **[1:12:54] Speaker A:** you are sharing data over the network right over the internet with this file storage you mount it with your EC2 machine or any of the storage uh any of the machine that you have and you can share it across the different EC2 also. So this is fully managed NFS file system shared across multiple EC2. It is like one storage that that can be accessed via all of the EC2. Automatically scales the storage mostly available for Linux. Okay. These are two modes like general purpose and max IO. It's it also has the storage classes and life cycle management for cost optimization. This is quite expensive. EBS is a block storage. Again we uh you if you want to go into deep of all of this you can always read the documentation you can read the uh the notes we have available if you want to go into the storage but **[1:13:43] Speaker A:** I just wanted to give the overview about this because we are not going to use this but understand the storage inside storage we have S3 object storage EBS as a block storage and EFS as a file storage available okay so let me just look at this so my CSV file files are copied. Let me copy this uh JSON files also. So I'll go here. If I refresh this, you will see all of the files are available. If I go back in the raw statistics, all of the files are available for this also. So this is done, right? Uh if I go onto my architecture, this is done. I have uploaded my data using simple script onto the S3. We will do this. Don't worry about this but this is we have the data available. **[1:14:32] Speaker A:** Now let's go to the next step. Okay. Uh we are using AWS Lambda. We are using AWS Glue. Uh lot of different services that is connecting with each other. Right. We have AWS S3 available. So AWS Lambda will read data from uh AWS Lambda will read data from AWS S3 to extract the data. Uh AWS blue okay will read data from S3. So inside the IM we looked about creating a uh IM user but there is one more thing that is called as a IM roles. Okay we looked about user. User is basically anyone who you want to give access to AWS console or the keys so that they can access it. Then there is roles. IM role is identity. You create specific permission with a credential that is valid for short duration. Roles can be assumed by entity that you trust. That basically means when a user like you or me want to access AWS services, we have to create a IM user that you can give share your AWS console access such as the username, password, access key, secret key **[1:15:39] Speaker A:** for them. But when AWS services wants to talk to each other, okay, wants to talk to each other, you use IM rules and you attach it to that. So instead of uh having a access key and a secret key return into uh into the services for example AWS Lambda wants to read data from S3 we have to create a connection between those two. Okay one way is that you can uh use AWS access key and the secret key put it inside the code use the package for S3 and connect it and then do that or second way is you use IM role. So it will create a automatic connection or it will give the permission to AWS Lambda to read data from Amazon S3. Okay. So for that you will have to uh now next thing is what we going to do is we are going to read data from Amazon S3 and do some cleansing to that particular uh data. **[1:16:39] Speaker A:** Okay. And in order to do that we have to create a rule. What we will do AWS service use case is our AWS Lambda. Okay, simple. I come create a role AWS service lambda. I click onto the next and now I will **[1:16:58] Speaker A:** search for AWS Lambda basic execution role. Okay, this provides simple execution access onto AWS Lambda. So I click onto this. I will click on to next. I'll click the role name YT data pipeline lambda roles dev. Okay, this is the you you can give the same name also. This is not universally unique. You can give the same name. I will click onto the create role. Okay, once this role is created, I'll go to let's say lambda role. Okay, this is my role. You can search it. So basic execution is role uh execution role is given but we want to also give access to AWS S3. Now one way you can do that is you can just go here and click on to add permission attach policy and give either full admin access or I can give S3 full access also. But usually on cloud you follow the principle of least privilege access. **[1:18:07] Speaker A:** That means you only give to give access to the services that they need. Okay? You don't have to give all of the accesses right now. So based on the use case, you only give the access. So what we what we will do, you will click on to add permission and click onto the create inline policy. Okay. Here you can click onto the JSON uh and give access this way. Okay. effect allow action I want to uh give this access. So s3 get object **[1:18:49] Speaker A:** uh yeah get object comma I'm just showing you for this one s3 put object uh comma **[1:19:02] Speaker A:** s3 list bucket. Okay, we want to give these three action. Put object list bucket to the resources. Okay, and you write resources like this. So, ARN uh you can get it from here itself. I think if you go to S3 uh oh I think uh bronze okay can't copy URL action copy. Is there any ARN to copy info? Okay, let it be. You can just write a arn col aws col s3 triple colon then you write the bucket name yt okay slash star okay **[1:19:56] Speaker A:** you do the same but without star and uh it will also need access to silver so I will also do silver here and same silver without the star. So what we are doing is that uh comma comma let me remove the comma from here. So what we are doing is that uh you can I think format this from there's no formatting option. Okay. So what we are doing is that we are giving action. Okay. Action is basically whatever action this lambda can take on S3. like I I'm giving access to S3 get object put object and list bucket to only these buckets not all of the S3 bucket only these four buckets okay the all of the subfolders and the main folder access this is fine so this is how you can give access to AWS S3 okay what I will do I will do next I will give S3 lambda yt I client policy and I will click on the create policy. **[1:21:05] Speaker A:** Okay. So now Lambda can access specific buckets and the basic AWS Lambda execution role. This is done. **[1:21:12] Speaker A:** So now we created the AWS IM uh role. What we will do? We will go to Lambda. Copy open it onto the next tab. Okay. Now let's look at what AWS Lambda is. So let's understand the AWS comput services. Okay. We have EC2 and the lambda available. We already understood about the EC2. It is a virtual server in cloud. These are the different available like general purpose, compute optimized, memory optimized, storage optimized. Pricings are on demand, reserved. You can have a spot instance available. You can have a saving plan. AMI is basically preconfigured server template. So let's say if you want to have a pre web server installed onto your EC2, you can have a E AMI created like a image created for that and security groups to protect the EC2 with the firewall. Okay. Then we have AWS Lambda. AWS Lambda is a serverless compute. **[1:22:06] Speaker A:** It runs code without managing servers. Here you have to manage little bit of server. You don't have to manage anything here. It is eventdriven. So you can trigger AWS Lambda with S3 API gateway, SQS or any external trigger. You only pay for the compute the amount of time you consume. It supports Python, NodeJS, Java, Go, Net, uh, Ruby. Maximum 50 minutes of execution and 10 GB memory available. Layers for use usable code and uh, dependency, concurrency control, ideal for microservices, data processing, and automation. Let's look at the UI side. Okay. So go to the AWS Lambda here uh here itself. Okay. This you will see I again I have all of these available. But in your case you will not see anything. You will go to the dashboard. Uh all you have to do is click onto the uh let's wait for this create function. **[1:22:59] Speaker A:** Okay. You can author from scratch. You can use the existing blueprint or you can use a container image. I'll go author from scratch. Give the name as YT data pipeline JSON to park dev runtime you will select let's say Python 3.11 okay this is fine click onto the change default execution role and use another role we created our own role here uh which was if you go to the role uh YT uh this this was the role that we created YT data pipeline lambda role Uh this is the role that you have to select. Okay. And then you click on the create information. Uh I'll close this. Okay. So now we created a simple lambda. Okay. This is the UI. What you will see here uh the function overview. This is a diagram or the template. You can also look at the back end of it. **[1:23:54] Speaker A:** This is the diagram layer of it. You can add a trigger also. So as we uh saw here uh you can uh use uh what is that event driven like you can add a trigger from S3 API gateway or SQL. So for example whenever any file is uploaded onto the Amazon S3 this lambda function code should run. Okay this is sort of like the event available like you can have a API gateway Alexa you can have IOT SNS SQS all of the different things are available here mainly uh it is fine all of this. So we have a code here available which is like a VS code. We have a testing for that. You can monitor it how it is using. You can configure some settings here. Then alias and versioning. Okay. We'll go to the code and uh we will uh put the code here JSON to park. **[1:24:45] Speaker A:** Now this code we have written in our previous version. Now there are two ways again. Uh in the world of AI no one writes code anymore. uh usually you can go to chat GPT and you can tell okay uh I have my data available onto let's say this this S3 uh bucket I want to read this data and convert that data into the pocket format okay so sort of like I can do that uh so uh okay can you write lambda script to convert this data from uh location above to uh into the silver bucket. Okay, I'll go here. Uh let me just copy the bucket name we have here to silver bucket in to park file format. Okay, you can write and this will give you the code into the Python. Uh I forgot to say Python to it but I think it it will by default give me the Python code. **[1:25:53] Speaker A:** Uh what it will do? It will give you the entire code and then you will be able to uh so what it will give you the entire code. Okay, this will give you the source bucket. You can put the target bucket park compression. This will give you the entire code. So if you know the basics of Python and if you're doing this project, if you are learning AWS, I expect that you know at least some Python language uh so that you can understand all of these code. If you don't know, you can go to the data vidya. We have like uh detailed Python module available. You will have everything here from installation to understanding about all of the basics of Python available. I teach you from basics. So if you want to learn Python, you can come here to the data vidya and learn it here itself. **[1:26:35] Speaker A:** Uh uh I'll put the link in the description also. So you can come here and learn everything. But for this uh what I want to do, I don't want to waste time writing all of this code right now. Okay. What we can do here is pretty simple is I have already written the code. I will explain you over the top because uh these days we don't usually write code by ourself. We can just use AI and put it over here. Okay. So this is the code I have written. I'll explain you the entire code and what we will do. I will put the code here itself. Uh inside uh you will find this uh entire repo onto the GitHub also. So inside this we will create a folder. We will do lambda. Okay. uh inside the lambda we will write we'll create one more folder inside the lambda also JSON JSON to pocket and the new lambda function file okay and I'll put it over here itself now what this code does is **[1:27:42] Speaker A:** uh first of all it imports all of the packages like JSON OS login date time URL lib boto3 uh to work with a services AWS wrangler and pandas there's a logger available if you want to log all of the messages uh inside then we have some configurations okay so what first is we store the silver bucket uh uh name inside the environment we will do that then we will also use the blue database and blue table we will do that right now we we have to configure because over here what we will do uh we will read data from uh the S3 bucket that we have. Okay, we will clean that data little bit and store it into the park file format as per the architecture. Okay, we want to read the raw data. We want to store this raw data into the park file format. **[1:28:32] Speaker A:** Okay, and to the store it over here and build a glue catalog on top of it. Okay, so this is the action that we want to do. Uh read from S3. Okay. Process on lambda. Now store on silver bucket. We store on silver bucket but also store uh create blue db to store data there. Okay. so that we can also explore the raw data and how the raw data actually looks and also explore on Athena. uh now I know that okay learning AWS and also executing this is quite tricky uh might be quite difficult uh but again while we are building a project we have to take care of lot of different things at the start only that is the reason we plan things okay so this is what we have in mind right now we have done this uh we are processing on lambda and then we will store it on the silver **[1:29:36] Speaker A:** bucket and this is the script available for that so we will put the glue database table information SNS topic also we will create that and silver path let's just do uh understand what this uh code does from the higher level. Uh again I expect you to know the basics of Python to understand this code. So whenever we uh upload any data onto the S3 or it will read the data from the S3 upload data. We will do the uploading of the data again. It will check if any if the event is empty or not. Okay. If the event is empty, it will not do anything. If event has some record, it will go one by one to the record. It will read all of the S3 data. Okay, based on the it will extract the bucket name and it will extract the key of it. **[1:30:22] Speaker A:** Then it will read the JSON. Okay, we provide the bucket and the key. So there's a function available on the top that basically uses S3 client to get the data, uses body to decode, then load the JSON content. Once the JSON content are read, it will return the JSON content to the raw data. it will do the processing. It will normalize the JSON. If it has the items, if it doesn't, it will just do the normalization. It will validate the category of the data. So there is a function available to validate the category. Basically, uh validate and clean the category of the data. So if there are any issues in that so in this architecture as we can see we have lambda function. Okay. Now that lambda function is converting our raw JSON data into transform park format. So on the lambda function we will write transformation for JSON data and on the glue we will use apache spark pispark to convert the cv file transform it and put it onto the silver. **[1:31:20] Speaker A:** So we will use lambda for the JSON one and spark for the CSV and convert both MD them to the uh spark format and put it onto the S3 bucket that will be uh we can see it on the AWS group. So we can do that easily. Okay, let's just start doing it. **[1:31:42] Speaker A:** And now before we do that right like we uh configured uh AWS uh this uh lambda role we will have to create role for uh many different things. So first we will do the setup for the entire thing. Let's just uh do the entire setup for the uh access control and then once the access controls are sorted we will start executing all of the code level things. Okay. So we have already uh created this lambda function YT lambda onto the AWS Lambda. We might have to modify few things here also. So you can go in into this click onto this policy. Okay. The IT data pipeline uh uh this policy if you click onto this you can edit it. And here uh I will paste this policy. Okay. Uh on top of this. So this is the policy we return for S3 where this is the so this is the policy that we wrote for S3. **[1:32:42] Speaker A:** Now uh let me make sure the bucket names are proper. Okay. Let me update this. Update this. Update this. And update this. This was silver and this was silver too. The same thing we did for uh blue. Okay. So, lambda can uh get the glue tables, get the databases, create tables, update tables, get the partition, create partition, batch create partition. This SNS thing we will uh create and update it. Here we just have to uh put the account ID and account ID you can copy it from the top right over here you can see and I can just click onto the copy account ID paste it here okay this entire thing you can of course get it from the GitHub uh US uh this resource AR and YT data pipeline uh this is fine we can update it afterwards when we create this and Athena also we have **[1:33:48] Speaker A:** to give access to query execution result and the starting the query. Okay. Uh update this, save the changes the access we gave so that we can always use this. Now we also have to create the glue execution rule just like the lambda uh it can access S3. We also need glue to access S3. So go to the rules, click onto the create role and over here instead of lambda now we will select glue. Click onto that. Go next. And here uh we will uh search for AWS blue service role. Select that. Go to the next. Give the role name as YT data pipeline. Uh glue rode dev. Uh create the role. Okay. And then you can open the same role here. Y data pipelines glue dev role. We will also add some inline policy. The same policy uh we had uh but this policy uh we will have it for this uh like like the way S3 could access uh no the lambda could access S3 and glue we also need uh based on **[1:35:03] Speaker A:** the architecture we also need our glue okay to access uh S3 bucket over here okay and also the glue catalog and all of the other thing. So we also need to give the same set of permission. So add permission, attach policy. No not attach policy, add permission, create inline policy and I will paste uh the JSON version here. Okay, this gives access to S3 and the resources we will have to update. So I will update the bronze bucket name. Okay, I will update my uh based on that you also have to update your silver silver uh I will update my gold bucket **[1:35:54] Speaker A:** and I will update my script bucket so that it can access everything. So this is done. All of this is done. Now we can do next and create policy and the policy name we will give it as uh YT data pipeline glue access something like this and create policy. Okay. So glue is done also. Uh so now we have done two things. Uh we gave AWS Lambda access to S3 and the glue catalog also and we gave AWS access glue to access to S3. Okay. So now it can do access the data from S3. Uh there was one thing we did is like we created SNS the simple notification service. Uh we talked about this whenever we want to uh you know send notification to email or uh mobile number we can uh send notification based on the quality check that we did here uh onto this right uh there's **[1:36:55] Speaker A:** a data quality check and SNS failure notification we want to send if there's there are any failures. So we want to also do that. So just search SNS. All we are doing is just managing the access and policy right now. And then we will start writing the code. Okay. Uh over here you can just uh on the left side uh this is simple notification service publisher subscriber messaging for microservices and serverless application. Okay. So SNS is usually used for that like when you want to send a notification to different devices and location based on some action has occurred you can do that. So we'll go to dashboard. Click on a topic. On the left you will do create topic. Uh we have FIFO and the standard. We will keep the standard also best for messaging ordering at least once message delivery and subscription pro protocol SQS. **[1:37:44] Speaker A:** So uh first and first out is basically whatever message that goes first will always go out. So if you need that kind of requirement you can always go for that. But standard is simple. uh just give the name as YT data pipeline alerts dev display name uh it's optional you don't need anything else and just click onto the create topic my topic is created now you have the topic is created right but we have to also subscribe to it in order to receive the notification so there is a button over here click on the create subscription now there are different ways you can subscribe what we will need is email address okay I want to get all of this information on my email and I will just write my email Dial this is my email address and I will click onto the create subscription okay now if you go to **[1:38:33] Speaker A:** your email you will receive uh wait message from the notification over here okay so if you go to your email this is my email and I will receive the notification from subscription I'll click on confirm subscription my subscription has been confirmed if I close this now I will start receiving all of the messageages which is related to this uh whatever happens right make sure you copy this ARN and store it here SNS AR N and we'll paste it here so that we can use it afterwards now uh I think onto our glue not on the glue I think onto our uh if I go to lambda ro let's let's me just check that okay lambda rode here. **[1:39:28] Speaker A:** Just want to confirm this. Here we have we had SNS, right? Blue access SNS, right? So, we have SNS available here. Uh we will have to update this SNS with this one. SNS AR N. Okay, this one. And so what you have to do, you have to go to AWS Lambda YT pipeline policy. Okay, inside this uh role YT data pipeline lambda role dev. Once you do that, go here and edit the SNS resource ARN over here to the SNS policy that we created so that we give uh access to the right SNS that we created. Okay. So I think uh this part is done sort of like we what we did we set up the two lambda role and added the custom permission and we uh created notification service and subscribe to the email. So we did that. Now in order to uh over here right in order to have this uh both of the things and write data and create a glue catalog we will need a database right because uh glue catalog stores data into the **[1:40:41] Speaker A:** database tables right now. So now so now our setup now so now our basic setup is done. We have our IM policy that has uh AWS Lambda has access to S3 and Glue. We have glue that has access to this. Okay. We also have our simple notification service ready. **[1:40:58] Speaker A:** Okay. So we our components are ready. Now we have to start executing uh other things. Now before we do that uh let's uh spend some time on understanding theory. Okay. It's important that you understand what we just did. So I think we already understood about the AC2 and Lambda. Okay. There is also AWS Glue and EMR. So let's try to understand what that is. Okay. So AWS Glue is fully managed serverless ETL service. Okay. Glue data catalog centralized repository, centralized metadata repository. So AWS Glue is usually used for ETL but it is serverless. Just like EC2 we saw uh it is virtual server in cloud where you can host your uh information, website and every other thing. And AWS Lambda is a serverless compute service where we can write our Python code any code and it runs onto a trigger basis or the uh as per your need. **[1:41:56] Speaker A:** Okay. So this is ideal. Okay. This is serverless has a limited capability but used to run the core without managing servers. This is where you have to manage a little bit of server. The same concept goes with the Apache Spark which is where AWS Glue and the EMR comes. AWS glue used for writing ETL jobs for Apache Spark or Python shell. Okay. So it is fully managed serverless ETL service. It has a data catalog right. So it is about data of your data. That means it stores the information about your data such as how many columns it has, what are the data types, uh the size of it, number of rows, all of the information about your data it will store. You will understand when we do that. Okay. Then there's a crawler. Okay, what it does, it goes to your S3 file. Okay, it reads all of the information from the data so that you can start writing SQL queries on top of it. **[1:42:52] Speaker A:** So, it will autodiscocover schema from S3, JDBC or any of the sources and it will pull all of the information. Again, we will understand when we do the hands-on. Then we have the ETL job. You can write it in Pispark or Python shell. Job bookmarking for incremental processing. Dynamic frame API extends the Apache Spark uh data frames. It also supports Spark A, JSON, CSV or C, ARO, integrates with Athena, Red Ship, S3 and all of the other services. So AWS Glue is a serverless ETL service for Apache Spark. Then we have the EMR. Okay, this is the managed big data platform for Hadoop, Spark, Hi, Presto. This is usually used for processing very large scale of the data. Right? Right now that is we are using as group because we have the limited data but EMR is used for processing very large scale of the data. **[1:43:39] Speaker A:** We have the autoscaling cluster with the spot instance you can have a spot instance available. Spot instance is basically if you want to you know get AWS instance for minimum amount of time for a lower cost uh for let's say 2 to 3 minutes. Usually when you do the standard it will uh run the instance uh and it can run up to the longer period of time as whenever you need spot instance are usually available for small period of time let's say the 10 minutes 20 minutes and there is a bidding happens uh also uh we don't want to go into depth but uh it's just a uh limited time instance available for your support. So we have the EMR on EC2, EMR on EKS or EMR serverless. We have three things available. Either you can run EMR on EC2 that will automatically uh or install all of the hard spark into the cluster. **[1:44:30] Speaker A:** So that will be the worker node and the leader node that will manage it. You can also run it on EKS is a Kubernetes service and EMR serverless is also available. You can use notebook for exploration. There are like Zeppelin notebooks available, Jupyter notebook available. You can use that. Uh it also directs uh it also has the direct access to HDFS. Hardly anyone uses it. Cost effective for large scale batch processing. Ideal when you need a full stack spark application. Okay. So again understood AWS AMR and glue. Let's spend time on glue now and understand. So on to the AWS just write glue and open it onto the new tab. Okay. Let's wait. Uh so this is what the glue console looks like. Uh earlier the console UI was different. Now it is a new UI over here. Uh what do you have right? Getting started ETL jobs right? **[1:45:24] Speaker A:** If you want to create ETL jobs here you can create it onto the script level. Uh or you can use notebook you can use also visual edit. Uh then we have the catalog tables available that we were talking here. Uh let me just get this. So again we saw that it it is a data catalog. Okay. It stores the information about your data. So as you can see all of these are catalogs. I'll just show you one about from my previous project. So it stores the schema information about the table like it has column name, date type, data type, partition also if it has the partition indexes, all of these information are stored inside the catalog, right? All of the information are the catalog and the data is coming from the S3. We will do this. Then you can also create connection with the external sources if you want to do that. **[1:46:12] Speaker A:** Workflow orchestration if you want to you know orchestrate the entire system. This is the detailed data catalog. As you can see, uh this is the catalog available about everything. So it's a COVID 19 catalog. Uh you will see uh all of the table inside the COVID 19 we had like if I open this I think we just opened this. So again same column name and everything is available. We have the databases inside the databases we have the tables available. Okay. Then uh schema registry uh we have connections we have and the important thing we need to understand is the crawler. Okay. Okay, this is a crawler available that we will run on top of our files. So these things are available. These are the legacy pages and this is for the data integration and ETL jobs that we will use. So this is the same uh overview but this is just the in detail. **[1:46:59] Speaker A:** So this is the UI of the glue. What we want to do is we want to create a databases for our uh three uh section which is what we learn here. One for the bronze, second for the silver and for the gold. So any data that we crawl from here we want to store it that information onto the that particular database. So go here onto the databases click onto the add database and uh let's just give uh YT pipeline bronze dev okay and uh uh description if you want uh you can just put it YT pipeline raw data and click onto the create database. So one database is created. We will perform the same action for the silver also. So I'll put the silver here and YouTube uh this is this will be our cleanse data. Okay. And the same for a gold one. So I'll put here I'll put this analytics aggregation. **[1:48:04] Speaker A:** So we have bronze, silver and gold available. The databases are available. Okay. Now in this example again we had in our kegel data okay we have JSON data available and the CSV data available. Now you can always run a crawler on top of the CSV data but sometimes JSON data breaks because of the complexity of the file available. Uh if it is a simple JSON data blue crawler might be able to crawl it and uh you will be able to uh run SQL query. Sometimes if the uh it is some nested JSON available like this it might not be able to do that but you will try on our end okay and let's just see how it goes. So what we have to do we have to run the crawler okay crawler is basically it will go onto the S3 files it will read read all of the schema details and put that onto the AWS clue catalog. **[1:48:58] Speaker A:** So I will click onto the create crawler and I will give the name as uh YT data pipeline bronze crawler dev. Okay, click onto the next. Now we have to tell uh crawler that where is our data located. So we have to add a data source for that. So click onto the add data source. Uh this data source is our S3. Again you have lot of options available here. You can get data from m uh RDBMS, red shift, iceberg, hood tables. We'll just go for the S3. Okay, we don't need any connection in this account. I want to find the data. You can click onto the browse S3. Uh the data is available onto our uh here uh this location YT data pip and bronze. So I will just put that bucket. Click onto this bucket and you have to go till this level. Right. **[1:49:51] Speaker A:** uh I want to uh get uh crawl data from all of these folders with this folder partition. So I will go till here uh I will just select raw statistics. Make sure you select till the raw statistics and choose. Okay, this is done. And add S3. Nothing else. Again I can add the data browse same and till the raw statistics reference data. This is our JSON data available. Okay. And I will add the data here also. Okay. So I have added both CSV and the JSON. I will go click next. Now we have to choose the IM. We have already created our IM over here as you can see. Uh this is lambda 1. Uh wait I think we yeah so this role we created YT data pipeline glue dev role that we have to find here. Okay this is it. If not you can always refresh it. **[1:50:51] Speaker A:** Uh everything else is fine. Click on to next. Choose the database in this. Again we are doing this for the bronze. So make sure you select the bronze database. If you want to add a prefix, you can add it. This is on demand. Click onto the next. Create the crawler. Okay, crawler has created. You can run this crawler by clicking over here. Just run the crawler. The crawler will start running. Created successfully. Uh if you see this is running over here. Crawler runs are being run. So it will take 1 minute to run the crawler. Okay. uh after the crawler runs you will see the tables created into a database. So if I go to tables here open it onto the new tab uh catalog is fine. Choose the database here as a bronze dev. Okay. Right now no tables are available because cl crawler is crawling the entire data available on S3. **[1:51:48] Speaker A:** Once it crawls it will automatically create the table and then we will be able to write SQL queries on top of it. Okay, till this uh till it it is happening, let's move forward with our theories and try to understand the Athena part, right? Because once the crawler crawls our data, okay, how do we query it? And this is where the Amazon Athena querying and catalog service, right? We Amazon Athena comes. It is a serverless interactive query service. Okay, it is a serverless interactive query service standard SQL on S3 data. So usually when we use relational databases what we do is inside the database right we create tables and we physically store all of these data in the relational database the data is present there but here Amazon Athena is just the interactive query ad hoc query interface okay so it does not own any data what it does it uses glue catalog that has the information about the data such as table name schema and all of the other **[1:52:50] Speaker A:** information to run SQL query is on top of this act the files that we uploaded onto the S3 bucket. So it is not storing any data inside the Athena. All it is doing is just running a query layer on top of the files. This is the power. So it you pay per the uh per the query that the data it scans. It supports park OC JSON CSV. Uh it uses blue catalog at as a meta store. SAS for ETL. Okay. You can write uh ETL query and data transformation also. when we do the Athena you will understand then there is red shift spectrum also okay so red shift is a data warehouse that we already uh saw previously I think I think we have we are yet to touch the database module uh we skipped it uh because to understand the other services but red shift is a data warehouse available on AWS okay it's a OLAP based uh if you don't know about OLAP and OOLTP I have **[1:53:46] Speaker A:** a data engineering masterass available on my YouTube channel also if you go here uh this master class is available. Uh you can uh uh where is start? Yeah 3 hours master class is available. Watch this and you will clear most of the fundamental concept of uh data engineering. Okay. So uh yeah it is an OLAP based data warehouse. Uh it is basically used for querying the data onto the red shift cluster. Uh it doesn't need data loading. You can use a standard SQL and it is cost effective for cold and archived data. We don't want to touch red shift right now. We have a glue data catalog that we already understood right. Centralized metadata stores. It has tables, schemas and partition shared across Athena, Red Shift, EMR crawls, auto updates the schema high compatible metast version table for definition. The data catalog we understood. Athena we understood. **[1:54:40] Speaker A:** Let's look at our crawler. It is completed. It says table changes two table changes 20 partition change. Okay. If I go here and if I refresh, this is my tables, right? That we opened. You will see two tables are available. One is the CSV one and second is the CSV uh JSON one. Okay. **[1:55:00] Speaker A:** Now I can query this data onto Amazon Athena. You can write Athena on top. Open it in new tab. Okay. **[1:55:18] Speaker A:** here uh click onto the query data in Athena console click onto the launch query editor first before you run the query you will have to set up a result location so on here click on the query setting okay and you have to uh you might not have this I have it already but uh location of the query result you will have to select any S3 bucket for that. So either you can create any simple S3 bucket for like that. Let's say uh something like go to S3, click onto the create bucket and write something uh YT data pipeline uh blue Athena query result. Okay, we'll do this and uh everything else is fine. Click on the choose bucket. Okay, you can use this bucket now inside the Athena. Browse S3. Click onto this. Select this bucket. Choose and save. So all of the result that you execute will be stored here. **[1:56:30] Speaker A:** Now this is what the Athena console looks like. catalog is this uh and uh you can select the uh database as bronze and you will see raw statistics and reference data is created. If I click onto the preview by uh clicking here you will see all of the data are available as it is. Okay, these are the some new lines available also some uh this is the raw data right is it is not properly clean data. So you will see a lot of issues in this data. But we are able to at least look at the data from Athena. So if you want to understand what the data looks like, what is what does the data have, you can always check here. You can also go to the catalog here uh bronze and click onto the this. You'll understand about this also. Raw statistics database name is this. **[1:57:23] Speaker A:** CSV location is this. Uh we have this columns. These are the columns available. And this is the partition available. Click onto the partition. You will see uh partitions are based on the region. So our files are divided based on the partition. That is how we have created our structure also. Okay. And you can view the files. You can view the properties. Okay. Then we have the indexes available. So it is index based on the region key that we have. And column statistics. Right now we don't have anything. Okay. So this is the catalog the data about our data. Okay. It stores the information about our data. This is the advanced properties uh how it actually created this table. So you can see the record count uh crawler uh information, size of the key, skipping the first header count, all of these different information are available here. **[1:58:12] Speaker A:** Okay. So I'm able to query this CC one. Let's try to query this JSON one. And you will see we get some error. The error is because it is a JSON nested data. So we are not able to actually read it on Athena. And it might have some issue in the data file also. So we are not able to read it. Totally okay. We don't want to worry about that. When we transform our data from JSON to park, we will be able to read it. Okay. So you can write some queries here also if you want analytics query to you know get information such as uh query like this. Select region and uh aggregate by count and then uh from raw statistics group by region order by count and you will see the highest videos are in like CA region then USD like that we're able to get the **[1:59:01] Speaker A:** information that CA has the highest video count like that uh you can also uh run query like this uh this is uh giving you information on inside the region of US and you are able to see wait it's not capital it's small US wait yeah and now here you can see that okay this is the video ID okay it looks like this trending date is quite weird as you can see right it is like uh 17 141 this is into like y dd or mm format this is like year this is month oh no not month the day and month so we also want to standardize this when we actually convert our raw data into the silver data when we transform our data we want to make sure that pending date is in proper format okay uh then this is fine I think this is fine this is fine so just by looking at the **[1:59:59] Speaker A:** raw data we can understand that okay this data has lot of issues that we need to fix right before we move forward so that will happen into our silver okay so this is uh done uh let me go Here uh over here we were able to what we did right we are getting this data okay we are yet to write the transformation block for this but what we did we ran the glue crawler okay we also have the archive that automatically goes here and now we are able to read our data onto the Amazon Athena okay the raw one for the raw now let's uh start writing the data for this also so let's get **[2:00:40] Speaker A:** the data first here then we will move forward okay let's just get all of the raw JSON data also added to this so that we have the consistent data coming also. Okay. So we have convert that into park and then we'll write this. So we will be writing our extract data pipeline from the YouTube data API. So we did everything for the static data. Now let's do it for our API. Okay. So let's go to AWS Lambda. **[2:01:14] Speaker A:** And on the lambda we will create a function and author from scratch we will give yt data pipeline injection dev. Okay yt data pipeline YouTube injection dev. Uh everything else is same. Uh click onto the execution role. Uh select the lambda role that we created. Uh where is that yt data pipeline lambda dev role. Create a function. Okay. Uh now we have to write a code. Again uh there are multiple ways we can do this. Uh one you can we can write code by itself or you can directly use chat GPD or cloud to generate the code. I have already done this project previously. So I have the code available. So what I will do uh again I expect you guys to know basics of SQL and Python because this is quite advanced project. So we are not going to go deep dive into the Python. **[2:02:05] Speaker A:** If you want to learn Python again uh we have a dedicated uh section available on datawidyia.com/arn. You will learn everything about the Python from strings, number variables. We also have a project available on Spotify data pipeline. So if you are very uh learning Python from scratch. This is the best resource you can go to and learn about the Python and everything we have like we also have advanced Python available and everything uh with the editorial and the video. So it will help you to learn faster. uh we have learn section for fundamentals SQL. We also have the practice platform for coding. So you can also code here for the interviews. But this is something else right over here uh for injection of the data. Okay. Uh go to the new folder in your VS code and just write lambdas. Uh we will uh lambdas. Okay. **[2:02:57] Speaker A:** We will put all of our code here. So first is YouTube API injection. **[2:03:06] Speaker A:** Okay. And here we will write lambda function. **[2:03:17] Speaker A:** Usually the lambda function name the file of the lambda function should be always lambda function.py pi because uh let me oh uh this lambda function uh I'll have to recreate this and delete this one. The reason for that is uh this injection one. Uh I selected the NodeJS we had to select the Python. Okay. So let me delete this confirm. If you have got that error then make sure you correct it also. So this Python 311 uh use role I'll choose lambda ro and click onto the create function. Okay this is done okay here. Okay so the lambda handler okay usually this is the lambda function.py the same you usually keep this only name and the lambda handler is the uh start of the uh execution of the lambda. So whatever you do uh it will start the execution from here itself. So let me paste the code to extract the data from uh API. **[2:04:29] Speaker A:** I'll explain you this entire code. If you want to understand the code is pretty simple. All you do is copy the code as a char. Can you help me understand this code line by line? Okay. And put this uh if you know Python, you'll be able to understand just by reading it. If you don't know this is the best way because if I again we'll be writing lot of code. So uh it doesn't make sense to you know explain each and every code one by one. So you can go here and this will help you understand entire thing one by one. But I'll explain you also from the top. First we import the uh uh important package as a JSON OS login and all of the other things. B3 to work with the AWS uh APIs. Then we import two clients. One for the S3 client. **[2:05:17] Speaker A:** So boto3.client S3. Second is SNS client uh boto3 client. SNS. Then we import configuration such as uh YouTube API key that will be stored into our environment variable. S3 bucket the bronze bucket name uh where we will uh put the raw data. Uh we have the regions that we want to work with. Okay. Uh environment. Uh then we have uh uh API of the Google and the max result that we want to get. So the function starts here. Okay, simple. Uh you define all of the dates uh you you go from the regions all of the regions that you have that you want to pull the data from just like our raw data, the static data that we have. Then it will do some uh uh processing of this. Uh it will go to the fetch trending videos. It will hit the URL. It will get the result. **[2:06:09] Speaker A:** Okay. It will open it and it will uh put the result back here. Trending data. Did we get any data or not? If we got, it will put that onto the dictionary. Okay, it will write that data onto the S3 bucket as a JSON file. After this, these are just exception. If any of the issue occurs, it will raise the exception here. After that, you want to fetch the category of the data. So, there is a function created fetch category. Again, it will hit the URL of the video categories. It will send the region code for each of it based on region we have available. it will request that uh once if the data is available it will return the data it will write that category data again okay and at the end will just summarize the result and send the SNS notification if there's **[2:06:54] Speaker A:** any failure again you can go through this code entirely uh don't want to spend the entire time explaining you the code because quite long code what we will do we will just uh do the basic setup required for this code uh to move forward so uh let Let me go to lambda. Okay, first go to configuration here. Edit the configuration and increase the memory to 512, storage to 1024 right now. And the timeout, make sure timeout is 5 minutes uh so that uh at least lambda runs for the 5 minutes. Okay. Then uh after that uh click onto the environment variables inside the configuration. One is you need uh first thing is UPI uh not UPI YouTube API key and click onto the value. V value you can get from console uh get YouTube uh API key. Just go online uh you will see this UPI data API account. **[2:08:00] Speaker A:** So you need Google account for that. Uh you can go to the Google developer account over here. Right click. This will open the console and it might ask you to enable the uh YouTube project API or if it is YouTube data API 3. Click onto this below. You might have to enable it if you're doing it for the first time. I've already done that. Uh then it has a credentials here. Okay, let me just reset this. Okay, so I have the credential available. uh in your case you can just see if I do the show key you will see this is the key available. I will uh deactivate this key after this video but this is the key available. If you want you can just click onto the create credential API key just give the name uh over here uh YouTube API version 3. Okay. **[2:08:57] Speaker A:** And uh we can have no no restriction and just click onto the create and it will create for you. Okay. So let me just put my API key. Uh copy this and you can put your API key here. Okay. So I put my API key. After that you have to put S3 bucket bronze. So we can go to the bucket. **[2:09:32] Speaker A:** Okay. YT. Oh, sorry. We can just copy it from here. **[2:09:43] Speaker A:** I have I had a typo here. That's why. Okay. Bronze. Forget it. I'll put it here. And uh SNS alert topic ARN. Okay. Okay, we also have this. **[2:10:04] Speaker A:** Okay, and YouTube regions. Now, if you want to process all of the regions, you can get data from all of the regions. If you just want to get data from selected regions, you can also do that. So, let's just uh see. Okay, let me just see the code. **[2:10:26] Speaker A:** Okay, YouTube regions it is getting uh from split. Okay, let's just put a few regions here. US GB in. Okay, the environment variables are set. And then let me just paste the code here. **[2:10:50] Speaker A:** Pasting the code. Click on to deploy. Okay. **[2:10:59] Speaker A:** So we are getting all of this information from here. Max result five. Now we can test this. So click onto the test. Click onto the new test event over here. Okay. Uh give the name uh YT bronze bronze test. Okay. And save this. and invoke it. Okay, if you pull this again, the UI is quite uh chaotic. You will see there is a success status message for me for this and no failure. So, uh it actually wrote something. If I go here onto my **[2:11:42] Speaker A:** this bucket with the typo, okay, it wrote data into the capital letters. Okay. So I think we will have to change this. So GB GB India and US. What I will do? I will delete this first. **[2:12:03] Speaker A:** I'll delete it. Okay. Go to my uh this configuration or wait. We can just do on the right level itself. Close this when we write region. Let's find region. Okay. So there is upper case going on here. Uh okay. Then region region region region before we write. Okay. S3 key. Okay. over here in US we can just do lower uh it's like this over okay and I'll deploy this oh is there any other region this is fine also for category also dot lower. Okay, I will deploy this **[2:13:19] Speaker A:** and I will test this. **[2:13:24] Speaker A:** Okay, this is done and I close. Now I refresh. Okay, it is still writing it in the capital letter. We don't want this. uh we need uh to write this inside this itself not into the new folder. **[2:13:48] Speaker A:** Okay, let's see. Uh let's just try to remove uh upper from here itself. At this point only we will remove upper. Okay, let me delete this capital. **[2:14:11] Speaker A:** Delete. **[2:14:30] Speaker A:** Okay, deployed. Let's test it. It's running. Oh, success. Is still getting the capital one. Okay, let me just delete this again. Uh **[2:14:56] Speaker A:** yeah. Uh let's see. Let's see. Okay. From the top. Uh I think this is very small. I'm not able to see. Okay. The use YouTube regions get this. Okay. Uh if we change environment variable, let's say we are getting regions here and after the regions, we extract regions somewhere here. Okay. This is upper. So we'll just do lower to this. I think this should fix that issue because if we are doing lower at this level, that will fix it. **[2:15:35] Speaker A:** It is deploying. Let's test it. Uh the output should come into the lower characters. Yeah, now it is coming to the lower character. So now if you go to raw statistics CA, you will see uh JSON is available here. And if I download this JSON and if I open this, you will see uh the same information. I mean again the data that was coming in 2014 or 17 was different and the data that up uh YouTube API returns is different. So we will have to tackle that uh that way. **[2:16:16] Speaker A:** So it has so so see this so this data has all of the information such as E tag ID publish channel ID title description thumbnail information uh all of the informations are available here only. So we will be able to uh use this data uh to write our transformation. Okay. So our injection is working now. So now we got our YouTube data API onto the bronze and also the kegle data set. We will not touch the kegle data set. We will work with the YouTube data API. **[2:16:46] Speaker A:** All of the JSON data we will now transform that AWS Lambda uh raw data into the transform park format. Okay. So let's just do that on AWS Lambda. We have already created the function uh which is JSON to park dev. Okay, let me just uh uh we can just paste the code also for that. So uh in our uh lambda create new folder JSON to park. Okay. And here uh we can paste the code new file lambda function.py. Okay. And again uh it is same. Okay. Uh JSON reference data to silver park data triggered by S3 when the new JSON lands into the pro bucket under the reference data prefix. Uh improves over original data. So it does the validation dduplication proper error handling uh ID import and right structure logging we need uh S3 bucket of the silver blue database name table reference and SNS uh ARN does the same we read the data **[2:18:00] Speaker A:** from environment have the two uh objects created uh based on the event that comes on the S3. So anywhere anytime any jaw data from the API comes to the S3 it will read that event it will read the JSON from S3. So there is a code like from get object get the body and it will return the S3. Okay. It will check if it is items available then it will normalize based on that. If not it will directly convert that into the data frame. Then it will validate the category. So there are some rules added. So value error if there is no category found uh it will check the title if there are no title found if check it will check if there are any missing columns available it will avoid uh it will drop the duplicates of the data here uh it will add the injection time stamp and the source file key uh region unknown if there are no regions available uh it will loop that and add the region according to that **[2:18:57] Speaker A:** and then it will write the park file at the end okay out of this if there any error occurs it will send the uh send alert to the uh Y data pipeline uh and it will send the message. So we will get the uh SNS topic here, right? SNS client.publish topic, subject and the message. Simple, right? All we are doing is just reading the JSON data, processing it little bit. Okay, cleansing it, removing the dup duplicates, making sure it is in the proper format and then processing that data onto the silver bucket. Let me just copy this and paste it over here. For this also we will have to configure our uh lambda function properly. So you can go to configuration of this uh make sure you edit memory to let's say 5124 **[2:19:48] Speaker A:** uh 5 minutes timeout and save uh environment variables. Uh click on to edit. We will have S3 silver bucket added. So, S3 silver bucket. Uh, you can copy the S3 silver bucket. Where is that? From here. **[2:20:11] Speaker A:** Uh, after that we have to get add glue DB URL. Uh, we can get it from glue itself. Uh, databases. We can store that this also. Uh, blue, bronze, **[2:20:32] Speaker A:** blue, gold, **[2:20:37] Speaker A:** blue, silver. Okay. Uh, so we need silver database added to the environment variable of lambda and SNS alert topic ARN. this we can use this to copy and paste here and save it. Okay. So environment variables are added. Uh you can do save. Now we want to run this lambda based on the trigger. The trigger we want is I want to add a trigger onto **[2:21:15] Speaker A:** S3. Okay. Uh the bucket name we have this Bronze Bros. Okay, I made a typo there. Uh this is the bucket. Okay. And event type all event created uh prefix you can have this uh only the category data that we are getting uh dot JSON. I acknowledge this and add. Okay. So what I did I added a trigger onto the S3 bucket of the raw statistics reference. Okay. So any data like see if I go any data category data that comes here we will be able to uh any data that comes here. Okay it will automatically trigger this lambda function. Okay that will convert this JSON data into the park format. Uh let me open that also. Bucket **[2:22:26] Speaker A:** uh silver. What's this? Okay. So it will process this JSON data and put that onto the silver bucket. after processing it. If you want to test it, all you have to do is go to the previous lambda function which was the API that is ingesting data from the YouTube. Test this. Okay, so I will test it. **[2:22:55] Speaker A:** Okay, it is successful. So it added some data. Now we will wait if any objects are written here. Okay. So it should automatically trigger uh this parket to dev. If you want you can go go to the monitor and click onto the view cloudatch. Uh it will show the logs of how our lambda is running and we'll be able to understand uh if we get anything or not. Okay. So right now uh it ran I think uh I think there are no logs added yet. So, let's just refresh this. Uh, if there are any runs or not, I think. Let me go back. No logs yet. Uh, it didn't run maybe. **[2:23:52] Speaker A:** Uh, okay. Might be some issue. Let me check the trigger again. **[2:24:05] Speaker A:** for YouTube raw statistics data. This is fine. Is complinks object created. This should run. Uh let me run this one more time. This lambda function of the injection test. Test one more time. This is running. **[2:24:26] Speaker A:** Okay, this ran. Uh let's see the logs if there any. Okay, so there is one log. So it started running. **[2:24:54] Speaker A:** Uh I think uh did we deploy this code? Let me wait. Okay, this is undeployed. Okay. Uh my bad. We did not deploy this code. So we have to deploy this code first. Uh JSON to park dev. Okay. And also we are using uh most likely pandas here somewhere. Okay. We are using pandas. So for that we will have to add a layer. So click onto the edit below add layer. Okay. And there will be uh AWS SDK pandas uh we have lambda SDK pandas for 311 38 39 310. Okay 311 is here. Okay. So based on our Python version this is my SDK. I will add and save this. Now I think it should work. So this is done. We have deployed our code. It was not deployed. Now if we do the test onto the YouTube injection. Okay. Uh if I go to the logs, there will be one more log. **[2:26:02] Speaker A:** I think it will found credentials in environment variable. So this is happening. Okay. It is running. I will see if any error comes. Okay. Draw shape return to silver. Okay. It wrote to silver and it ended. If I go to silver bucket and if I refresh this, okay, I have the YouTube reference data this into the park format available. Okay, so reference data is available now. Uh raw statistics reference data is available onto the park. So this part uh of lambda to writing a park is done. Okay, we are able to read the JSON data of the reference and we are able to write that data back to the Amazon silver. So this is done. Now we will do the glue part. Okay. So **[2:26:49] Speaker A:** wait let me close uh these things extra things. I have lot of tabs uh front of me right now. So let me close them so that it's more more easier. So we are able to do this much now. Okay. As you know we able to get the data from API automatically. It lands the data on S3 and automatically triggers the data onto the JSON and it runs the parket and stores the data onto the S3. We will do the group crawl afterwards. But this is happening. Okay, we are able to do this. **[2:27:17] Speaker A:** So amazing if you reach to this section. Uh good job. Let me just have water. We have long way to go yet. **[2:27:29] Speaker A:** The reason it is taking time because uh if I was just doing this project I can finish this in maybe one to two hours explain you everything on top but we are also learning about AWS while we do this right we are also understanding about the services what is happening the UI different errors and issues I'm trying to show you so that is the reason it is taking some time so just relax if you want to uh take a break take a break and then continue Okay, but we'll move forward. Uh if you have taken break, good. But let's move forward and let's try try to write the uh glue job. Okay, now before we write a glue job, uh we want to also write this uh data quality function, right? Uh let's just check the data quality, right? Once you do this, uh do we uh get the data into the proper quality or not? **[2:28:24] Speaker A:** So that we can uh receive the notification. So go back to the lambda function. uh click onto the create lambda function. We can just finish the lambda part right now only. So put the data pipeline data quality dev uh python 311 uh execution role uh we have yt data pipeline lambda create the function. Okay. Uh configuration again update the configuration first. Uh memory you can keep 512 storage 512 is fine. 1024 to be safe. Time out to 5 minutes. Uh environment variables here. We will need environment variables as just uh SNS ARN. Okay. And you can copy this and paste this and save. Uh this lambda is created. Uh so uh we will write this lambda function code. Uh okay so this lambda is created okay for the transformation and injection for the lambda is also created uh now we will uh write the transformation afterwards okay uh to check both of the park files but right now **[2:29:39] Speaker A:** let's just go with the glue uh I think that's uh because we can finish the entire uh this uh silver layer so that we don't have to uh worry about all of the future things okay so let's just go to the glue so open the glue blue. Okay, this is a glue uh console. Go to the ETL jobs. **[2:30:01] Speaker A:** Okay. And onto the ETL jobs. We click onto the script editor. Start script. Uh we click onto the start of the fresh script and click click onto the create. Now here we have to write a script. So let me just uh get the scripts also for this. So create a new folder here. Uh we will do glue jobs. Okay. And uh new file bronze to silver **[2:30:43] Speaker A:** also silver statistics.py. Okay. And uh let me just get the code. Okay. So again this code uh gets the bronze to silver statistics data. Okay. Uh reads the CSV JSON statistics from the bronze layer. Applies schema enforcement data cleansing duplication and cleans the park to the silver layer. Again this will do the data quality check dduplication parsing handling both kegle CSV data and the live JSON data that we are getting. partition by the re uh region and uh date uh bookmarking incremental and proper login. Uh we will put these job parameters while running uh basically both database name from the bronze and uh uh this bronze database and table silver bucket and database and table path. Okay. **[2:31:41] Speaker A:** So, so this spark script. Okay. It's a simple Python also. If you don't know spark uh I will highly encourage you uh take a course uh we have it available onto the data vidya you go to datavia/python pispark uh you will learn about the pispark also we have like projects available onto the basic level projects available onto the pispark also uh this will teach you everything from basic about the pispark and long editorials about the architecture and everything is there so if you want to go deep dive into pispark you can always come to the resources this right now I'll explain you uh this code from the top uh what it is okay so this is a glue spark uh we import all of the glue related packages like spark context glue context job dynamic frame and everything these are data type included these are the parameters that we will find these are the start of the glue job that we have to define uh all **[2:32:37] Speaker A:** of the environment variables we are getting here the simplest thing is simple we are reading the data so we re read the all of the region that we have selected CA, GB, US. Okay. Uh we create a data frame of basically we read the data from the table. Okay. Uh and then uh have our data source created. Then once we have our data source created, we convert that into the data frame. Uh get the count of it and start processing all of these data. Processing basically means uh convert the ID from video ID. uh make sure our uh the date time uh I think the date format we had like mm dd uh y y y y y y y y y y y y y y y y y y y y y y y y y y y y y y y y y y y y y y y y y y y y y y y y y y y y y y y y y y y y **[2:33:18] Speaker A:** y, mm DD format, we convert that into the proper datetime format. Uh we alias some columns, we drop some columns. So this is basically column operation that we are doing. This is for the JSON file that we are getting. Else we do it for the CC file that we have. Okay. So we have like two types of files. One is JSON and one is a CS file. So if it has if it has the title or title that this way then it will process this way else it will just do the processing and know change the data type. Then the these are the data cleansing is happening. So we drop the null values standardize the region quotes to upper case. Uh I think it should be just lower case. We don't want to go the upper case. Uh and then with column uh pass the trending data from this format to the proper date format. **[2:34:06] Speaker A:** We do that. Fill the numeric value with the zero. Add derived columns like ratio or something. Then we have engagement rate also. Okay. And then uh we add the processing metadata such as what was the time of processing, the job name, do the duplication of the data. So we use window function to dduplicate some data. Do some data quality checks to make sure uh we don't have the null values into the columns and filter some of the views that is less than zero like views can't be negative. So all of these things and then we write back to the silver. So all it is is just a simple transformation script. Just go through it. Uh no one writes code anymore. Uh I just want to say tell you this is the code that I've generated using cloud. Uh if you want to understand uh again I'll create the video onto the AI using data engineering. **[2:34:58] Speaker A:** Uh you can easily do this. Uh you don't have to write all of the code. You just have to define the context and AI will generate all of the code for you. But you need to know you need to understand the basics of coding in order to understand what this code is doing. But still you can copy this code and paste it to uh AI and it will give you a complete explanation of what it does onto the deep level. But if you know basics of Python or the Spark you can easily understand all of this. Okay. And again if you don't know I highly encourage you to take a any course available in the market. We also have a course available but you can go with any of the resources that you like and upskill because this project is again we are doing a cloud project so I expect you to know all of the **[2:35:42] Speaker A:** basics of uh data engineering at least spark python sql so that you can do this project. Okay. So this is done. Uh what we will do we will put this script to the glue only. So let me just come here copy this and paste this. So the script is pasted here. Uh we can change the job name. **[2:36:07] Speaker A:** Uh so **[2:36:11] Speaker A:** give the job name as this white data pipeline bronze to silver dev. Uh IM role. We have already created the glue IM role. This is YT data pipeline blue rode. Uh all of this is fine. GX1 worker. We don't want 10 workers. We just want two. Uh other than that, advanced properties, we have to make sure we have things proper. So, um **[2:36:47] Speaker A:** yt script file name. This will put the script here. This is fine. Uh temporary location, we have it here. uh if you I think uh it will automatically create if it does not you can just create a simple bucket and put it this way most likely it will create automatically uh we just have to pass I think parameters right uh job parameters over here okay so make sure this is proper uh if you make mistake here it will break your glue script so let's just write uh onto our information mdo only Okay, first is dash bronze database. Okay, and that is our uh this one. Okay. **[2:37:44] Speaker A:** Dash bronze table and that we will have to uh inside the bronze this is raw statistics. This is a table name. So this is that then silver bucket. So bucket we already have that. Okay. And then uh silver silver database. a database. Uh okay, we have it here. Okay. And then we need silver table. Uh we will give the name as clean statistics. Okay. Uh so these are the parameters. Uh the job expect here. Uh if I go to blue uh the where's the script? Okay. Here. So these are the parameters is expect uh the table information, database information, all of the other information that it will use to write the data, read the data and update it. Okay, you have to enter it here uh in the same format. So I'll just copy this bronze database uh value as this. Okay. And then uh bronze table. **[2:39:03] Speaker A:** Again you can hardcode all of these value uh inside the uh code itself but it's a good practice to provide it on the parameter level because you can always uh change the parameter as per your need instead of changing the entire code. **[2:39:28] Speaker A:** Okay. and silver table. Okay, so this is done. Uh other than that, uh I think I'll just save this and I think we can just run this script now. Okay, hopefully uh we don't face any error. If we do face any error, uh we will take tackle that. Okay, you can go to runs. So it started running. Uh we'll just wait for this to finish. It might take one to two minutes. Okay. Uh let me just revise what we just did. So again we have data that is being put here. Kle data that is being put here. You using YouTube API we are getting both of the data for category and uh the statistics. We use AWS Lambda to convert the JSON data but we are using AWS glue to transform the both JSON and CSV data. This is just uh category data and this is raw statistics data. **[2:40:36] Speaker A:** Both are the data we are putting out over here and then we will create a glue catalog on top of both to see what is happening. Okay. So this is happening. Uh let's see let's see the runs. Let's hope it doesn't fail. Most likely it shouldn't fail. Uh if it fails uh we'll just do claw and see what is the issue. **[2:41:00] Speaker A:** Most likely uh if you face any issue the issue will be on the parameter level code is running fine because I have tested this code. uh if you have to you know build this project from the scratch it might take a week every day if you sit for 3 to 4 hours uh you can do that using AI you can finish this quite earlier if you give a proper context of what you want to do and know test the script face errors and keep fixing it so it is possible uh but this was the expanded version of our previous project so in the previous project it was quite simple I already explained you we were just using simple glue and lambda to just pass the data from three layers using Athena and we were done. But now we have like uh upgraded the uh entire project with the actual YouTube API. **[2:41:48] Speaker A:** Added the kegel data set uh Amazon S3 clue crawler here uh silver layer to transform glue into the park. Then data quality gate and all of these things are coming. Okay. So that is the reason this project is quite advanced. So uh I'm not going to onto the core level but onto the higher execution level how things are working overall. Okay. So it takes 2 minutes. I think it is running. Uh I I have a hard feeling that it will fail. Let's see. Uh fingers crossed. I'll just wait for this. If it fails, I'll see what is going on. Uh if it doesn't, then we are good. Uh if you're watching this video till here, do let me know if you reach to this part. Uh just right. I am running glue or I am running my first glue job. I will know you reach to the this section. **[2:42:42] Speaker A:** Uh and do let me know if you like this video or not. Make sure you hit that like button. And if you are someone who is interested in learning about data engineering, we have a complete platform also available uh for data engineers that is not about just learning it. We actually help you to prepare for the interviews. So we have editorials but we have a coding platform like a lead code for data engineers where you can come practice pispark, python all of the other things like posgress, dbt. We also have uh uh this data modeling uh canvas where you can prepare uh like you know data modeling rounds architecture design if you want to build architecture you can also come and build here. You can also uh review your resume and get tested by AI. Okay. Uh let's see it's still running. Oh 3 minutes it's taking long. **[2:43:32] Speaker A:** But usually uh when you run a glue script, what usually happens is uh the bootstrapping of the glue. Okay, because it is serverless, it actually has to run all of the server and it have to invoke everything the spark cluster and everything in the back end. So it takes time, right? It is not going to be like a Python or pandas uh pandas or the lambda script that just gives you the uh result right away. It is going to take 3 to 8 minutes to run the entire thing. Okay, it is successful. Okay, awesome. Uh, okay. We'll go to the silver and we'll refresh this. We have statistics data available, all of the four regions and the park data available. Woohoo. Good job. So, if this has happened right, we have crossed one barrier of the glue because most of you are going to stuck on the glue. **[2:44:25] Speaker A:** And if you are able to do this, if you are able to see the things, see it worked for me because everything was correct. I'll tell you again, I'll remind you that I have done this project previously. That is reason I'm able to do it properly and teach you this. I have to prepare the project. Uh okay. So, so our project is so so we are able to get the park data available for both of it, right? the reference data we had and the also the statistics and also it is into the very clean format as you can see the raw data we had quite messy a YouTube data API is coming the raw data is there from Kaggel everything is just you know put into the folder but this is quite clean uh we have the one park here statistics park here simple nothing uh messing it around so this is fine right our glue **[2:45:14] Speaker A:** is running properly and we are able to get the data and we'll be able to get the recent data Now we can verify all of these data onto the Athena. **[2:45:26] Speaker A:** So you can go here onto Athena. Click onto the silver dev. Okay, we have clean reference data, clean statistics. Where did this come from? From the script itself. If I go back, if I show you the script over here, it writes data to the blue context. Okay. And it creates the uh catalog information. Okay. So if we have the catalog, we can query that data onto the Athena. Okay. We have data onto S3. We have data available on we have catalog available onto the glue catalog. So that we can use Athena to directly query it. So it writes data to S3. Okay. So it writes data. So it writes data to S3 but also writes data to the chat log. So we are able to directly query without running the crawler. So I can just do preview table for the statistics and you will see all of the data are available here properly is the cleaner version and trending past. **[2:46:30] Speaker A:** Okay, this is the par state. This date that we saw into the trending date which was messy. Now we have proper date available also. So we process that. We also have this job name and the process ad information available for the further tracking. Okay, the same for the reference data. We were not able to query the JSON file but now we are able to query the JSON also because we converted that data onto the park file format and both of the data are visible. Okay, this part, let me see this part. Uh uh uh uh f color transparent. Okay, this part is done. **[2:47:28] Speaker A:** Okay, this much part we have completed successfully. Uh let me just keep this. We are able to do this have a life cycle to glacier lambda to this glue ETL to this catalog is registered automatically. Now we can start working on this and at the end we will just automate the entire thing. Everything that we are doing individually we will just do the entire thing together and we will have our final version available. So let's move forward and work on the data quality one uh that uh is pending. Uh we already created the lambda function for that. So let me just put the code here first. Uh data quality and uh uh new python **[2:48:20] Speaker A:** dq lambda.py. Okay. **[2:48:30] Speaker A:** Now this data quality okay called by the step function. Uh we will not call this uh this is just we are putting it so that when we work on a step function it will validate our data before allowing our data to send to the gold aggregation layer to make sure we have the row count enough data available. Null percentage check how much null we have. Schema validation value range check. So we want to make sure let's say views are not in the minus negative category or into like weird way like 5.2 is not a view. Okay. Freshness how recent data is enough. Right. So these are the custom logic as per the business requirement that we have to keep to make sure uh let's say we say like okay only data should pass if we have like 20,000 rows available or it should not pass or null percentage should be less than 2% only. **[2:49:25] Speaker A:** So these are our own requirements based on the business that's does the data pass the quality check as per our standard. Okay. And the environment variable is just a silver bucket and the SNS topic. Pretty simple. For each of the data quality check, we just have a uh custom logic written. So we fetch the data from the database table. Okay. And then we start uh uh querying the data and sending it to the different function. So uh we will read the sample data from the file. Uh we will uh check the row count. So there is a function available that will check the row length of the data that will return the entire check if it passes or fails and the same way it will do for the percentage schema value freshness. Again you can just go through the code. This is a simple Python code with if statement and checking if the data makes sense or not. **[2:50:20] Speaker A:** Okay. Uh what we will do we will go to the yeah data quality pipeline. Okay. Uh SNS topic is already configured here. Uh we have to uh first paste the code here and make sure it is deployed. And after that uh configuration edit and we have to add **[2:50:52] Speaker A:** uh oh oh I think yeah I think this is fine. Uh this SNS is here. Uh uh uh uh what else do we need? I don't think we need to add S3 silver bucket. Uh it will wait let me check the data. **[2:51:14] Speaker A:** Okay, it is here. YT pipeline silver dev. Okay, so it is directly reading it from here only. Okay, and table clean statistics. Okay, so this is fine. I think uh this is the name of our also uh YT. So this is glue catalog we have and clean statistics. Okay, this is fine. Okay, we don't have to do here anything here. So glue also this is fine. Uh general configuration is fine. So again this is deployed. Uh so we don't have to uh worry anything here. Okay, my voice is breaking. Uh, let me just have some water. **[2:52:05] Speaker A:** Okay. So, our code is deployed. Uh, okay. So, uh, where is that? Okay. So this much part is sort of done because this is handled inside the lambda only. **[2:52:22] Speaker A:** We can start working on the gold layer. So now we will do this. So now we will write AWS glue aggregation ETL and we will build the analytics layer. Same we will have our data available onto the AWS S3 gold layer and also automatically created uh catalog that can be easily queried by Amazon Athena and at the end we can uh basically uh orchestrate the entire thing using step function. So we have two things to do. Uh so let's get started. Uh let me open the this glue jobs. Uh new file silver to silver to gold analytics.py. Okay. Uh I'll just paste the code one by one so it's more easier to understand. First of all, we uh import all of the packages. So this uh resolve options spark context glue context job dynamic frame uh window function and sparksql. Okay, this is what we will be doing inside this. **[2:53:27] Speaker A:** So silver to gold read the clean statistics and reference data from the silver. Okay, join them and produce the business level aggregation in gold layer. So gold layer are analytics usually optimized for analytics query and Athena and Quicksite. Three tables we will produce. Trending analytics. This is daily trending summaries per region. Channel analytics, channel performance metrics and category analytics. Category level trends over time. These are the parameters we will need. Job name, silver database, gold bucket and database. After that uh we will read the uh this uh parameters that we submit. So this is how we read get resolve option system arc. And these are the argument that we want to read. Then we have to define the spark context, glue context and start reading all of these argument into the proper uh variable. So we read uh create the spark context uh context put them into the glue context create the spark session uh get the glue context inside the job register our job and set up the logger and these are the **[2:54:30] Speaker A:** variables that we will read from the parameter. After that we will read the silver tables. Pretty simple. Put the logger information here. Glue context. Create dynamic frame from catalog database table and transformation. Convert that into a data frame and log that information. And after that uh we will read the reference data and category lookup. So uh silver data reference data for the category columns. We will read same uh read from the catalog. Check the category. It will check if the ID and title are matching. then it will change the alias of that. Uh if not it will change the alias to this way. So this was a fix that I had to add because uh sometimes from the YouTube uh API the category uh ID uh was not coming and the title was coming into the different uh uh column name. So we added this small check here. **[2:55:27] Speaker A:** It will check if the ID and the uh snip.title is like this in the column then it will add the alias this way. If it is coming this way then it will add alias this way. Else it will uh just uh log the warning and if there are no category then it will create the category. This was add it just so that uh if the category is missing from it then we will uh add this check. Okay. So this will always guarantee the category always exist even if it's if it is null. After that we will create the three gold uh table. First goal table trending analytics. This is pretty simple. We just do the group by region and trending. All of the core metrics we put here add to the column and then we write that to the S3 file and also the glue context. **[2:56:18] Speaker A:** Pretty simple. This is a simple SQL operation. You can write it into the SQL also. But this is a simple par script. Get the data frame group that okay add column. This is a path. put that onto the S3, put that into the group uh catalog. Same goes for the channel analytics. Uh similar uh functions will be applied. Uh group by uh title and region all of the metrics, average, sum, uh max, minimum, whatever it is. Uh do the ranking. Uh write that onto the S3 and catalog. Same goes with the third one which is category analytics. So group by partition this uh write this and at the end you commit it. So this is the entire silver to gold. Uh we will do uh paste this code onto the blue uh ETL. So I think it is pretty simple. We should know that. Go to ETL jobs script editor create a script. **[2:57:21] Speaker A:** Paste this script. Okay. Uh we will update our uh script name. Wait for this. **[2:57:34] Speaker A:** Uh the job name is this. Okay. Uh then we have to choose the IM which is glue data this uh request uh worker. Let's go with two. uh after that uh we have to paste the this one which is the parameters that we need. One is dash dash silver. I think we already have this. Let's just go here and copy this from here. See, one is we need silver database. Second, we need gold bucket. **[2:58:15] Speaker A:** Okay. And gold database. Okay. So, let's just so it will read from silver database. Okay, we have it will put data to the gold bucket and gold database. It will create the table and the catalog. This is done. We will save. Let's hope we don't face any issue. We run this. This starts running. Now, if it is successful, uh basically uh what will happen? Okay, the data that is stored here under the silver, it will read that. It will create that into the three different tables. It will put the those three different uh table information here also and the Google blue catalog. If this is successful, then we'll be done with this and sort of done with this also. Okay. So, let's just wait for this to finish and if it is finishes then all we will have to do is understand about the step function. **[2:59:17] Speaker A:** So, while this is running, okay, this will take some time. If it fails, we will tackle the issue. But uh if it is successful then we are done. So we sort of done everything. Uh I don't think anything else is remaining here. Uh if it is then we will check that. But I think most of the things are done. **[2:59:35] Speaker A:** Uh let's continue with our theory. Okay. Till this is running. So I think we have completed this. Uh let's understand about the quicksite and kinesis till our glue job run is runs. Uh we have Amazon quicksite. This is basically a visualization tool a cloudnative serverless BI service. You can build interactive dashboard and visualization. ML power analytics so you can do anomaly detection and forecasting. It has spice memory. Basically, you can bring your data from the source to the quick side so that it's much more faster. You can connect to Athena, S3, Red Shape, RDS and more. You can also embed your analytics onto the apps and pay per session pricing is also available. Can is bas basically the real-time data streaming uh just like Apache Kafka. It has three things. Data stream uh custom real-time data processing uh with shards. Fire hose is basically the message delivery. **[3:00:26] Speaker A:** So all the data that comes if you want to deliver somewhere to S3, red shift or open search you can do that and you can run SQL queries on top of the real-time data using data analytics. So this is the kinesis. It's not part of this but uh in future maybe I'll create some project onto on this also. Okay. AWS Lambda compute service. Uh databases also we have to cover. Uh RDS is our relational databases like MySQL, Postgress, SQL, Oracle, SQL, Mariab all of all of these are available. Automated backups, patching and scaling is given by AWS. Multiple available read replica for read scaling. So this is the database as a service given by AWS. Dynamob is basically the NoSQL database uh with key value pair and document. uh we might not use this that much. Singledigit millisecond latency to any scale on demand on demand or pro provision capacity is available and global tables for multiple region replication. **[3:01:25] Speaker A:** Red should be red should be we already looked into it. It's a data warehouse for AWS column storage for massive parallel processing spectrum for SQL S3 queries. We looked here uh it was here I think. Yeah, red shift spectrum and uh RAC nodes decouple compute from the storage like if you want to separate it and AWSDMS migration service if you want to migrate data from one source to another source you can easily do that supports both homogeneous heterogeneous so if you want to migrate from onremise posgress SQL to RDS poses SQL it supports that or on premise my SQL to RDS posgress SQL it supports that like homogeneous and heterogeneous it also has CDC So continuous uh data capture. So anytime the changes happens on the source, it also gets replicated and you have a tool also available schema conversion tool for migrating schema. So this is a core of uh AWS again uh this is just an overview again uh in this masterass we wanted to mainly cover the tutorial side of it the theory little bit also hands-on while looking at the **[3:02:33] Speaker A:** UI and also execute the project. So I've tried to cover all of this together. Uh if you want the detailed version of course you can uh check the learn section onto our website that goes into the detail and there were there will be lot of modules added in the future with the videos and all of the other things. So that is for that but for now for this YouTube video I think this is good enough to kickstart your career into AWS uh with a good project available and this project is quite complicated also. So we were able to cover everything. Okay. Now the this job is successful. So if I go to my buckets, if I go to gold, I should be able to see uh YouTube three data, category analytics, channelics, extending. This all will be park data available based on the region. So this is done. **[3:03:23] Speaker A:** I can go to Athena. I can change to gold layer and all of the three tables are available and I can now start uh querying for example let me do this okay uh top trending views uh top trending days by view so I'll do select region uh trending date pass okay this is the past date we have total videos uh total total views um average views per video, average engagement rate unique channels. Okay. from trending analytics where where uh region is equal to US order by total views at let's just say limit 10. Okay. So uh this query will give top trending days by the views. So if I uh put this is this okay here. So based on the days okay **[3:04:50] Speaker A:** so this is the so this day is the top top trending for the US. Uh then after that it was like you know 11th 112 then 82 then 102. So total video 17 total views are this average video per average views per average views per video is this much engagement rate is high unique channels are 17. So like this you can also find top channels. So say this is a query I can get all of the information channel analytics and US. Okay I'll do this and these are the top channels in the US. Let's say let's do India. Okay. I am and the T-series is the top channel uh with these number of views. Then we have the Marvel Entertainment, Fox, Amit Bana. This is like 2018 data. So makes sense. Speed records, gold mine, Zilm, YouTube, Spotlight, Sony Pictures, Entertainment, everything. Okay. You can also uh category performance uh also you can check. **[3:05:57] Speaker A:** So if I do based on the category, okay, and get all of the metrics. Oh wait. Yeah. So entertainment is the highest category. Uh then we have the music, film, animation, music, uh comedy, news, politics, people. So most of the category are the consumed by the YouTube people is the entertainment. So I'm understanding okay entertainment is a good way to start. Then we have the music, film, animation have so like that we have in India it's entertainment you can also go with the specific region also in GB it's music uh then in US it's entertainment again people blocks music okay so we can understand okay uh what is going on we can also do the cross region uh comparison if I want to do uh so uh you can see average engagement highest engage engagement is in the GB. Uh lowest is in the India. **[3:06:58] Speaker A:** Data points are this much. Cumulative views, highest views are here and the lowest are actually here in see here. Okay. So this is cross region uh comparison. You can also check viral channels like this like you can get the channel title engage that from uh group by having uh region. Wait, let me run this again. Uh so highest uh viral channels like uh trending in three plus regions is SpaceX, then PewDiePie, uh Universal Pictures, Paramount, Vivo, all of these are the uh viral channels. Okay. Okay. So these are the analytics, right? Simple basic SQL queries. Uh again you can chat GPT all of these queries uh write it and get more analytics. Uh but what we want to do we want to move to the next section. Uh again these are the basic stuff. So we don't want to spend time we want to learn AWS. **[3:07:52] Speaker A:** So that is what we are focused on. Okay. **[3:07:56] Speaker A:** So last thing okay we have to do here is this one **[3:08:08] Speaker A:** AWS step function. Okay. What is AWS step function? It's like an orchestrator. The AWS step function is a serverless visualization workflow service that enables developer to build distributed application, automate processes and orchestrate over 22 220 services using state machine. Okay. So step function basically it starts like this. You can have a workflow like this. You will choose your path then call the HTTP endpoint extract something here. Do invoke lambda function do something. Basically if you know about airflow uh similar to that but using AWS services so you can orchestrate the entire thing uh for that uh which will go to step function. Okay **[3:09:02] Speaker A:** click onto the get started. Uh you can run the simple hello world. You can choose existing template or create your own. Uh we we will create our own. So uh wait. **[3:09:19] Speaker A:** So there so this is what so this is what it looks like. Okay. These are the different actions available on the left uh that we have. Then we have all of the flow available like choice parallel at separate branches. If you want to map out something, we want to wait, success, delay. There are also pattern available like process ASV file from S3. you can directly track that and uh put the information right information do the configuration from the design angle and then there is a code also now before we do anything with the uh AWS glue step function we also need a IM role for this because it should be able to access all of these services that we are calling like we'll be calling AWS Lambda we'll be calling glue and the SNS so we have to create IM role for that so what we will do we will go to IM am I am uh wait let me just see yeah there's IM available we go and create a role here **[3:10:20] Speaker A:** wait for this yeah uh then inside the role we will find step function click on to next uh click next wait Okay. **[3:10:43] Speaker A:** What is this? Why it is giving lambda road step function. **[3:10:54] Speaker A:** Okay. Uh go to next. Uh then just give the name as YT data pipeline step function role dev. Uh click onto the create role. Uh find that again. Click here. Add inline policy. Okay. And paste this to the JSON version. Okay. Uh invoke here. Uh you will need to add account. So I will add my account here uh and here also. So it invokes uh gives the permission to invoke lambda function. It gives the permission to start AWS blue job and also uh publish messages to the uh SNS notification. Okay. Next uh uh policy name uh data pipeline S SFN access. This is done. Okay. So now we will go to step function. Okay. to write step function we have a JSON available for that also. **[3:12:03] Speaker A:** Uh let me create a new folder here step functions and pipeline or gestration dojson. **[3:12:22] Speaker A:** So first of all we have the command. After that we have the start at. This is a standard format like a start at where uh you define where your actual execution start. Then within the state you define all of the states that we have. States are basically invoking lambda function executing glue job. All of these are the different state. Okay. So the first step is inest from YouTube API. This is a custom name that we give. Okay. type is task resource is a AWS ARN lambda invoke and then there are parameters. Parameters are basically the function name. So here we have to define the ARN of the function that can be uh that you can go to the lambda function. This is our data pipeline YouTube injection div and you can copy the function ARN and paste it here. After that we have the payload. **[3:13:11] Speaker A:** So triggered by step function that we have to add and this is the execution ID context variable. So all of the uh when when this executes right it will keep the execution ID for its reference and the result of this will be stored inside the injection result. This is again custom name you can keep whatever you want. Okay. So this is your result path. Uh so after the result path we want to retry and catch. This is like a airflow only. So if you want to retry okay uh you define the different errors that uh it can catch maximum retries it can uh so what whenever this happens it wants to retry after these interval seconds and in case some error occurs okay it will store that error information into the this part and then we define our next what the next thing we we need is the wait for S3 consistency reason when we execute our YouTube API it **[3:14:06] Speaker A:** stores data onto the S3. So we want to wait for some second, okay, till the data is added to the S3. So wait for S3 consistency. Uh this is a state available into here also. If you go to design inside the flow, you will see choice parallel, there's a weight available also. Uh there is parallel available, pass available, success and failure available. The same way for uh the type is weight. The type of this if you see if I drag this here. Okay, if I go to the code you will see the type of this is wait. Okay. So we are just using the state type wait seconds 10 comments is just wait for the brief weight for the S3 consistency and we define the next which is the process in parallel. Again this is a name given by us process in parallel. We use type as parallel. **[3:14:58] Speaker A:** uh if I put this here you will see uh inside the code the type is parallel. Parallel basically means I want to execute two different machines. Okay, this is just simple command. Then we have two branches. One transforms the lambda function one over here and second is for the glue job. So here we will have to define few things. So starts state is transform data. Inside this we have the task which is like lambda invoke same one nested JSON. Here we have to define the write a arm which is JSON to park. Uh let me copy the ARN for this and I paste it over here. Triggered by step function. We will put the result here again. We will do the retry. Uh and is true. Then for the glue job uh task is glue job run. We'll have to define the glue job name here. **[3:15:53] Speaker A:** So I'll go to the glue and the glue job is uh bronze to silver one. Uh we will have to uh let me check if it is a job name or the ARN. **[3:16:08] Speaker A:** Okay. So here here we have to add blue job name. **[3:16:18] Speaker A:** and all of these arguments that we were passing. So arguments we can find it here. Uh this is my bronze database. **[3:16:36] Speaker A:** Wait uh let me copy from here only. Okay. a bronze database, a bronze table, **[3:16:49] Speaker A:** uh, silver bucket, **[3:16:56] Speaker A:** uh, silver database **[3:17:03] Speaker A:** and silver table. Okay, result path we store result. Uh we also put this into uh retry. Uh both result is added over here. And next is the run data quality check. The same thing we define a task as lambda. Here we will have to add a data quality one. So I'll go to lambda uh data **[3:17:30] Speaker A:** quality pipeline div. Get the function arn. Paste it over here. Payload silver. This is fine. Tables. These are simple payload result. Yeah, we will have to update the silver database here. Okay. YT pipeline silver dev. Uh this is fine. All of this is fine. Result path dq result uh evaluate data quality is done. We have a choice. Uh if the data quality is passed, boolean is true. Uh then next is run silver to gold or we notify the failure. Uh run silver to group job. We'll have to provide the job name here. Uh I'll go back silver to gold. Job name uh silver bucket, gold bucket and gold database. Okay. updated this. Uh this is fine again. Retry cache error notify success uh SRN topic. Uh let me copy this SNS of SRN. If you want to notify it is successful. If it fails, okay, publish here also and here also. **[3:18:59] Speaker A:** Okay. and here also. Okay. So in all of the places I just have to uh provide the uh SNS, SRN for notify success, notifying failure, transform failure, DQ failure, gold failure. Okay, these are the different failure states I have added. I think pretty much all we have done. Pretty simple. Uh let me just go to the step function. Add this here. Uh see it created the automatic graph for us and what we will do we will click onto the create permission for this cannot be generated stress publish notify topic ARN is invalid why uh we have copied the right ARN **[3:19:58] Speaker A:** per uh step function start. Let's just go with this and see if we are getting any error. Detected at name satisfy members must have one name. Okay. Uh uh name name. Okay. All of these are fine. **[3:20:23] Speaker A:** What is the issue here? Error creating your state machine level value one validation error detected. **[3:20:35] Speaker A:** We have any issue here. **[3:20:52] Speaker A:** This fine notify is fine. Alert tab. This is fine. This is fine. This is also fine. **[3:21:14] Speaker A:** Okay. Yeah, I haven't added the name of the state machine. Oh, uh we will give the name as YT data pipeline orchestration config. Uh IM role. I think we have to use this one. Uh and I think yeah, we're good. Okay. Uh let's just do the start execution. Uh we don't have to do anything. Just start execute. Uh graph level view. We will see this now. Okay. So it ran the first function successfully inest from uh API waiting for the history consistency. Uh this ran uh this might have started running. So you can actually go and check. So bronze to silver there will be one more run. Uh here okay it is it is running. Uh so all of these things are automatically happening. Transform it might have already run. So inside the transform Is that uh JSON to park? This already ran. Uh this is running. **[3:22:20] Speaker A:** So it will take around 3 minutes to finish this. So this executed automatically this executed. This will execute. If the data quality check fails, okay, it will notify us over the email that okay that the data quality check failed and then it will move uh move to the next one. Okay, which is evaluate the data quality. After that it will go to the run the silver glue and then it will uh give us the final notification. Okay, so let's wait for this to complete **[3:23:06] Speaker A:** uh 54 second again. uh while this runs uh I just uh give you the overview that uh again this is quite complicated if you are looking at for the first time and you don't have to write all of these by yourself all you can do is just explain uh in this age explain AI about your entire context that I have created this this lambda function and this is my architecture can you give me the uh this state machine for AWS step function and it will generate the entire template for to you all you have to do is just replace the right things at the right place and you will get the entire thing. Uh earlier we used to have you know trial and error version where you used to spend time on UI or on the JSON to modify things and then execute but these days the code generation part is completely automated. **[3:23:58] Speaker A:** You don't have to do anything by yourself. You can actually automate this entire project using just terapform script also. But again the goal of this project was to teach you on execution side of the AWS from the console and also teach you about AWS from the scratch. So this was the core idea of the project. Uh that is the reason uh we spend time onto this. Uh again this is running 2 minutes 25 seconds. This will success. Most likely what will fail is the data quality check. Uh we haven't done the data quality check on our side. So this will do the data quality check after it puts the data here. If there is any error, we will get to know that. Okay. Uh one more minute. Okay. So, this job is still running. I think it will success in few seconds only. Um new data might have got added. **[3:24:54] Speaker A:** So, just waiting for this to finish. Okay, this is successful. So this will uh get the result here and this will do the data quality check. Okay. **[3:25:22] Speaker A:** Okay. So done data quality check failed. There was some issue here. No flag **[3:25:50] Speaker A:** logs. Let's check this. Lambda inputs no module name AWS Wrangler. Okay. The issue uh might be with the lambda not with the data. So let's go to the quality def add the layer go to below edit add layer AWS wrangler. **[3:26:24] Speaker A:** Uh we can add Python 311 SDK pandas. I think this should be fine. Uh save. We can test it here. We did not test this actually. Uh pandas. Not sure about Wrangler. Where is that? Let's see. **[3:26:52] Speaker A:** Okay, now it should actually work. We can rerun this I think. Action test state start test. Okay. **[3:27:11] Speaker A:** When the public topic error type **[3:27:20] Speaker A:** test input state definition invalid parameters it's not executing this table view let's just uh test here itself first make sure this is deployed test create new test event test save and invoke. I think it requires a custom parameter so might not run it. Uh uh invalid parameter topic name publish operation topic name parameter. Okay. Oh okay. I think **[3:28:17] Speaker A:** topic **[3:28:21] Speaker A:** set this topic topic name issue SNS topic right uh it is coming from SNS topic ARN configuration environment variables SNS alert topic ARN. I think this is fine. Let's just check one more time. SNS if we have the right topic ARN. Uh this is the ARN. Okay. Uh okay. Yeah, we had a wrong ARN added. So I copied the right ARN from here. Uh that could be the case of here also. Uh when we created our machine, we might have added the wrong ARN. I go to state machine orchestration. I edit it here. Go to code and here. I think we added the long one. Topic ARN is this uh let's save. I think this end part is not needed. We can remove this end part **[3:29:44] Speaker A:** here. **[3:29:48] Speaker A:** Okay, this is done. **[3:29:54] Speaker A:** This is done too. Is there any other place it has? Function name is fine. This is fine too. Okay, I can save this. **[3:30:13] Speaker A:** Okay, let's execute this again and start execution. Okay, so again this is start from scratch. Most likely we had issue onto this lambda. Now if we test it, I think it should work. Uh **[3:30:34] Speaker A:** this failing publish operation is not authorized to perform onto this resource. So I think it's a uh again the IM we might have to update it before it reaches configuration permission. this IM role. Uh basically I'm going to the IM role of this and I think we created inline policy. Uh but we have to update uh this topic SNS this one. Yeah, this is not right. Oh, this is we have to put here also. Next. Save changes. Now if I test this. Okay. YT test. Let's test this. I should have tested this before. I thought it will work. Uh I'll say oh now it's working. Okay. Total checks checks pass zero. That's fine. If an error occur denied ex exception Athena get work. After you administer you have to update the permission. So it is not able to access the Athena. uh we can add a permission here itself onto the data pipeline lambda dev role Athena for let's just give Athena full access for now uh and uh let's test it one more time **[3:32:04] Speaker A:** local checks checks passed that work group Athena work group resource weird we just gave the access Oh, I think it might take some time to reflect maybe. **[3:32:22] Speaker A:** Okay, let me just uh copy this to close. Okay, and what we will do, we will uh copy the inline policy also. Basically uh if any error comes right this is uh uh you put this uh make sure there are no sensitive data right now there's nothing uh so again we have to add this give me complete policy and this will give me the complete policy I just have to paste this as it is copy **[3:33:10] Speaker A:** Paste this. Next. Save changes. Okay, this is done. Now let's test it. I gave the access. **[3:33:22] Speaker A:** The hell **[3:33:29] Speaker A:** work group get work group on resource. I think it might be issue with S3 cuz we have limited permission onto S3. Could be that. Let me test uh access denied exception. This will run. Oh, that's weird. Still getting the same. Still the same error. Always **[3:34:08] Speaker A:** work with a specific group. The action work group ARN **[3:34:15] Speaker A:** the key change resource with explicit work primary default work replace primary with the actual work groups the query execution uh custom workbook not primary. Okay. Let's see. Let's go to Athena. Oh, work group is primary. Okay. Uh I don't think there's any issue with that. Uh default replaces by and data calling needs a query execution. **[3:34:51] Speaker A:** This is weird. Should not happen. **[3:35:05] Speaker A:** And this will execute. **[3:35:09] Speaker A:** Uh uh uh let me save this. **[3:35:16] Speaker A:** Save changes. **[3:35:20] Speaker A:** Let me test this. Uh same exception. Add exception. Let's just add admin permission for the test to verify if this is a permission issue or something else. **[3:35:36] Speaker A:** Uh same. Okay. Uh wait, let me just put the code. This is my code. Uh even after adding admin permission, still still getting the same issue. might be some issue on the code level. So default workflow implicitly you need to call explicit work group by Athena query update the read SQL query. Okay. Okay. Uh we have not added this the Python and update your S3 location and use them with the query. Give me complete code internally to validate the fetch output even with the resource I have some requires to be attached and refreshed. Okay. Okay. Okay. Okay. **[3:36:34] Speaker A:** H. So that means uh this needs to be updated uh the work group the query location **[3:36:48] Speaker A:** primary is the work group Athena S3 output add Athena work group S3 output and region environment variables on the **[3:37:10] Speaker B:** Okay, let's try this. Okay, data quality put this. I'll deploy this. I'll go to configuration environment variables edit and put Athena work group uh primary put Athena S3 output **[3:37:44] Speaker B:** as this Huh? **[3:37:52] Speaker A:** And visions **[3:37:58] Speaker A:** on **[3:38:06] Speaker A:** Athena. I plan to Athena **[3:38:11] Speaker A:** South one blue Athena query result. Oh, this is fine, I think. **[3:38:24] Speaker A:** Okay, **[3:38:31] Speaker A:** is not possible. **[3:38:38] Speaker A:** Let's just put this as it is. Okay, deploy this. **[3:38:47] Speaker A:** test this. Come on. **[3:38:57] Speaker A:** Okay, now it is passing. Okay, so I think the issue was most likely the permission issue or maybe we had to explicitly define the Athena. Uh if I remove the admin policy here and AWS policy here uh maybe it should work again. So if I test it. **[3:39:19] Speaker A:** Yeah. Uh it was not the policy issue but the code level issue. Okay. This is running. Okay. Uh SNS authorization DQ failure. So uh data quality was successful but there was a issue in this **[3:39:42] Speaker A:** cause data pipeline **[3:39:49] Speaker A:** where is this notification SNS publish **[3:39:58] Speaker A:** let's check this SNS publish Publish for assume role. Okay. Uh notify DQ failure. Okay. Fine. Notify DQ failure. **[3:40:15] Speaker A:** This is fine. State publish. This is exactly same SNS ARN. Topic ARN is fine. I think what's the issue? **[3:40:34] Speaker A:** Uh this is subscription. Okay, subscription is this one. This is topic one. I can copy this. This is ARN. Right. Okay. And this is pasted here. This is fine. Subject is fine. This is fine. What's the issue? Mr. Hu **[3:41:04] Speaker A:** data pipeline because no identity based policy allows publish action. Okay. Okay. Okay. Okay. That means if inside the config we have IM policy available which is this one. And if I go to I am role of this and check if we have permission given. Yeah, we have given uh let me edit this **[3:41:42] Speaker A:** uh and put the SNS AR proper one. **[3:41:52] Speaker A:** Okay, this was missing I think. Next save changes. Okay. Now I have given that and if I tell to retrive from failure retrive execution. **[3:42:27] Speaker A:** starting retriving. Okay, now. Okay, it's working now. Finally. **[3:42:34] Speaker A:** Okay. So, this was successful. This was successful. All the test cases passed through. So, if I go to Gmail, **[3:42:50] Speaker A:** I will say I will get this warning data quality check failed because uh uh there was something happened but data quality check actually failed because of the some reason. Okay. **[3:43:09] Speaker A:** What is the reason? Quality pass zero check to check false may occur with excess den. Okay. Uh the same issue happened on the lambda because of the this part and that is the reason it failed. I think we have fixed this issue. So if we do the new execution and start the execution now the entire pipeline will be successful. Okay, let's wait for that and let's see what happens. Uh I'll wait for this to complete and if it is successful I'll show you if there are any error we'll continue from there. Again I ran the script uh but there was an issue and the issue is still related to some permission that it is not able to read this bucket and looks like our uh IM onto this data pipeline. Okay, does not have permission to read uh the what is that? Let me find this. **[3:44:11] Speaker A:** Yeah, this does not have permission to read the S3 bucket where our Athena results are being stored because inside our inline policy. If I edit this, we have just added uh silver and the bronze bucket. But we also need to add the information about the Athena. Okay, over here. So just do we are inn this and uh where is Ailina the bucket name **[3:44:58] Speaker A:** result star comma copy this paste this and then save it. **[3:45:17] Speaker A:** Okay. And if I I think test this state again maybe uh data is still not passing even after updating it. **[3:45:36] Speaker A:** I go back. I finish test **[3:45:50] Speaker A:** uh YT result S3. Okay, let's just add this into the step function. Uh I am also **[3:46:12] Speaker A:** let's just right now add a complete S3 access to debug this S3 full access. And if I now run this **[3:46:25] Speaker A:** start test **[3:46:30] Speaker A:** still fails. **[3:46:35] Speaker A:** Input is fine. **[3:46:40] Speaker A:** The Q result is failing. **[3:46:56] Speaker A:** Table view **[3:47:06] Speaker A:** access denied for this. Show bucket is encrypted. Canot read access denied. Ah the full access is given. **[3:47:24] Speaker A:** I think both of the lambda has full access. Let me give lambda also s3 full access for now to check if there is a permission issue. Okay. And I will run this one more time and see if this is working properly. uh new execution start. Okay, let's just wait for this and I'll continue from here. So in this one I can see uh the data quality check passed and now it is running this glue uh which is this one the second glue job that we had. So which is silver to gold one and it might be in the running state right now. I execute this. This is running. So it will take one and half minutes and this might send if it is successful notify us success and we'll receive a su successful message. The most likely the issue was with the permission I added the full access but I think in the both step function and in the lambda function you will have to give the s3 access to the resulting bucket so **[3:48:32] Speaker A:** that uh it can access the data from there and execute it. Okay. So this was the issue I think uh on the step function we face lot of issue because we were orchestrating everything and for that we also need to uh provide the right setting when we access uh when you you know execute all of this individually it works. Uh but when you try to uh execute it together there are chances that some bucket might not have access to each other and that is the reason we have to provide the all the permission. So waiting for this to finish. Uh I think uh this will finish in few second and we will have our uh uh this completed and we will receive the final notification. Right now we received all of the fail notification only uh because of the denied access but now we will receive a final uh message. **[3:49:22] Speaker A:** Let's just wait for that uh if it is complete in running state. Once this is done, I think uh we are done with our project mainly because this will be done also. Uh we use SNS, cloudatch, IM for most of the things. So our entire project will be complete and we also covered most of the things here uh about the AWS core for data engineers. Again, if you want to learn more about AWS, I have few projects available online. You can check them out. If not, uh best is you can prepare for the certification. uh uh I'll recommend you AWS certification if you want to deep dive. Uh the simplest certification to start with is the cloud practitioner one uh that will give you the basics but if you want to jump to you can prepare for AWS data engineer associate that will help you to learn **[3:50:14] Speaker A:** more about data engineering services. You can just YouTube you will find bunch of tutorials uh on data also if you want to learn basics of data engineering from start to end we provide everything. So this will also help you. I think our execution is complete and we received the notification pipeline completed successfully. So that means if you check this state notify success is executed. Okay. After this so our pipeline is running fine. So this was everything. We have completed this entire architecture. Let's just revise it again. What happens? We have data coming from Kegel data set. Initially we created buckets and we using the bash script we uploaded all of this data to Amazon S3. Same way we ingested uh created a lambda function and ingested data to the Amazon S3. Once that was done, we wrote the lambda function to transform our raw JSON to the park format. **[3:51:06] Speaker A:** Same way we wrote the glue function in pi spark to transform CSV and the JSON file and put that into the Amazon S3 and the glue catalog. Here also we executed crawler onto the glue raw data so that we could uh understand what the raw data is. Here we get the clean data. We had the data quality check where that just checks uh the issue with the data. If the views count are less, if there are no lot of null values and all of the other things and uh sends the notification according to that. Once that is done, we do the gold year onto Apache Spark where it reads both of the data creates and aggregates them into the three different categories create table onto catalog put that onto the Amazon S3 and we can query that using Amazon Athena. Same way uh we have step function that executes the entire thing in one shot. **[3:51:56] Speaker A:** We faced a lot of issue over there but we were able to tackle it out using AI and you based on some common understanding and we fixed it. So uh the entire project is completed. Uh now next step what you can do you can clean your entire as account basically delete everything if you want to delete it. If you want to keep it you can keep it. I don't think you will get charged uh because we haven't like uh you know automated this entire thing right now. So you can keep it, you can delete it up to you. If you want to uh keep uh clean it, you just have to uh go to the each and everything that we did. Go to IM, delete all of the resources, go to S3, delete all of the buckets, go to Lambda, delete and go to glue to delete the ETL jobs. **[3:52:39] Speaker A:** Just delete it everywhere. It will clean clean it up. Okay, so this was everything about the AWS. Uh again this is just an overview of AWS. I just touched a few services but if you want me to create part two of this uh just do let me know. Probably I'll upload the part two of this video onto the uh my website itself data onto the cloud infrastructure. So you will find part one and part two and multiple parts of this and individual videos for all of these different things here itself. uh do check that out and app learning. So this was everything and I'll see you in the next video.