# Jarvis Runs 85% of My Business - Here's the Exact 6-Layer Stack **Channel:** Kimball AI **Source:** https://www.youtube.com/watch?v=z6jFqt0QwIE **Transcript page:** https://www.withtranscript.ai/video/z6jFqt0QwIE ## Chapters - 0:00 — What Jarvis Really Is - 1:32 — Building the Core Agent Stack - 4:12 — Models, Tools, and Safe Access - 6:39 — Instructions, Memory, and Automation - 9:38 — Voice, Workflow, and Final Principles ## Transcript **[0:00]:** Jarvis runs about 85% of my business. These are the exact tools I use to manage it. Everybody keeps building these beautiful Jarvis dashboards. They have glowing circles, voice animators, moving charts, and a giant button in the middle, but most of them do absolutely nothing. The dashboard is not Jarvis. Jarvis is the system running underneath it. It is the computer that stays awake, the AI model doing the thinking, the connections that let you use the tools, and the schedules that allow it to complete tasks without you ever opening a chat window. So, in this video, I'm going to show you the actual architecture for building your own Jarvis. Not just something you can talk to, but something that can actually monitor your business, complete repeatable work, remember what happened, and bring you the things that need attention. To make this simple, we are going to build this in six layers, Jarvis, J A R V I S. **[0:46]:** Before we start, let's define what Jarvis actually is. Jarvis is not one AI model. It's not ChatGPT, it's not Claude, and it's not a single piece of software. Jarvis is a system made up of multiple components. You have a computer that stays online, an agent harness that controls the systems, an LLM that acts as the brain, and connections that give it access to your tools. Then, you have instructions, memory, skills, and schedules that allow it to repeatedly perform work. Think of it like building an employee. The computer is the office, the LLM is the brain, the tools are the employee's hands. The memory is what the employee knows, the SOPs explain how the job gets done, and the schedules tell the employee when to show up. If one of those pieces is missing, you do not really have a Jarvis. You literally just have a chatbot. **[1:30]:** So, let's build the stack. **[1:32]:** J is for job runner. The first layer is the job runner. This is the computer that keeps the entire system alive. For most people, the easiest option is something like a Mac mini. It's small, quiet, relatively power efficient, and it can stay plugged in 24 hours a day. You can also use a Windows computer, a Linux machine, or a cloud server. The specific machine is not the most important part. The important part is uptime. If your agent is supposed to check your email every a minutes, it literally cannot live on a laptop that's asleep inside of your backpack. If it's supposed to create a report every morning, the system has to be online when that report is scheduled. That's why I like having a dedicated computer. It becomes a physical home for the agent. This machine holds the files, runs the automations, and keeps everything operating while your main computer is turned off. **[2:14]:** You also do not need the most powerful computer in the world. If you are connecting OpenAI or Claude through an API, most of the actual computation is happening in the cloud. The computer is mainly just coordinating work. However, RAM becomes more important if you want to run local models. A local model is an LLM running directly on your machine instead of sending every request to another company. That can be useful for repetitive work, private information, or tasks where you don't want to pay for every API call. The trade-off is that local models normally require more RAM and may not be as capable as the best cloud models. So, your first decision or both. Use a local model for cheap repetitive tasks, then escalate difficult work to stronger cloud models. The second layer is the agent harness. An agent harness is the software that sits between the AI model and the rest of your computer. **[3:00]:** You could use something like Hermes or another agent framework that can access files, run commands, use tools, and perform multi-step tasks. The harness is what turns an LLM from something that answers questions into something that can actually take action. Without a harness, you can ask an AI to write an email. With a harness, AI can read the email, check the relevant context, create a draft, and bring it to you for approval inside your Gmail. That is the major difference. The model provides intelligence, the harness provides agency. The harness controls what the model can access, which tool it can use, how long it can work, and what happens when it encounters a problem. This is also where loops become possible. Instead of giving AI one prompt and receiving one answer, the harness can allow the model to read the task, create a plan, use a tool, review the result, and continue until the task is complete. **[3:47]:** That loop is what makes agents feel different from from chatbots. But, this is also where people get into trouble. Do not give your agent unlimited access to everything on day one. Start with a small sandbox. Give it one folder, one email label, and read-only access wherever possible. Then, expand its permissions as you understand how it behaves. An agent with full access and bad introductions does not become more useful. It just becomes more dangerous. **[4:12]:** R is for reasoning model. The third layer is the reasoning model. This is the LLM. LLM stands for a large language model. You can connect your harness to OpenAI, Claude, Gemini, or a local model running on your computer. And, you do not have to choose only one. One of the biggest advantages of using an agent harness is that your entire system is not locked to one AI company. You can route different tasks to different models. A small model could classify emails, a local model could organize information, a stronger reasoning model could analyze a complicated problem. Think about this like assigning different employees different jobs. You would not pay your highest-level strategist to rename files all day. Your Jarvis works the same way. You can create a basic routing system. Tier one tasks are cheap, repetitive. Tier two tasks require better writing or judgment. Tier three tasks require deep reasoning and should only run when necessary. **[4:59]:** This matters because API costs increase quickly if every tiny job is being sent to the most expensive model. The goal is not to use the best model for everything. The goal is to use the least expensive model that can reliably complete each task. The fourth layer is virtual connections. This is where Jarvis gets access to the tools you already use. There are two acronyms you need to understand here: API and MCP. API stands for application programming interface. An API is essentially a structured way for one piece of software to communicate with another. An email API could allow your agent to read messages or create drafts. A CRM API could allow you to update a contact. An analytics API could allow it to retrieve business data. MCP stands for model context protocol. MCP gives AI models a standardized way to connect with tools and data. Instead of creating a completely custom connection for every application, an MCP server can expose specific tools that the agent is allowed to use. **[5:51]:** For example, you could connect your agent to your email, CRM, file files, browser, and analytics. This is the layer where the agent gets hands. But access should be intentional. For each connection, decide whether the agent can read, write, delete, or send. Reading is normally lower risk. Writing introduces moderate risk. Deleting and sending should require the most caution. For example, my agent might be able to read an email and create a draft, but I would not immediately allow it to send that email without approval. That approval step is called HITL, which stands for human in the loop. HITL means that AI completes most of the work, but a human remains involved at an important decision point. The agent might prepare a proposal, but you approve the pricing. It might draft an email, but you approve the message. That is how you get the speed of AI without handing over every important decision. **[6:39]:** The fifth layer is instructions and memory. This is probably the most important part of the entire system, so listen up. People assume the intelligence comes from the model, but the quality of the system often comes from the context surrounding the model. Your agent needs to know who you are, what your company does, what you are currently working on, how you want things done, and what it should never ever do. The easiest way to start is with markdown files. Markdown is just a simple text format. You do not need a complicated database to begin. Create one main file that explains the business. Include the company overview, current goals, products, ideal customers, and important rules. Then create SOPs for individual tasks. SOP stands for standard operating procedure. For example, an email SOP could explain which emails matter, when the agent should create a draft, and which messages it should never answer. **[7:26]:** A reporting SOP could explain which KPIs to retrieve, what counts as an unusual change, and how the report should be formatted. KPI stands for key performance indicator. These are the numbers that show whether the business is moving in the right direction. Your agent should not just collect numbers, it should understand which numbers matter. The same thing applies to content. You can create a content SOP that includes your audience, tone, examples of successful content, and phrases you don't use. This is where people make a major mistake. They write one giant prompt and expect the AI to remember it forever. Do not treat this like a prompt. Treat it like an employee training. If you would explain something to a new employee, put it in the file. If you correct the agent twice, update the SOP. Every correction should improve the system. That's how our Jarvis compounds over time. **[8:10]:** A normal chat disappears, a file gets better. Memory works the exact same way. The agent would not save everything. It should save the preferences, decisions, product updates, and lessons that would matter during a future task. Memory is not a transcript of everything that happened. Memory is the information you do not want to explain again. The final layer is skills, schedules, and speech. A skill is a repeatable capability. It could check the inbox, build a report, create a proposal, research a prospect, or repurpose content. Each skill should have a clear input, process, output, and stopping condition. The stopping condition matters because the agent needs to know when the task is finished. It also needs to know when it should stop and ask for help. For example, if pricing information is missing, it should not create the final proposal. If the customer is angry, it should not send a response. **[8:55]:** If an action can be resolved, it should ask for approval. If an action can cannot If an action cannot be resolved, it should ask for approval. Once you have skills, you can attach schedules. A schedule is what allows work to happen without you manually starting it. This could be a cron job. Cron is a scheduling system that runs commands at specific times. You could create a schedule that checks for urgent emails every few minutes, builds a business report every morning, and summarizes unfinished projects every single Friday. You can also use triggers instead of schedules. A trigger starts a workflow when something happens, such as a new lead entering the CRM, a customer submitting a form, or payment failing. The trigger launches the skill, the skills follow the SOPs, the agent uses the necessary tools, and then the result either gets completed or sent to you for approval. **[9:36]:** This is an actual AI employee workflow. **[9:38]:** The last piece is speech. TTS stands for text-to-speech. A tool like 11 Labs can turn your agentic text responses into realistic voice. You can also connect STT, which stands for speech-to-text. That allows you to speak to the agent instead of typing. Your voice gets converted into text, the agent processes the request, and TTS converts the answer back into audio. Now, you can walk into your office and say, "What needs my attention today?" The agent checks the connected system and gives you a spoken summary. Pretty cool. The voice is impressive, but remember, the voice is the final layer. It is not the system. Do not start with a glowing dashboard and voice assistant. Start with one useful workflow. Here is the exact type of workflow I would start with: an email and daily business briefing agent. Every morning, the agent runs automatically. First, it checks our calendar and important unread emails. **[10:26]:** Then, it checks your CRM for leads that need attention. After that, it retrieves a small set of KPIs and compares them to previous performance. Finally, it creates a short briefing that includes your messages, the emails that require responses, the leads that need follow-up, and the actions that matter most. The agent could send a briefing to Telegram, Slack, email, or a simple interface. Later, you could connect 11 Labs and have it read you the briefing aloud. That one workflow teaches you almost everything you need to understand permissions, memory, SOPs, schedules, tool connections, and human approval. Do not begin by asking your agent to run an entire company. Give it one job, make the job reliable, then add the next one. There is one final point that matters. The goal is not complete autonomy. The goal is useful leverage. Your agent should handle repetitive work, information gathering, organization, drafting, and monitoring. **[11:15]:** You should still own judgment, relationships, strategy, and irreversible decisions. I would not let an early version of this system independently send sensitive messages, move money, delete important files, or publish work without review. The agent can prepare those actions, it can recommend those actions, it can even is the reasoning model, the cloud or local AI that handles the thinking. V is virtual connections, the APIs and MCPs that connect the agent to your tools. I is instructions and memory, including your business contacts, SOPs, rules, preferences, and decisions. And finally, S is skills, schedules, and speech, including repeatable workflows, automatic triggers, and voice interface. That's Jarvis. Not the dashboard, not the glowing animation, not the voice. Those things can make for a fun experience, but the real value is in the infrastructure underneath them. A computer that is always available, an AI that understands you and your business, tools that allow it to take action, and documented systems that tell you exactly what to get done. **[12:10]:** Start with one machine, one agent harness, one model, one folder of instructions, and one workflow that saves you time every single day. If you want to see exactly how to build Jarvis, go ahead and follow and subscribe because I'll be building my own Jarvis here in the next video.