Transcript
Use with ChatGPT / Claude Auto-scroll A
The conversation opens with skepticism about spec-driven development and introduces Benedikt’s work in agentic engineering and product engineering. He explains how engineering and product roles are converging and why the focus should be on outcomes, not just implementation.
Speaker A
I actually like.I actually don't really like spec driven because I think the.So spec driven to me is kind of like, why are we doing this now?Because we did waterfall development before and now we are again doing some form of water and we know from without AI that's not going to work.And now we are doing the same again and we assume that this is going to work.
So I'm kind of confused about the strategy of spec driven development.
Speaker B
Should you do spec driven development when you work with agents and should you still do code reviews?And in all of this, how do you grow your team through psychological challenges, change of skill sets and more?So today we are talking about all of this with Benedict STEMILT, more than 15 years in software, host at several conferences, and founder of Hackers and Wizards, where he spends most of his time training software engineers and product teams about agentic engineering.So let's dive right into the action.
Welcome, Benedict, and thank you so much for being here with us today.
Speaker A
Hi, Luca.I'm super happy to be here.
Speaker B
So, Benedikt, you are the founder of Hackers and Wizards.You're also the co chair at the CodeTalks conference and at your company you do training in agentic product engineering and agentic workflows.
So am I right?Is this a good definition?
Speaker A
Yeah, that fits.We try not to do only the training part of it.So training is one of the products.
We usually try to do the full enablement thing and transformation of the company.And training is the most, like, I think the most famous and most easy part of it.Upskilling the people.Yeah,
Speaker B
yeah.
So I'm excited to talk about this with you because I know you and the team are practitioners, you have more than like 15 years of experience in tech, and I really want to understand how you think about this.So first of all, you talk about product engineering, which is a term that is discussed a lot these days.So I want to understand, what do you mean by that, first of all?So if it's about marrying the two disciplines, it's about a role that people have in a company.I mean, what do you mean by that?
Speaker A
It's a good.But it's also like a very difficult question to stay the
Speaker B
full hour just on that.
Speaker A
Yeah, yeah, yeah, yeah.It's also like, it could be super philosophical as well.Right?
Speaker B
Yes.
Speaker A
Yeah.I think the idea is that we try to.I mean, it has to do with everything you said.Right.Like roles are merging more together.So as an engineer, I also need to think more of the product As a product and not only about, okay, I am going to implement this feature now, but also as a product manager, you kind of are trying to get more closer to engineering.Is this that a viable feature in terms of like how hard is it going to be to implement that and so on.So it is about the converging of the roles, but it's also more on a level, I don't know.I think even before AI, people were not really that people could have thought more about their software than as a product.And that is still an issue like independent of AI, that you really need to look at the outcome of things and the vision of the product.
And so that's why we call it product engineering and not like, I don't know, like maintenance of a system or whatever.And that's kind of the idea.
They discuss how teams react to AI adoption, including fears of replacement and resistance to changing established processes. Benedikt argues that engineers often resist most because they are protecting hard-won quality standards.
Speaker B
Before we continue, I want to talk to you about today's sponsor, which is TRM Labs.
So I have a quick question for engineers listening.What's the hardest system you have ever built and who was on the other side of it?So TRM builds investigation software.So the other side is things like drug trafficking, human trafficking, terrorist financing or nation state threats.And TRM just launched Ardens, an AI powered platform that helps investigators keep up.
It returns findings backed by evidence that humans can explain and defend.And these problems have life and death consequences.So TRM has incredibly high talent density and is looking for the best of the best, dedicated people who want to be on a mission.The company has doubled revenue in the past two quarters, like explosive growth in a remote first team with roles around the world.And they're hiring like crazy right now.So if you're looking for the work of your life towards the mission that matters, you can go to trm labs.com careers yeah, absolutely.And you work on that both with people who are like product managers, I mean people coming to it from the product angle and with actual engineers.Right.That instead need more to go to the other end of the spectrum.Right.So do you think the goal is kind of making these roles closer so that each of these types of roles can actually do some of the other and converge on a workflow that is more common for everyone?
I mean, when you said at the beginning you do not only work with individuals or with companies, do you have a workflow you have in mind at the end?How do you think about this?
Speaker A
Well, I think, I think we all don't really know yet what the perfect or best practice workflow will look like, but I see some patterns that are working well and One of them is that, for example, we engineers like to do pair programming.Right.
Or at least a lot of engineers that I know.I'm a big fan of pair programming and also of ensemble programming even.Right.Like we're as a whole team, we developed together on a feature and now that we have the AI body implementing everything super quickly, I think these types of work can be used also in the full process so that a product manager and an engineer and someone from the business can ensemble together and work on the feature together instead of having like them work individually and handing over the work.So what we see a lot is that if teams really collaborate now, that makes them really fast.
Speaker B
Yeah, yeah, makes sense.And if that's the goal, like when you jump in and help people get closer to that, what are the most common problems you find?
Speaker A
So first of all, the biggest, there's a huge barrier in terms of, on a psychological level, like, is this going to replace me?Is that something that like.And then there's this, I don't know, kind of like an escape.So, okay, the code quality is not good enough.So.Right.So people are kind of like there are these psychological barriers and they are often something that you need to overcome.And then if you do that, there is the second barrier that is very, very big, which is the current process and the current organization structure of the company that is super hard to kind of like a wall you need to punch through to.And once you have this or like a very hard, tight knot and once you untie it, then, then you can make progress, you know.But yeah, there's this, this is like a strange, I don't know, there's so much force in the current in structures like people, how they work together that it's really hard to overcome.
Speaker B
Yeah.I remember when I spoke with Kent back on the podcast, he talked about power structures, right?That Agile, you know, had to untie and loosen up for things to work properly.But the power structures are resistant to change.And he was talking more about hierarchy because, for example, waterfall is better for hierarchy in top down direction and stuff.But I think there is structure in process and how people work in general.Not because there is a king that says how things should be done, but it's just inertia in how people work.It's hard to change probably.
Speaker A
Yeah.And I think the comparison to the agile change we had a few years ago is kind of very valid because now with this new tool we have another way of working that needs to change.And agile is now something like a power structure that is kind of holding us back because, I mean, Agile, not really agile, that would be wrong because I think Agile is super true, still super true and super valid.But Scrum, for example, as an implementation of it feels now kind of not working very well anymore because the ideas of Scrum are tied around the speed that we had without the AI.And now with the AI, the process does not really work anymore.But yeah, like, people are used to work like that now and that kind of holds them back on trying new things or leveraging it as much as it can.
So
Speaker B
given that you work with all kinds of people and roles, like engineers, product people, even people, I think from, from other roles, from.From what I understood.When you talk about this kind of psychological resistance and adopting the right mindset, do you find more resistance, for example, in engineers rather than other roles?Or it's, you know, everyone has their own biases.It doesn't matter whether you are a PM or an engineer or whatnot.
Speaker A
Oh, no, that's actually like engineers do have the most resistance.That is like.And I also don't think that this resistance is like, not.How do you say that?Like, it is okay to have it right.
There is a truth in the resistance because that makes you.You just don't want to build like software that is not of high quality.Right.So you can't just simply throw away all the principles you have.And we learned a lot and we needed to defend ourselves a lot to get where we are at the moment.
And now it kind of feels like that there's high risk of going back to this world that we had before with a lot of bugs, with a lot of brittle architectures and so on.And it took us years and a lot of effort to get where we are.And yeah, now product managers think they can develop software.Right?And for them it's like they are super happy now and you have other psychological barriers you need to check or.
Yeah, but yeah, the engineers are the most resistant.Yes.
Speaker B
Yeah, I asked that because I suspected that.But I mean, this could be like, for very valid reasons.So I'm not saying they shouldn't.
The discussion shifts to the gap between faster code generation and actual product progress. Benedikt explains that teams often produce more code without improving outcomes, and that AI time should be spent on validation, refactoring, and architecture.
Speaker B
So I really want to go into the details now because of course, we have also spoken before and I know how you think about some things and one of the things that you told me is that when you work with companies in working more embedded with AI, what you see is that they often have faster engineers and engineers that are going, might be going very fast by themselves by using AI with their own workflows in their own ways.But that does not result in shipping more product or in faster companies overall.And I think that's also backed by a lot of research these days that shows that basically, yes, there are way more commits, there are way more PRs, but then some point the pipeline gets stuck.You know, for some it's like code reviews validation.For others it's other things.For others it's like requirements are too slow, whatever, but the outcomes are not just as getting as faster as we are at producing code.
And you told me that's something you're seeing as well.
Speaker A
Yeah, definitely.So that's also like many faceted question, because what we see as a first thing currently is that, yes, you get way faster with AI, right.So if you have a backlog and you can produce much more code quickly, and if you do the.
Let's assume that you are not doing the review with humans anymore, you build your setup in a way that you can actually review with agents as well.And then you produce a lot of output basically.And that leads to.That your, maybe your backlog is even empty at a certain point.And then, yeah, you don't really, I mean, you don't really know if this output also creates outcome.Right.And before we, we didn't really measure this and now we kind of.I think there's two things now that happen.Like, one thing is people just build a lot of stuff in the product that makes the product worse because you can, you just can.And the other thing is that, yeah, that people don't really kind of know how to answer the question, what do I do when the backlog is empty?Right.Like, or like how do I, how do I ensure that I build the stuff that is actually valuable?
Speaker B
Yeah, yeah, I think this is a very good point.And I, I was talking with a friend of mine last week and we were saying that I think before AI there was for the fact that building things had a much higher cost.You were justified in spending a lot of time beforehand in figuring out not only how to build things, but also what to build, because then you will be building that for quite a long time.
But, but now maybe some teams are getting into the fallacy that since it feels like it's cheap to build more things, you can just make your product better by building more and more things without asking yourself too much, what should I focus on?Should I do A or B?No, let's just do both.Right, but maybe what you're saying is it does not always result in a better product when you do like that.To
Speaker A
put it in line.
Speaker B
Yeah,
Speaker A
yeah.I think the problem is that even before AI, people were not really measuring if that what they do is something that really creates an outcome.But they could have.
I mean, now you have the chance to actually do that.So instead of kind of like just implementing all the stuff that you think is nice, you can now.So the question is basically where do I spend the time that I gain from the AI efficiency?So if I'm loud, I'm 10x faster.Should I spend all of that faster on just implementing features faster or should I spend that on validating features, on being more hypothesis driven or how much do I spend on cleanup for my workflows?For example, only only like 20% of my agents are working actually on building new stuff into the products and 80% of the time and of the tokens are spent on doing refactorings and improvements of the architecture so that it stays also good.Right.And I think people are over like when we didn't have AI, it was like super great if we could finally implement some things.
And so people are now kind of implementing too much in my opinion.Like they are like too focused on just getting stuff into production or kind of building stuff and then instead of like thinking about if that is actually valuable.
Speaker B
Yeah.So you said that you have many agents that for example, do refactoring and not just do more product work, which is the low hanging fruit instead.Right.So if you're talking of these areas where it doesn't come as easy as with new features to invest more in terms of using AI and agents, but instead it becomes very valuable if you do that.
Benedikt describes where teams underinvest, especially in early validation and continuous architectural cleanup. He explains how scheduled routines can keep systems healthy and how he measures token usage across these workflows.
Speaker B
What would be these areas in your opinion, if you think of the companies you work with, areas where there is an underinvestment with respect in using agents and AI with respect to what teams could do.
Speaker A
Yeah, that's a great question.So on the first hand it's kind of to work more hypothesis driven.So trying to figure out, and that is really hard because, okay, so trying to figure out if the feature is actually something that the users want, like if this is actually improving the product.And I can do that by doing prototyping or by having like a alpha canary deployment version and then collecting data from the users and only once the numbers tell me that this is actually something valid to bring it into the product.So to get signal very early and not kind of trying to do product management way where you kind of say, okay, we need all of these features, they need to be all like do we actually need them?You know, like is that something that, like, what is the data that backs that up?
And on the other hand, it's.Yeah, what, what usually gets also lost is that, that like updating the architecture, constantly updating dependencies, doing refactorings.And kind of usually if you see these longer software projects, like, stuff gets kind of built from the side and on top of it and then it looks like a. I don't know.And now with the agents, you can basically, with every new iteration, kind of do a full overhaul so that it's always like a nice picture.
So that your architecture is kind of always in a clean, holistic, well defined shape.And where you would before maybe have some issues in terms of like, oh, we have a monolith.And now for this type of thing, microservices would have been better.And now you can just say, well, then let's cut it and make it a microservice.Right.
And these kinds of things are not.Yeah.I think people are still from the old world and feel, okay, it still needs to stay monolith, for example.
Speaker B
Yeah, yeah.So now it's a couple of times that you mentioned refactoring, keeping like the whole architecture sound.
How do you, how do you do that in your projects?How do you use agents for that?Because I think we are all used in at like for example, using agents to build feature by feature.But then how do you think, like, big picture?Because I think when you say architecture and refactoring, it's more like broader tasks.So how do you keep that sound?
Speaker A
Yeah, so I really like the scheduling functions that you have.Right.I mean, it's like,
Speaker B
yeah,
Speaker A
very basic to do a cron job, but it's really helpful because you can just schedule, for example, daily cleanup routines or weekly cleanup routines.And there's a very good combination of these things.So you can daily, for example, let it look over everything that you did on this day.And sometimes if you let an agent implement a new UI feature, it rebuilds the component.Although there would have been a component already that he could have used.Right.So creates a new component.Now we have two components that are doing the same.And the daily cleanup routine will then notice that and say, well, we introduced a new component here that would have been necessary.So it merges them together again.
And on a weekly basis you can do something broader, you know, that you say, okay, now that we have all of these features and so on, can we kind of change the architecture, change it.And of these routines that are running in the background schedule, I have a few of them.And that's why also, the most token spend goes in these cleanup routines because they run constantly.And.Yeah, so it's like 80% of the spend that I have.
Speaker B
Do you measure actively where tokens go?And did you find anything?I mean, for example, this was already interesting to hear.Like, 80% goes into this cleaning up, you know, catching up with things I could have done better and so on.
Speaker A
Yeah, so I actually do.
I mean, it's super interesting as well, regarding all terms of, like, caching and not caching and how the effect to cost is, and could you use that with a smaller model or a smaller effort and so on.And I collect the data so that I'm able to kind of optimize the right models to use for which of these routines, because you can pin the model for the routines, and so it's often a nice idea to pin it to models that don't use so many tokens.And.Yeah, you can't really do that without measuring, you know, so.Yeah, so I'm, like, measuring it quite heavily.
Speaker B
Yeah.
They talk about choosing the right model for different tasks and whether that optimization is worth the effort. Benedikt then defines a software 'factory' as an unsupervised, automated engineering workflow managed by exception.
Speaker B
Do you think that it's worth investing, like, in trying to understand what's the best model for the best routines?Because my feeling is that most teams feel like they don't have the time to do that.They just stick with the latest and greatest, you know, and they say, I can go any kind of worse than X, you know, by doing that.But maybe they're missing out in terms of savings and speed when it comes to tasks that could be just as well accomplished by much smaller models.I don't know.
Speaker A
Yeah, I mean, when I'm using the agent in a supervised way, so I'm talking with it, and I'm kind of like, I don't really optimize the models in these situations as well, so.Because this is, like, tedious.I don't.I mean, yeah, but if I build something like a routine or like a factory.
We haven't talked about factory yet, but if I kind of try to automate a workflow to an unsupervised way, it's often helpful to do this because you don't need to manually switch.Right.So you don't need to be.I mean, switching is always very tedious because who does that?Right.Like, who is, like, thinking about, oh, this is a small task, so I'm doing the other model now, and so on.I kind of have a default, which is Opus Medium at the moment, because Opus Medium feels like being super strong for every case and also fast.But, yeah, then there are some Cases, some of my routines are running on Opus High and others are running on Sonnet High instead of Opus Low or something like that.So I really tried to tune that a bit.Yeah,
Speaker B
that makes sense.And since, I mean, you said we haven't talked about factories yet, so thank you for asking, but would you say your setup is a factory and what qualifies?I think we are not.I mean, I don't think anybody's on the same page about what factory means in terms of software.So let's try to get some consensus.What's the factory for?What qualifies as a factory?First of all?Yeah,
Speaker A
that's a hard question.
So I would say so for me, at least my definition of a factory is.And I don't really like the term factory at all because developers don't want to work in the factory.And I mean, okay, we don't work in it because we built one that the agents work in it, but we can try
Speaker B
to invent a better term.
Speaker A
Yeah, we need to invent
Speaker B
it.
Speaker A
So if we put the term aside, what this actually means is I'm automating my process, right?So it's like a default.I could automate a marketing process.Now I'm automating the engineering process and we just made up the word factory for automating the engineering workflow.
So let's put aside factory and then it's easier to define it because I would say that supervised, it's not a factory.And unsupervised, it's a factory.And then even if I have only a few small things that run autonomously unsupervised, that would mean that I have some kind of small factory.It will probably not be the whole process and it will probably need supervision at some points.But I'd say that like a factory is actually something very small already.
Yes.Yeah,
Speaker B
yeah, makes sense.I like this revised definition.So it's like the factory is supposed to be run, to be managed, like by exception.You know, people jump in if something goes wrong.Otherwise you have more of an automated process.
The conversation turns to specs, requirements, and why Benedikt dislikes spec-driven development. He argues for exploratory prototyping first, then productionizing later, and discusses how model changes force him to rethink his harness and rules.
Speaker B
And I think we have talked about different parts of that already.And especially like the part about, you know, catching up with what falls through the cracks and architecture and refactoring.Do you also have opinions about how to work on the product side, like what comes before coding, like specs and requirements?
There are big trends and keywords and buzzwords there to like spec driven development and other things.Have you found something that works for you?Have you found something that you hate?We like talking about things that we hate in this podcast.Yeah,
Speaker A
that's true.Ok.Okay.So I actually like, I actually don't really like spec driven.
Speaker B
I like it.
Go on.This will go in a clip for
Speaker A
you.Okay.Because I think the SO spectrum to me is kind of like why are we doing this now?
Because we did waterfall development before and now we are again doing some form of water and we know from without AI that's not going to work and now we are doing the same again and we assume that this is going to work.So I'm kind of confused about the strategy of spec driven development.I mean I understand that the idea is if we have like a better defined requirement, then the agent is going to implement it better.But this was the case also before AI.Right.And we noticed that we are not able to write the requirements in better ways and we need the engineers and the process and the time to figure out what the right requirements are.So I really don't understand what we are doing with spec driven development, but what do I propose instead?So I really like the idea of thinking about like a little bit in two speeds or in two, I don't know.
So the product work, I usually do it supervised, like chatting with the agents, talking with the agent, building a small prototype, showing the prototype to someone, getting feedback from them.So this is kind of more like a collaborative thingy.
And then when it goes to, I want this feature now to be in the real product, then I can kind of just push it into the factory and the factory is going to productionize it.So for me, my factory is a little bit like an extension of the CI CD pipeline.And I mean you could argue that this is now kind of spec driven development because when I have the prototype and all of that, then I have my spec.But to me it doesn't really feel like I think the process of how you get there is quite different than what the spec driven people are kind of advocating for.
Speaker B
Yeah.So the way I personally think about that, because I'm not a big fan of spec driven development either, but especially because I think it leads to wrong slash confusing interpretations of what this means.Because if I think in my own product work, I mean there are features where when I want to build them, I know exactly what I want to build.Okay.Because I have the full thing in my head and so I can totally understand that if I create a more detailed spec that helps the agent adhere to that and stay closer to the vision, then the outcome is probably better.But the problem is that most of the Times I don't have that.
Most of the times I have this idea that I want to build, but I don't exactly know what's the shape.And so it helps me to just get something out there, like a cheap crappy prototype done by the agent out of a very thin spec, like couple of lines of stuff just to update, you know, inform my intuition and understanding of what I really want.And then after that maybe I'm able, as you said, to come up with the full spec.But it's like you do that in, in multiple steps so it's like iterations like we've always known and done.Right?
Speaker A
Yeah, I think that this spec driven thing is kind of advocated by the engineering community a lot by the developer community.And it comes from the pain that they know like not well defined requirements and so on.But when I talk to the product managers they are more in this, what you described, this exploration mode.This I don't really.Discovery mode.Right.Like I don't really know what this is.Now I know there is a problem and now we need to figure out how to address the problem.So I can't really specify it yet what I want.So let's kind of try to explore it.And, and then we have this design Sprint thingy way of working.And yeah, I think that, yeah, that this is completely different, you know, and then maybe you need a spec to hand it over to this productionizing thing.
But it could also be the unfinished code, you know, like the beta version code, which has a lot of bugs, which has a lot of performance issues and unsafe things, security wise and so on.And then the factory doesn't even need a spec anymore.The factory would get the code and would look at it and say, oh my God, who implemented this shit?And now I kind of do all of the cleanup stuff that makes it good code.
Speaker B
Yeah, absolutely.Another kind of problem I have with overly detailed things that might be specs but also harnesses, also parts of the processes that they look to me like they're more fragile with respect to new models and models changes because it's like, I mean whenever a smarter model comes out, I always ask myself and that's something where I don't feel like I'm doing a good job.So if you have any recommendation for free, but I feel like I should update, you know, the harness and how I work on things because maybe some of the things that I thought I need to say intentionally to the model, otherwise it will not apply them or it will forget about them.Now maybe the model is smarter and I don't need to.
And so spec driven development in particular, you say very specific things.Maybe some of them the model would have figured out anyway or come up with better solutions.But yeah, thing is, I still struggle at keeping, I think my harness state of the art with respect to what's appropriate with the different models.I don't know if you have any suggestions or any experience about that.
Speaker A
Yeah, so first of all, I have the exact same issue, so that might be good the way that we share the same problem.
So I also really find that hard.And it's also the spectrum of like how deterministic do I need to do things and how non deterministic can I keep things?Because it's also helpful for these things to be non deterministic.And I think with every new model the scale changes.And for me it's a little bit like reverse engineering.
So because I tend to, if there's a real new class of model, let's say like Opus 4 something to Opus 5.Right.So there's like a, like a real major version update of the model.I actually try to just remove all of my skills and rules and then observe it, how it is doing it without any guidance, and then figuring out which guidance I want to put back in there.
Benedikt explains how he structures skills and agents, keeping rules thin and using a small number of skills with many specialized agents. He describes an orchestrator agent that coordinates work, handles incoming tickets, and keeps the whole system organized.
Speaker A
Yeah.
Speaker B
Do you use many.You mentioned skills, so just popped in my mind.Do you use many skills for different things?Because you know, there are different like schools of thoughts there.There are people who are segmenting everything into separate skills and have like big collection of that.And those who say, no, let's keep things simple.We I use no skills or very few skills, put most of the things instead in the agents file.So different approach.So how is it for you?
Speaker A
Okay, so usually my Claude MD file and rules files are thin.
So there's only like this guidance of how this repo is structured.You could say like where to find things so that it's easier for the models to kind of orient themselves in the code base.And then I tend to have a few skills, but not more than like five or six.So also try to keep it minimal.But I do have a lot in specific agents.
So my skills are often just like a wrapper to call a specific agent.So I tend to have like 10 or 15 of these specific agents.And in these agents I have the configuration of what this agent is going to do or focus on.So when we look at the cleanup thing, for example, there's one skill for this cleanup thing and the skill kind of orchestrates the call of multiple Agents.
But I only do this because I figured that depending on the harness.So I'm using CLAUDE code harness a lot and CLAUDE code is not really calling agents on its own.I don't know why.So it is kind of shy on calling agents, even though this specific agent would have been matching to what, what the task is.So I, I built this bridge using the skill that it kind of in the skill, then it's kind of written like call this agent and that agent and so on.
Yeah,
Speaker B
yeah, yeah, Interesting.And so while you were talking, I was thinking that you mentioned like 10 to 15 different agents to do different things.I think something that we still having gotten to some consensus is how to think about agents in terms of their scope and what a single agent should do versus when you need more of them.Whether you think about agents like people, you know, with broad responsibilities and roles, or you think about them, I mean, when it comes to software, like maybe they're associated to steps of the software development life cycle, there's an agent that does reviews an agent that does coding, which is different.
Different with respect to how we work with people.So you cannot say agents are people anymore.So how, how do you think about agents in terms of structuring your, your fleet or agents do and assigning responsibilities and splitting them and so on?
Speaker A
I think for me it's a mixture of both actually.So, and I also, I mostly use it for technical separation of things, so to also optimize the context window and token usage.Because I can again, an agent, I can pin that very well to a model and I can, I mean pinning skills to a model is kind of.I don't really understand how this works.So if anybody knows, please tell me.But if I like, if you have three skills and they have all a different model and now I load all three skills, like which model is being used?I don't understand how, like in the documentation it says something like that.Yeah, I don't get it.Like, how is this going to work?
Speaker B
Yeah, we hope that someone at Anthropic is listening to this will jump in in the comments.
Speaker A
Yeah, and if it first loads a skill with a and then changes to a different skill later, then all my cash will be wiped.So this is also very bad.Right.So I don't understand the skills being tied to a model thing.
Speaker B
But
Speaker A
agents to a model is kind of nice, right?Because I know that this full agent is then going to work with this model and that is kind of what I use them for, to kind of scope their brain power.You know, and sometimes it's helpful to do that by people, ideas or front end developer.
Speaker B
Yeah, but
Speaker A
this is kind of a mixture of tech and person, like specialists for certain things.But I also have someone that is kind of like the Orchestrator that is also an agent that is kind of scoped to just keeping the organization of all of the implementation.And I actually start each session not with the default agent, but with the Orchestrator agent.
Speaker B
So the Orchestrator Orchestrates because that's the name the different work, different work from different agents that.And these agents work on different parts of the development process.Maybe they're like coding agents.I don't know if you said front end.I don't know if it's like front end versus back end.It's separate agents.Or do you also have like a reviewer agent?Yeah,
Speaker A
exactly.So I don't have a reviewer agent because the review is then done by.
I have a review skill and the review skill uses then multiple agents to kind of do the front end review, the back end review.And I'm not really using the front end agent for implementation.I have like a full stack guy for that.Right.Like someone who's the product engineer, you could say, who tries to build it end to end.
And then in the review the specific Personas then kind of review the parts that were built and optimize the parts.Yeah, but the Orchestrator for me is just someone who kind of keeps the overview about everything.So I can just dump stuff in there.Or if there's a new like for example, a lot of the products that we build, people can just provide tickets in the product itself.They can just say, here's the bug or I have a request here.
And all of that stuff is just being picked up by the Orchestrator.And then the Orchestrator kind of organizes all of that stuff so that we can keep track of everything and don't forget anything.
They discuss code review in an AI-heavy workflow, where Benedikt no longer reviews every line but instead reviews higher-level artifacts and system understanding. He uses visual tools like heat maps and domain maps to track architecture quality and coupling.
Speaker A
The Orchestrator is also like running all the time basically.
Speaker B
Yeah.Okay, and so you mentioned reviews, which is another big controversial topic today.
What's your take on code reviews?So on a spectrum that goes from I review every single line of code that my agents produce to I don't even know what programming language they're running things in.
Speaker A
Okay, that's a huge thing.So I actually have a little bit of a different view on code review.
So why do we do code review as engineers or as teams as.Well, I think that this part of the quality of the code that is part of the code review is at the moment, I think agents are much Better in that than humans.So I see a lot of agents figuring out stuff that I would have kind of skipped as a human.So I don't really review code lines anymore at all.But I still know the technology and so on because I made the architecture decisions beforehand and so on in this kind of supervised, collaborative, exploratory way.
But I don't review every single line of code.But on the other hand, we do the review to keep our mental model of the technology right, like, how is our application shaped, what is our heat map?And so on.And I really like to review that.So there is kind of like a reporter in my setup that kind of updates my artifact.So in cloud code you can use artifacts.And artifacts are kind of nice because they are like living HTML documents that you can also share with others.So if you have like a client who wants to know what is the status of the project and where are we and so on, then you can just share them.The artifact.And the artifact can constantly get updated.
So my reviewer agents actually update the artifacts so that I can ensure that the overall model of the system goes in the right direction.Because reviewing is about understanding the mental model of the application, for me at least a lot of.Yeah,
Speaker B
so these artifacts, as you review, like diagrams of the architecture or like, for example, I use architecture decision records a lot.So what are you reviewing?Like, exactly.
Speaker A
Yeah, so for me, I try to have as little text as possible.So I do like the concept of architecture decision records, but I don't really read them because it's like too much stuff to read.So I try to visualize.
I mean, the model is doing that or the agent is kind of writing them and then also reasoning on them.But I don't really review those, but I try to visualize a lot.So here heat maps are interesting of the code base architecture diagrams,
Speaker B
heat maps, like which files in the code base are hit the most with changes and churn or like, exactly.Hidden coupling, stuff like that.
Speaker A
Yeah, actually, so for coupling, I really like to use like a domain map.Like you can see which context, like which bounded context you have and how are they coupled to the others.Heatmap is for me exactly what you said, like which part of the code is being changed the most.So this is probably code where I need to focus on or where I need to kind of make sure that this is going to be of high quality or also an indication for something which we would maybe need to refactor in a way that I have a better spreaded heat map.So that it's not like, you know that you don't have this one class where you need to work on every time when you do something which is split like kind of like a code smell on the architecture level, you could say,
Speaker B
yeah, okay, makes a lot of sense.
The final discussion explores how software engineering roles may evolve, with some people focusing on factory automation and others on product engineering. Benedikt argues that skeptical engineers can use their instincts to improve systems, while generalists and specialists may diverge into different paths.
Speaker B
And do you think, I mean, I was thinking while you were talking, this, this looks to me like also indicative of a trajectory of, from what a software engineer has been doing so far to how a software engineer can kind of work in the future, like reviewing higher level artifacts, coordinating maybe a small fleet of agents doing this and that.So how do you think about this transition, you know, that many people are going through?Maybe you're a founder, so it's kind of psychologically easier for you because you're used to do many, many different things.But I think many software engineers are either anxious or skeptical or concerned about where the work is going.
Do you have an opinion that is more like optimist or negative about the industry and where it's going?How do you feel about that?
Speaker A
Well, I think the first thing is that I think that people just should try that stuff out, right.I can be skeptical of it, but I can still use it or try it out.So if I want to, kind of, I would encourage everybody to just use these tools to figure out where their place is in this new world.And you can only do that by kind of trying it out.Right.And then if you kind of would ask me what.There are two roles that I currently see that might be emerging out of this.And one role is someone who's going to work more on the factory.
So on the process automation side, which skills do we need?How should we organize all of that and so on, that our code is being productionized the right way.So for this later part, and these are often so, at least from my experience, often the more skeptical or more critical, more engineering type people that are kind of trying to actually review the code and then figuring out how the factory should look like so that the code gets better and so on.And then I feel that there's this other group of people that will be more, that will evolve more into the product engineering role.And in this design sprinty thing, building prototypes and figuring out if this is a real feature and will help the product managers to build better prototypes, you could say, or maybe product managers that are currently very far from engineering will also evolve more into this engineering direction and this could be like a new emerged role, you know?
Speaker B
Yeah, yeah, I like it.So basically you're saying if you're Skeptical.Like weaponize your skepticism and use it to build better systems that make you less skeptical of what's coming out of that.Right,
Speaker A
yeah, exactly.I mean, the.So the question is like, why are you an engineer and what are you losing?Right.So on the more psychological element, despite the skepticism.So there's also kind of this problem that engineering is really something that is a lot of fun.
Right.And you get into the flow and when you develop and so on, and now you fear to lose that flow and because only the agent is doing all of the fun stuff anymore.Right.And you kind of like type in a few sentences and then the agent has a lot of fun.
Speaker B
Yeah,
Speaker A
but I think that if you look at the factory side of things, there is again this very similar task, like engineering.You know, you need to figure out why did it work?Like, why did the agents not do it in the right way, what was missing?And like this reverse engineering of the model, do we need to add something?Can we leave something out?
So there's a lot of flow there again as well, in my opinion, at least it feels to me like that.Right.I feel the same engineering flow.And that could be something that also skeptical engineers really enjoy if they are not on the, you know, like on the flow side of things when it turns out, like building products is also something that brings people into the flow.But that's kind of a different type of person.
Speaker B
So have you found this also kind of relates to being a specialist versus generalist type of engineer?Like specialist engineers might be more keen to working on platform and optimizing for quality on developer tools, while instead the generalist ones are the ones that are more likely to turn into product type engineers.
Speaker A
I mean, you never, you can never generalize these things.But I.
Speaker B
Yeah,
Speaker A
I think there's a lot of comparison.Like, like, yeah, there is a lot of truth in that.Right.So it's a similar thing with full Stack developers.You know, Full stack developers were also more tending to build the full product and so on and are happy about like, like shipping products and so on.
I mean, there's still also full stack developers that are super skeptical and then maybe also more on the factory side.But when it comes to specialists, I would say that you can really leverage them to kind of optimize their special role in the factory.
The episode wraps up with closing thanks and reflections on the discussion. The hosts sign off and the show’s outro encourages listeners to subscribe, rate, and find more episodes.
Speaker B
Yeah, makes a lot of sense.So, Benedikt, thank you so much for this great chat.I think we covered a lot of ground, a lot of controversial things, but I think we gave good answers.Not sure if our answers will get to the consensus of everyone, but we did our part today.
Speaker A
Yeah.Thank you so much Luca.It was a lot of fun.I hope everybody is happy and could kind of get something out of it.
Speaker B
Totally.I'm sure.
Thank you.Thank you so much Benedict.
Speaker A
Bye bye.
Speaker B
Thank you so much for listening.If you found this chat valuable, you can subscribe to the show on YouTube, Apple Podcasts, Spotify, or your favorite podcast app.Also, please consider giving us a rating or leaving a review as that really helps other listeners find the show.You can find all past episodes and learn more about the show at Refactoring fm.See you in the next episode.