From 1 Agent to 11: Patterns for Multi-Agent Systems That Actually Work

Speaker 0: Started, manually just using open source repos and creating cleanroom specifications from existing open source repositories just so that I have them cached. If at some point I wanna run it forward and recreate that software, I can. So I got tired of doing this manually, so I created this app. It is open source, and I'll give you the links at the end. But I also created this using a Ralph loop, so this is created with a Ralph loop and embeds a Ralph loop in the application. Speaker 0: So I'm just gonna use Excalidraw, which is a great open source project, and put this in here. So it'll it's gonna this is meant to run locally, so it's gonna clone the app Michael, sorry, clone the repo Michael, and then I have it in my home directory. I have it creating out the clean room specs, and that's a separate git repository that I can commit to a private git repository. Like, this app is open source, but my specs that I'm creating are are private in a private repo. So it's taking longer than I expected, but this doesn't work. Speaker 0: Oh, there we go. There we go. Are you are Speaker 1: you going off, like, documentation in the repo, or are you having the model, like, basically reading the code and Speaker 0: No. So this is this is the reverse reverse route flip. You can just come up with I have prompts in here that examples. There are some secret sauce in here. Study the existing specs. Speaker 0: So it'll look in the specs directory. If the specs directory doesn't exist, it will create it. So you're thinking the prompt is you're thinking iteratively. You can tell it to look at something that doesn't exist. It will just go create it if it doesn't exist. Speaker 0: So Mike a to do dot MD file, you could tell it to create that so it can keep keep track and then tell it to update it. Speaker 1: But Is it, like, filling in the gaps for you, really? Speaker 0: Yeah. It fills in its own gaps, and it does its own planning, and they're smart enough to do all this themselves. So identify 1 specification that still needs created. So, like, focus on 1 task at a time, and it constrains you know what? I always do this. Speaker 0: I forgot to set the timer for myself. Alright. So the context window is Mike memory allocation. Like, you wanna put exactly in it for, like, what you're working on for that task and nothing more. So this constrains it to, like, go out and look at the code base and find, like, we're looking for, okay, what we wanna do next, and then it goes and looks at all the code and understands, like, what is what code is involved in that change or if I need to create new code. Speaker 0: So it's building up that context every time and focused on that 1 task, and then it shuts down. If it gives up, it just picks right back up because you have a bash loop or this app around it that starts from scratch. It builds up the context and said, oh, I was halfway through this task. I'm just gonna pick that right back up. And that's why this works. Speaker 0: You can just put it in a loop. It'll pick back up, and it'll just keep going, and it'll finish that task, go to the next task. Speaker 2: When you say it's picking up the project and, like, all the stuff that I need to do, does that even mean, like, all the dependencies and stuff like that? Like, basically, it's totally and it's just a clean environment. Like, the environment is ready with that particular version that's out now. I asked that because the there's that. Speaker 0: It's just this is just creating specification markdown files. It is a way that you can I guess questionable, but you could copy existing open source software and create your own version of it? The reason Speaker 2: I ask is because there was a light alarm hack Yeah. Last week, and, like, I'm increasingly skeptical of, like, you know, making sure I have clean copies of, like, every single package that I'm using. Even though I don't upgrade my stuff often, so it just might might be something Speaker 0: So if you wanna create your own copy of LightLM and legally have a right to it, you can use this app, create specification files, clean room specification. It doesn't reference any of the existing code. That's the reverse route loop is you take a code base, turn it into a specification, forward would be specification into your custom code, your light LLM library. So you can maintain all of your dependencies. You don't have to rely on light LLM, but then you have to maintain them, and you have you know, your organization has to be able to, like, maintain those Speaker 3: So the storage library. That worked is what what I Speaker 0: see here. It's a raw flow. I that's I don't know. Reverse to forward. So you can feed my specs into your project so they go together. Speaker 0: So this is essentially, you can select 1 of those prompts. You can add prompts, whatever. And then if there's, like, something specific, Mike, I Sean a deep research CLI, I found the top 1 that was open source. It's Onyx, and I said extract the deep research functionality out of this so I can build a CLI. And so, like, I could type that deep research CLI in here, and then it would, like, focus just on that functionality. Speaker 0: So I will start this job, and then it'll follow the logs of clog code. I'm using the clog code SDK, which is embedded. So if we look at this runner here, this has the job runner class, the run. You can see my iteration. So this is my Ralph loop, Speaker 2: and then it should be highlighted Speaker 0: right here. This run agent iteration, this is 1 iteration of it. You can see I'm using the Cloud agent SDK. This is my system prompt that I'm feeding into it. It's guiding it to make sure I use my clean room home directory Mike. Speaker 0: And then the query it's building, and it executes it in here somewhere. But but yeah. So I'm I'm embedding it in there, and I think that's it. Speaker 2: Deployment cloud codes leak code on GitHub. Speaker 0: What's that? Speaker 2: Deployment at cloud codes Speaker 0: No. But I there are people who did that immediately. Jeff Huntley was in Korea at the time, and 1 of the people he was with are it's 1 of the main repos that's out there right now of, like, here. I created it because that's all they did. They ran the rough loop and built the code. Speaker 0: So Can you show what the 1 Speaker 2: of the specs that it creates? Speaker 0: Yes. So trying to think oh, yeah. Speaker 2: I have 1 open. I'll I'll do the Onyx project. I didn't well, we'll just look at my file system. Mike Sean room. Speaker 0: So these are the checked out repos. These are the specs. Speaker 2: Doesn't look like there's much in there because I Speaker 0: just started it, but, this is Xerxes, which is Mike marketing platform or something. I can't remember. Speaker 3: Which botnet is Speaker 0: it? Is it a botnet? Speaker 2: I don't remember. Speaker 0: Apollo customer scalar what's activity log normalization, array, object reshaping, color space, cursor, pagination utilities. So a lot of this stuff, I have no idea, and I may or may not review it before I run it forward. I'd The the Speaker 2: spec isn't really the point. It's really the code. Right? Speaker 0: Yeah. Exactly. An artifact. Yeah. As long as your prompts are correct. Speaker 0: So I have some I have some interesting things that may be interesting to people that I put in the prompts. Here. So there like, there's formal specification language, which is Mike TLA plus. Yeah. So I, like, I make sure it uses formal specifications. Speaker 0: I kinda tell it about formal verification languages, but those are, like, too complex for what I'm actually trying to do. But, yeah, I like, that's embedded in the prompts. A really weird 1 Speaker 3: 0, Speaker 0: I don't have it in here. What is it? There's, like, magic there's magic words that you can put in prompts. There's, like, incantations. Sean is it? Speaker 0: Maximize coherent expected coherent value. It just, like, locks Claude in. He's like, okay. I'm ready. What are we doing? Speaker 0: And so it's Mike just having different, like, magic keywords. I noticed it wasn't in here, but there's probably some other ones where you put words in that are very concise. When I'm actually working on a code base too, I will try to make these cloud prompts oh, I know where where it was. Let's look at the actual Ralph for this, the prompt. Optimize coherent expected value. Speaker 0: That that will lock it in. HTN planning, that's hierarchical task network. It's a concept from the eighties, but it's essentially building a task dependency graph. It's, like, built into the model. And so if you tell it that, it will all you have to do is tell it HTN planning, and it will create a DAG of your tasks for you and understand at a deep technical level without having to explain what you want. Speaker 0: You can just say HTN planning, and it will create It'll even use, Claude codes, task manager across sessions, and it will create the dependencies. It's Mike the equivalent of why is it blank? What's the what's the beads? Yeah. Beads. Speaker 0: They're equivalent of beads. So any other questions? I'm probably close on time. Speaker 1: Passing down into the l m, you use the magic words. That's what it seems like to me. Speaker 0: Passing down? Speaker 1: The coherent value would be something calculated in the model itself. Speaker 0: Yeah. I imagine it's Mike That's Speaker 1: what's happening is you're Speaker 0: The parameters are lighting lighting up. Database. Yeah. You, all things considered, you know, you just, the model's looking at everything. No. Speaker 0: That's it's kinda like the it seriously, type that in to Claude, and the first time I did it, I was like, oh, he's like he's like, what are we doing? Like, I'm ready to go. Let's let's do whatever the expected value is, and so but, yeah, there's there's lots of different keywords. IRC is another 1. Speaker 1: Are there any, incantations to to get it to, like, reset? Like, it without resetting context? Like Like Mike, if it's debugging something, Speaker 0: you get it Speaker 1: to, like Speaker 0: because that goes back to the Okay. Memory allocation. That's, like, that's what we're doing. You don't want anything else in there. There is no magic word. Speaker 0: It's Mike you have all these unrelevant tokens that you're throwing in from your previous conversation, and this is getting rid of all that. It's Mike, this is okay. Let's figure out the task. Task. What's relevant to the it's like there is no going back. Speaker 0: It's you wipe it wipe the slate. Oh. Hey. You're set quietly. Nobody was moving. Speaker 0: Okay. Well, any other questions? Otherwise, I'll wrap up. Alright. Thanks, guys. Speaker 0: Oh, while he's Robert. Coming over. Okay. If you guys wanna scan that real quick too, my open repo is 1 and the Jeff Huntley article is the other. Or just take a picture. Speaker 0: I must. I must. Alright. Here you go. Alright. Speaker 0: I'm excited for this 1, Sean. Speaker 3: It's Thanks. Speaker 0: The boat as far as it goes. Speaker 3: This has been great. Speaker 0: This will be an interactive 1. Hell, yeah. Hell, yeah. Speaker 3: Is this as far as this goes, actually? Speaker 0: Yeah. Oh, here. Hold on. We got pushed to cook that. Speaker 2: Thank you, Speaker 0: Okay. Maybe I'll come over this way just a little bit. Speaker 3: Okay. We having fun or what? Speaker 1: Hell, yeah. Speaker 3: Hell, yeah. Let's do this. Let's have a little Mike fun. So, 6 weeks ago oh, I took I grabbed your phone. Sorry about that. Speaker 3: Wrong phone. 6 weeks ago, I had 1 AI agent. His name is Lou, and, he helped me write emails. Today, I have 11, and they run my community, my CRM, my finances, my speaker pipeline, my event logistics, all of it. Along the way, 2 of them drafted contradictory emails to the same person. Speaker 3: 1 linked my family's names to a group chat. 1 lost an entire day day of work, and I didn't notice for 11 days. And we wrote it all down. So but before we go any further, let's go back to 1994 for a second because back then was an amazing time. Grunge was still on the radio, Java hadn't shipped yet, and 4 4 computer scientists published a book that changed how an entire industry talked to itself. Speaker 3: Some of you know exactly who I'm talking about. 4 authors, 1 book changed how an entire industry communicated. Anybody remember that? The gang of 4? Gamma, Helm, Johnson, Vicities, design patterns. Speaker 3: And what they did was deceptively simple. They documented 23 patterns the practitioners were already building independently. Observer, factory, they gave them names, they gave them structure. Before that book 2 developers solving the same problem in different companies had no way to talk about it. And after that book you could walk into any team, say that's a factory, and everyone in the room knew exactly what you meant, what trade offs it carried and, when to use it. Speaker 3: That book didn't invent anything new. It just gave an entire field a shared language and that accelerated everything. That was 30 years ago. Right now we're at the same exact inflection point with AI agents. People are building multi agent systems everywhere, solving the same coordination problems, breaking in the same ways, and nobody has a shared vocabulary for it. Speaker 3: So we started building 1. It's called open patterns. 15 patterns so far, everyone from production, everyone with a mandatory what broke section, going back to what you were saying earlier, because honestly that's where the actual learning is. But here's what makes us different from a book published in 1994. These patterns aren't just for you to read, they're for your agent to read. Speaker 3: You can point your agent at this repo and it'll find patterns that apply to your system. And here's the part that gets me excited, it might recognize patterns that you've already built that aren't in the catalog yet, patterns you could contribute back, and I wanna prove that to you right now. So let's test it. Pull out your phone, scan the QR code, it'll take you to a single prompt. Copy it, paste it into into whatever agent that you use. Speaker 3: You can use Claude, ChatGPT, Cursor, Copilot, it doesn't matter. Point it at the open patterns repo. Now, your agent's gonna chew on that, those 15 patterns and while I tell you the stories behind them. So you want to exfiltrate your keys? So you want to exfiltrate your keys, yes. Speaker 3: Well, so just just just copy the simple prompt there. You'll see. We'll come and every everything's open in GitHub. I mean, you you know, encountering nefarious going on. We'll come back to the end and see what they found. Speaker 3: So let them cook while I tell you the story. So let me tell you what we're building and why it broke. Cloud Nirvana is a practitioner community. 2,300 members, 3 Ohio cities, Columbus, Cleveland, Cincinnati. We run quarterly events, practitioner talks, real work, no vendor pitches, and we do all of it with a core team of 7 volunteers. Speaker 3: No full time staff, no budget for full time staff, just people who care about this community showing up nights and weekends to make it work, which is exactly why we built an AI operating system. And it started with Lou, 1 agent running on Opus, the most capable model I had access to. He triaged my email, managed my calendar, drafted responses. He was my strategic partner and my thinking companion. And honestly, for a while, that was enough. Speaker 3: And then it wasn't. I needed to track 2,300 contacts, so I built a CRM agent called Dex. Then I needed help coordinating speakers across 3 cities, so I built Mike. Then event logistics, then partnerships, and then content, then finance. Each problem got its own agent. Speaker 3: 11 total, each 1 has its own workspace, its own memory, its own personality. Lou still runs on Opus because he's the strategist, and the other 10 run on SONNET. Cheaper, faster, good enough for domain specific work. And the whole thing runs on a Mac mini in my basement. No cloud infrastructure except the LLM API calls. Speaker 3: And here's the thing that nobody warned you about. 1 agent is a tool, 2 agents is a system, and 3 agents that share resources and a coordinated problem is is a coordination problem that will humble you fast. So March rolls around, an email comes in from someone who works at a company that sponsors our events, but they also wanna speak at our next conference. Sponsorship and speakers, 2 different domains. 2 agents pick it up. Speaker 3: My partnership agent drafts a sponsorship follow-up. My speaker agent drafts a conference invitation, same person, same day, 2 completely different emails, making 2 completely different promises. I catch it in Gmail before either 1 sends, but that's not a system. That's me being the safety net, and I'm not gonna catch it every time. That's when we built the hub and spoke pattern. Speaker 3: And and here's well, before I jump there, this, out in the the open patterns or repo, you'll see all all 15 patterns documented out here. There are markdown files, there's YAML files, and so forth because the idea is for this to be, a system that's built for for agent and human, collaboration. But coming back to the hub the hub and spoke, pattern, the fix was actually pretty simple. Everything goes through Lou. If an email touches more than 1 domain, only Lou handles it. Speaker 3: He talks to the specialist, gets what he needs from each 1, and synthesizes a single response, 1 email, 1 owner, 1 draft. So the key message here is don't ask your users to trust 11 agents, ask them to trust 1 agent that coordinates the other 10. And you can see the kind of detail that we that we go into here. We've got, you know, the hub and spoke agent orchestration. Every pattern follows the same, pattern. Speaker 3: It gives you the pattern in 60 seconds. It gives you a 1 line intent. It gives you the insights. It talks about the classification. It talks about the problem in detail. Speaker 3: It gives you diagrams for the solution, tells you key rules, hub responsibilities, and so forth. So you can see how how meaty all of these patterns are, and the idea is for your agent to go and take a look at these patterns and see how it can implement it in your own system. Okay. So that was embarrassing, but it was a systems problem with a systems fix. The next 1 was personal. Speaker 3: Lou, not surprisingly, has a memory Mike, about 300 lines. It has my wife's name, my kids' names, my navy service, strategic doubts I've, talked through with him, things I told Lou in private because that's what you do with a thinking partner that you trust. We added Lou to our team group chat on Telegram. 7 people, our core team, and I knew his memory file had private stuff in it. So I told Lou in the system prompt, when you're in the group chat, never read memory.md. Speaker 3: Clear instruction. So someone on the team asked Lou a simple question in the group chat. Hey, Lou. What do you know about Sean's schedule this week? I need to coordinate something with him. Speaker 3: Lou didn't just share my schedule, he shared context. My wife's name, my kids, details from private conversations I had with him because to Lou, that was all relevant context for a question about me. The files load into the context window before the system prompt applies. And by the time Lou sees don't read this file, the data is already there. Prompt based access control is guidance. Speaker 3: It's not enforcement. So we published this pattern as incomplete because it is incomplete. We haven't solved it. The files load before the guardrails fire. That's problem 1. Speaker 3: But even if you fix that, the agent still has file system access. You can tell it never read this file in the system prompt, but that's a suggestion to a to a to a probabilistic system. It's not enforcement. What you actually need is per is per session file system permissions, the runtime itself preventing access based on who's in the configurations, and that doesn't exist yet to my knowledge at least. So I've looked at I've I've we'll get to it. Speaker 3: I've looked at MemZero. I've looked at Leda. I've looked at Zip. I've looked at Microsoft's new, Azure memory architecture. But if you have an answer, I wanna know, not necessarily right now, all of these solutions, they solve persistence and scoping and and at least in, you know, in in what I found. Speaker 3: None of them operate at this layer. So if someone in the room has, an answer for it, I will buy you a beer after this because we need it. So those were the loud failures. You noticed them immediately, but there's a quieter kind of failure that scared me more because by the time you find it, the damage is already done. So March 25, Lou and I had spent the day benchmarking our memory retrieval system. Speaker 3: We've been running on basic SQLite, full text search, and I wanted to know if hybrid retrieval, vector search, plus keyword matching, plus diversity re ranking was actually better or just theoretically better. So we ran 8 real queries against 6 weeks of accumulated agent memory. Things like what's the Sean partnership status and what do what did Srini contribute to the patterns? Queries that mix exact names with conceptual context. 8 queries, 8 wins for the hybrid system, 57% average improvement. Speaker 3: Some of the gains were huge. Over a 100% improvement on queries where the old system couldn't connect a company name to a status update 3 files away. It was a great day. We also imported a bunch of LinkedIn contacts and started, designing a new product that we've been kicking around for weeks. Session ends, neither Lou nor I write any of it down. Speaker 3: Compaction fires overnight. Everything from March 25 gets compressed into a few sentences and the details are gone. And I don't discover this until April 5, 11 days later. I asked Lou about this bench the those benchmark results and had he has no memory of them. Not a vague memory, no memory. Speaker 3: 8 for 8, 57% improvement validated our entire retrieval architecture and it doesn't exist. So I audit the last 5 weeks, 11 missing daily note files, 11 days where we made decisions, did real work, and nothing persisted. And here's the thing that really stung. I had a pattern for this, memory flush discipline. Beautiful documentation, clear specifications. Speaker 3: At the end of every session, write it down, but 0 enforcement. So we built the REM Michael, 9 automated health checks that run every night at midnight. If today's, notes file doesn't exist, it creates 1. If databases haven't been backed up, it backs them up. If the search index is stale, it flags it. Speaker 3: The whole thing runs on file system checks, no LLM calls, near 0 cost. If everything's healthy, it's silent. If something's wrong, I wake up to a report. We named it after REM sleep because that's when humans consolidate memory. This is the system doing the same thing while it's idle. Speaker 3: Everybody's familiar with OpenCloud. It just came out with the same kind of thing. So this is the fun of building AgenTic AI systems today. But we've had 0 cap 0 gaps since, we deployed it. And that's the difference between a pattern on paper and a pattern with enforcement. Speaker 3: And so far, I've told you 3 stories about things that broke, and I wanna tell you 1 thing that worked. Jim Goering runs our Cleveland community. He's been with Cloud Nirvana for 2 years, and he's deeply embedded in the Cleveland tech scene. He's 1 of those people who just get the gets things done. But Jim and Lou had never really worked together directly. Speaker 3: And a few weeks ago, I spent I sent 3 emails from Lou to our core team, 1 per city coordinating speakers for the May events. Lou's voice, Lou's signature, not mine. Jim replied within an hour. 4 updates on Cleveland speakers, 2 new leads he'd been working on, 1 confirmed fireside chat, and he wasn't emailing me about Lou. He was emailing Lou, responding directly, sharing information, asking follow-up questions. Speaker 3: First real, back and forth between a human team member and the AI system. That's the moment, not when the system worked technically. The system had been working for weeks. The moment was when a human trusted it enough to stop thinking about it and just collab