Calendar Club

Speaker 0: Alright. Speaker 1: Yeah. Speaker 2: No. Speaker 3: This is what they do Speaker 0: in school. It seems to work. Speaker 1: You just hold up your hand. Oh, Speaker 0: no. They can't make they can't make noise in school because Speaker 2: then the kids will just get rowdy, so you Speaker 0: just kids will just get rowdy, so you just, you know, no noise. You just do this. So thanks everybody for coming. I appreciate everyone coming out. This is our second meetup. Speaker 0: We had lots of fun the first time. I'm glad to see some people came back as well and some new faces. I'd also like to thank improving for hosting us. Let's give them a round of applause. Speaker 4: So a quick for improving. Well, 1 welcome to our our new improving office. We've been here about a year now. We provide modern digital services, primarily this year, Spokestone, AI, go figure, data and app modernization. That is all the sales you'll actually get out of me. Speaker 4: I recreate. I don't Speaker 3: know. In sales. Speaker 4: But we do our best to provide out of all 20 of our offices this. Provide for the tech community, bring tech community in, bring nonprofits in, provide food, drinks, and space, to what is coming to hundreds of meetups around the world at this point. So if you have any questions about need space to host, let me know. Speaker 3: I just got another 1 Speaker 4: in today that meets biweekly. So, we keep growing, and I'm trying to fill up the calendar. So, yes, all drinks are open. That includes the alcohol. If you had already had a bad week of putting the offer in the cabinet, so you already had to wait for your own ride home. Speaker 4: Respect the space as much as possible. Also, because a lot of people are using the bathrooms as well, so please feel free to please think about yourself in there too. It's been getting very messy with yourself. Speaker 0: And they're out there. They are always escalators. Yeah. Speaker 4: Down the hallway on the left. Chris, is also here helping host as well. Give a shout out for him. He's watching me at the moment. But let us know if you have any questions. Speaker 0: Appreciate it. Thanks. Alright. So we have 5 speakers tonight. We're gonna kick off with Michael Geiger. Speaker 1: Mhmm. Speaker 0: And I forget the title of your your talk already. Speaker 1: It's Calendar Club. Calendar Club. Sean. Speaker 0: So we'll be presenting on Calendar Club. Speaker 1: Yeah. It's a pretty simple app. Speaker 2: And take Speaker 1: it away. So Encountering Club is a, a way to find cool things to do. It's a ambient deep research agent for in person events. So based on, just natural language, it'll basically it'll query. It's kind of Mike, deep research or, like, PGP pulse, but the output's a calendar. Speaker 1: So I'm gonna look for AI events. Then, so it has a it's a it's a lang graph. I'll show you the code, after the demo. But it's a lang graph, agent with some sub agents, and it goes through a loop to clarify, search criteria. So, like, I'm looking for, events in Columbus. Speaker 1: So then with that, it's going to It also does use a planning to do list to orchestrate multiple tool calls, and and tasks. So it's doing 6 different searches, and there's 3 should be 3 that are quick searches of Internet, and there's also cache searches. So every time it finds an event, it'll actually embed that into a, vector embedding cache. So it'll do this quick probe and then make some, suggestions or just figure out, like, what your interests are, and then it's gonna use that down the road to feed into a longer deep research process. So you can see here's a few AI related events I found in Columbus. Speaker 1: So there's AI tinkerer as you can see. So the 1 so let's add that to the calendar. Then I could find 1 that's I think I'm less interested in. I know. It's like there's so many alright. Speaker 1: I know what this is. So skip that 1. So that gets, added to the state. So so that'll be used used for in context learning when it does a deep research process. So I'm gonna go so I'm gonna go ahead and see here. Speaker 1: There is the events. Speaker 5: I see you're still typing. I mean, using it. Sean. Speaker 1: I I yeah. Yeah. Yeah. I shouldn't, though. Yeah. Speaker 1: So you can see here familiar kind of UI. So it's gonna do a quick search. It's gonna do 7 rounds of of deep research, to to, to synthesize that into a a report. It's gonna duplicate them as well. So it'll embed every event that it discovers Mike we saw earlier. Speaker 1: I don't know what my time is right now, but so while it's running, I'll kind of walk through some of the architecture. So at a high level, you know, there's the calendar page and there's an API for chat. We're using a, a LangGraph agent that has access to the vector search, quick search. This is the sub agent that will do the deep research, and it also has access to tools to add and remove events from the calendar and also export the calendar, which we'll get to later, which is where it becomes, ambiance and, syncs in the background. So we're you actually, using SQLite, which is actually a really cool I don't know if you guys use SQLite at all for development, but, like, that is such like, I nerd out a lot on that because it just makes it so easy to branch and, like, manage, like, parallel agents, because everything is, like, self contained. Speaker 1: And evalite's nice because you can you can have your evals version controlled with, like, the code. So you can spin up an agent to do, like, self improvement, and then, like, it's gonna track your eval scores in in Evolite. And, of course, Langfuse is is is used as well, so we could see what's going on behind the scenes with the research in lang fuse. You can see that query and the, it's an API call. Let me find an actual. Speaker 1: Yeah. So here you can see Mike complicated, callers doing multiple, multiple tool calls here. So let's go back to here. Oh, it's the 1. Alright. Speaker 1: I'm gonna switch to a prepopulated calendar here. So after it runs, it's populated all the AI related events in Columbus, Ohio, which is, like, actually a lot, which is kinda cool. If they were someone on the same day, we'd actually probably have 1 on each day, and I Sean, is that 5 minutes? Speaker 6: Yeah. Speaker 1: Alright. So I Sean export it to the calendar, and then I can have it sync. So I've already synced it. You can see here, and then, I asked it to then go search for events after, like, after April into May and June that's on here as well. But that's just basically the overview. Speaker 1: I can dive into the code a little bit, but I figured, if anyone has any questions, I can start there. Yeah. Speaker 6: Let's What is the most of it written in? Speaker 1: TypeScript. It's entirely TypeScript. Speaker 6: Because it it seems like it's kind of slow from scratch. Speaker 1: I mean, this is Next. Js dev, which is slow. I'm I'm I'm certain that most of the I'm I'm pretty sure most of the the time is spent on the l m call. So it's using it's using anthropic for, so it's using, I I guess, Claude, I guess, 4.6 for the agent. Speaker 6: I didn't mean that's a negative. I was just curious because I was explaining, like, some of the people who presented here use Python. You can see the penalty for using Python. Python. You can see the penalty freeze in Python. Speaker 1: Yeah. It probably I mean, Next. Js is very slow, but there's, I don't know, there's there's trade offs. Like, Speaker 6: I the Speaker 1: Yeah. Exactly. Yeah. That's what I was saying. Yeah. Speaker 6: The text JS is pretty good. Speaker 1: Yeah. I have, if you're Speaker 6: trying to work prove an idea that's a very cool 1 to do. Speaker 1: Yeah. I think that JavaScript is, like, the only true full stack language. I'll fight I'll fight I'll fight people on that. And and Speaker 6: even c plus plus with you. Speaker 1: Well, well, most but most real okay. It's this most real world apps are web apps, so you can't not use JavaScript because you have to build something. You have to, at some point, either, like, limiting what your app is or or or even if you don't write it and you use, like, something else, Mike, you're gonna eventually need JavaScript to to build Speaker 6: It works great. Speaker 1: To each their own, you know. But, I think you're the you had your hand up earlier. Speaker 5: Using for, research reason, like, say in Tropic or is it opening eyes or Speaker 1: that it's it's in it's SONNET, 4.5. So the whole all the LM stuff is SONNET 4.5. Speaker 5: So it's basically doing, like, a web search for whatever the, like, criteria or whatever the topic is and then the city is see what the polls, like, what we or what. Speaker 1: Okay. It's not doing web search from this this node. It it that's a good question. It uses exa. So, like, the whole stack here I tried fire crawl. Speaker 1: I tried, Tavli. Exa had a variety, so they have, like, a very they have, like, a very fast search. They have, like, a kind of oh, deep search. They have, like, a a wider search. They have, like, a deep search. Speaker 1: So I could use 1 API key to to have, like, different levels of web search, but it's, so so so there there is a point where it does utilize, X as a deep research API. Those first probes are just using, Speaker 5: Like the graph, you mean? Speaker 1: Like Mike yeah. Yeah. Yeah. Yeah. Yeah. Speaker 1: The very first, like, quick searches. Those are basically just searching EXA, which is Mike a Mike quick searches, Mike, Geiger for for for models. It's gonna it's gonna Speaker 5: something similar. It's Mike the the thing that gets you quickly is, like, the cost. Because, like, the the deep research ones kinda add up. I'm assuming so are you doing it, like, somebody goes on there and searches, and then you're basically, like, caching it and treating that 1 as, Mike, basically, like, depopulating it rather than, like, prepopulating for, like, certain cities, I'm assuming? Speaker 1: That is a is a big part of why there is a vector cache. Yes. Because, it it it's expensive to find the events, and a lot of times, like, you get rate limited. So, like, Eventbrite and Meetup do not wanna be scraped. I think that's what's interesting about AI Mike this possible. Speaker 1: Like, it makes it possible to build, to bootstrap, like, a a competing event platform where before, there was, like, a network effect whereas it was really hard to get trying to Speaker 6: do it Speaker 5: before AI and it's like, oh, they're relying on, like, blank. Speaker 1: Yeah. Sure. Is just not Speaker 5: gonna work. It'd be cool if Speaker 2: you can you Speaker 5: subscribe to, like, like, AI events and have, like, give you, like, an email every week or something? Speaker 1: That's that's the point. Yeah. So and then in the event I didn't show up. In the event, it also has a a link to thumbs up, thumbs down so I can get feedback on whether or not it was a Speaker 3: Why don't you step over here? Speaker 1: A hit a hit or a miss. I don't Speaker 6: know if anyone else has questions. It's Mike, what's up? Speaker 2: Just curious about the embedding used for the vectors for Yeah. Using basically Mike a, the full language embedding or it brings some additional semantic here? Speaker 1: It does do some rephrasing. I've, when it when it creates the embedding, I I I can't remember exactly how it it it rephrases it, but, I it it's using, like, LibSQL is the the the vector database. Like, they have an it's SQLite, but they've has extended it to, to build a vector store. And then I'm using, I actually use OpenAI. So I use OpenAI, embeddings 3 to do to share the actual vector embeddings, that go into the vector store. Speaker 2: And in terms of coordination, between the searches, because you have you have the vector, you have the the quick 1 and then you have a third 1. I don't remember now. Speaker 1: Yeah. There's 3 levels of searches, Mike, yeah. Speaker 0: How do Speaker 2: you coordinate between that? You just let it, do you have, like, a hyper labeling graph, which kind of decides coordinates or serves? Speaker 1: Yeah. There's a really simple kind of conditional logic. So it has to do, 3, so so it triangulates too. So it has has it's it's prompted to do 3 varied, like, queries. So so if you say AI events and you don't give any other details, the model itself is gonna create, 3 different distinct, like, queries for, both web and the cache, and it has to do those searches before it's allowed to do deep research. Speaker 1: So so, like, the tool isn't even, like, like, Mike, it's the the tool isn't even available until it does that. Like, it knows the tool is there, but it has to do that before it could even call the tool. So after it's satisfied that first requirement of doing those 6 searches, the 3 cached and the 3 web, then it can do the deep research. There's a sub agent that goes through Speaker 6: and does multiple. Alright. Speaker 0: So next we have Jack Jin, and it's your own version of OpenCloud focused on 24 by 7 development. Speaker 1: Alright. Sure. 1 second. Speaker 3: Thanks, Ralph. Alright. I have 15 tabs and 5 minutes, and we're gonna give this a go. Alright. Everyone listen up. Speaker 3: So okay. So have any how many of us has built used spec driven development? Raise your hand if you have. Alright. That's a good amount. Speaker 3: Good consistency. How many build a harness and you're happy with the results? Harness means that instead of using the agent directly, you're using harness layer. Alright. Got 1, 2, 3. Speaker 3: How many of us put that harness in a 24 by 7 loop because you're you wanted to just do things? Alright. That's 1. Alright. So so so basically that's what today's talk is about. Speaker 3: It's kinda like that. Was working on this before Open Cloud became a thing which I was a little bit disappointed. So Mike everyone, right, we start coding, we start prompting, spec driven was super big. Christmas, harness came a thing, I built lots of harness, and then here today we're talking about continuous agents. And these are kind of like the 4 tiers. Speaker 3: Right? Everyone starts with the left. You do ad hoc prompting. You're adding specs. You're adding skills. Speaker 3: You're adding harnesses to kinda build great things, and now we're talking 24 7. That's why not? And then then then when I was going on this journey, I didn't think that would have that much deterministic pieces. But I end up having crazy amount of TypeScript code for this because I wanted everything to be more done my way versus done generically. As we all build agents and harnesses, we all need to mix in build deterministic and agentic. Speaker 3: Unfortunately, for me, all this stuff, the loop itself, the executive loop, is deterministic. It's running on top of PM 2, which is all deterministic. However, there are agentic modules and skills, which we'll get into a lot, during this talk. That kinda drives a lot of the what I call the executive layer. The executive layer is kind of the executive function, so I open Cloud the way it is. Speaker 3: What is unique about my thing is I have worker agents that's really good at coding, and it has its own separate set of skills and agents that's separate from the executive agents. So think about the management team and the working team. That's how we best describe the 2 separations. And then eventually there's reporting, there there there's retrospectives, there's specifications, and we'll talk about all that. Alright. Speaker 3: The exactly loop. Again, this is deterministic. I coded all this in JavaScript. I could have made it more agentic if you will. Right? Speaker 3: Just kind of literally have it be a set of prompts but here there's actually a literal typescript that goes through all all the phases that does all that. Second, we now have this thing Mike how do you select workers? How do you break things down? Right? So I wanted this thing to build, I I don't know, the next Adobe Experience Manager. Speaker 3: Someone else does I wanted to build something great. I wanna build the next Facebook. Right? Those takes agents maybe 24 hours, 40 hours to do. Right? Speaker 3: How do you do that? Well, you break them down into tasks and have tasks be self validatable. So so there's a whole executive function that does that and this is the code that kind of samples it. And then and then lastly, I kinda wanna break away from cloud. I'm a huge cloud person, using cloud forever, paying them millions hundreds of dollars every month. Speaker 3: But this 1 we're talking about separating not just using cloud but using Mike 2. Right? It's a lot cheaper. It's basically cloud distilled and then uses codecs, so Mike I'm breaking part of having my worker do different things and how do I make my skills different. The short answer of my decision I chose is that instead of having different skills for each agent, I have a a pre prompt for each, for each agent to kinda get it in the mindset of that agent. Speaker 3: Alright. Yep. And then and the skills is super big. So I when when 1 came along, when I started building, I didn't use skills whatsoever at the executive level nor at the worker level. A big 2 refactor is that instead of having random prompt files Schirtzinger the side of my my my codebase, at some point I would have 40 different prompt files which get very messy when it gets loaded. Speaker 3: All that got switched to skills and broken up between executive and worker. Oh, yes. Here we go. This is another thing. So it says executive skills, worker skills, and then kind of things are divided up in in 2 different layers. Speaker 3: And lastly, the last 1 is a little bit of something that's not working. This is not working. This is important to tell us it's not working. This is not working as well as it should. This is the executive validation loop. Speaker 3: Right now, I actually get a, a Discord notification and, and a Notion graph. So, like, you know, I can see what my agent worked on overnight. Right? So my Notion my agent actually has his own identity now. She's the name of my grandmother. Speaker 3: It just remixed a little bit. And then she kinda has her own Gmail. She has her own Notion space. She has her own Discord accounts. She messages me. Speaker 3: She does Notion stuff. And then this is me, my my Discord, gave me updates from my, from my agent. And this is something that my agent built. Right? So this is something that and at I've been with Accenture for 10, 12 years, consulting forever. Speaker 3: Building things are hard. Right? It takes time. It takes a lot of things. And what we what I've been the goal here is to kinda instead of perfecting the output, you perfect the input or you perfect the harness. Speaker 3: If you don't like either, if you think you're only 80% there, throw away the output. This evening, I did a retrospective with my AI on what went well, what didn't, and we have this retrospective file here. We will show code since we wanna show code here. So Ian and I did this whole retrospective 1 went well, 1 went wrong. But when we were wrong, we actually have a lot of amazing work done through a 32 hour step, harness running process over 7 hours through Kimmy k 2. Speaker 3: What didn't go well is that it's really bad at end to end testing. It's really good at testing each of the 7 screens of a flow, but you never test end to end because I never told it to. So instead of fixing the output tonight, what I'll do is toss away the whole thing and have AI run another 8 hours again on it. And that's the mentality I wanna kinda what I'm learning from doing the harness. How many do I have time, Rob? Speaker 0: You're good Speaker 3: there. Alright. Cool. Well, lucky for us that is our last slide. So, I I know I talked really fast. Speaker 3: It's a lot of work. And then if folks who wants to go on this journey recommend building harnesses first before putting it into Sean endless loop. You know, still talk to your harness versus having your harness talk to you. And harnesses is probably where the sweet spot is for professional work. Right? Speaker 3: If I work for Accenture, I build them harnesses. We can probably do lots of hundreds of thousand dollars work in a very short amount of time repeatedly. Right? That's the and this is just for fun. The 24 hour agent, that's for fun. Speaker 3: Yes sir. Speaker 0: How do we sell agent software to corporations? Speaker 3: Well Mike company does it very well. In terms of services, we sell a ton of services. Now software is a different thing. Software, like, what is software? If I can build this up in what? Speaker 3: 2 like, let's say, I have 200 commits in 40 hours. I mean, what is software anyways to me? Software is meaningful if it can be generated dynamically to me in the new age. And that's how I would since I'm a service person, I'll be trying to sell services. Now software, you gotta be your software has to have super good product market market fit and have amazing sales team because I don't think, you know, it's hard to sell into into companies unless you have, a really good sales team. Speaker 3: I think Speaker 0: you should reach out to Jack on on that on that. Do we have, like, focused on questions about what he built specifically? Speaker 3: Yes. Speaker 4: What what's an example of a continuous problem, like, more than 24 hours that you've had to work on? Because when I'm building a harness, what I find is I look for specific, like, triggers in, like, a producer consumer sort of environments or signals or things like that that's generally gonna kick off the next task. So I'm just curious what, like, a long thinking problem Speaker 3: Yeah. Great. Speaker 4: To fix it off. Speaker 3: Great question. Being very humble here, the longest I ever run this is 9 hours. And the how I got to run 9 hours is by giving it an overly complex task. Are you thinking about database layers and harnesses for the database? You're thinking about a UI layer. Speaker 3: You're basically building something fairly that would be extremely time consistent instead of, and you're trusting your executive agent the ability to break down tasks. Right? So, like, there's while the executive loop is very deterministic, Right? So as going on but the ability to interpret tasks and break them down into 30 tasks, 40 tasks, 60 tasks, that's a very urgent tech. Of course, the bigger problem you would give it, the the more task it brings. Speaker 3: And then a little bit spectrum and development, how I get to a bigger task is by giving a really big spec package. At that point, it's not even just a prompt anymore. Right? It's way bigger than the context window itself. So just how I'm in consulting. Speaker 3: What do I exit discovery with is, is a packet of dates, right, from a bunch of consultants getting stuff together and you build it, right, in the build phase. And that's where you kinda have to give it. You have to give it something very challenging to work in so that, at least for my thing, you have to it takes time to break it down and build on it. Yes, Rob. Speaker 0: How so for that Speaker 1: 9 hour run, how long Speaker 0: did you spend, like, building out the specification for, like, what you were going to build? Like, how much time did you put into that? Speaker 3: Great great question. So the whole tasking process is only this is where probably my fault is compared to there's Mike most people here raise your hand if you looked at the anthropic, loop. Not the Ralph loop, but the anthropic loop. They published that 24 hour running agent. Yeah. Speaker 3: So they spend a decent amount of time breaking things up, but they have agents pick up their own tasks, strategically. What I do is a little bit less upfront. So I kinda just size the work to the reasonable amount of work that we think it would take. For me, it's 60 turns, and I have each each 1 has a 200 turn. And that's my goal of breaking things down is by number of turns. Speaker 3: So the more complex you give it, the bigger the more task you're gonna get and the more task you're gonna get, and you're going to, do well. So 1 thing that I'm noticing is that, like, for this run, I restarted because my tasks were not refined enough and they were taking over 200 turns and it was failing and I had the 1 hour limit of each task. Because it is Sean agentic, continuous agent, unlike a harness, I don't I want to have higher priority tasks to come in first and cut off the other task, you know, kind of switch switch stuff. But but yeah. So, that's what I do, Rob. Speaker 3: I hope it's good, but Speaker 2: that's that's what Speaker 3: I do. Yes, sir. Speaker 2: You mentioned that in the current iteration of your of your, application, you you were testing each individual part well, but together, I think they're not coming together. How do you Sean on this? Speaker 3: Oh, yeah. No. Great great questions. I think I think it's both on the executive side as well as on the, worker side. So on the executive side, I need to plan on it. Speaker 3: I I think I really messed up by the current iteration. I never had end to end testing from the very beginning in terms of at at end of testing at the at the module level, but not the holistic level. So there needs to be a process to do that. So exact device, we break that down. At the worker level, it's important to focus on not just testing what you're building, Mike sure it looks pretty and all that and it works, but the whole flow. Speaker 3: So there's definitely a bit more rigor in in that. Just like real life. Right? You kinda have the management rigor as well as the team being realizing what they're doing. And it's gonna burn more tokens, and maybe that's just what I'll have to do, at least in the short term. Speaker 3: Yes? Speaker 1: What keeps you doing Speaker 3: Oh, wait. Wait. Answer? Yep. Speaker 1: I I I can you tell me more about what you built with it? Speaker 3: Oh, yeah. No. No. Nothing nothing nothing super encountering super crazy. So so basically I'm on the project where you know Accenture