What Is Your Coding Agent Actually Searching?

Speaker 0: Hey. Thank you. Speaker 1: AI gonna let that run. I don't know how good your Internet is. So I have a very, very specific goal. I would like to try and beat BJSS at AI code search. Nice. Speaker 1: So fast. We love Tecstone. Actually, I'm not even joking. I normally run just, like, straight Linux on my MacBook, and I had Tools, in, like, 5 minutes, run this thing on my Mac with no developer tools installed, and this ran just fine. So I'm pretty proud of that. Speaker 1: Anyway, I have a very specific goal. I would like to beat BJSS. I think that structured code search tools can perform much better than, bash in some cases, and I think ideally in a lot of cases. Because I think what we can do is encode the heuristics about how your codebase is shaped in a deterministic Tools, so that your age Jun doesn't have to relearn your code base every single time you're running this thing. And you could just make it an MCP tool Sai that you can just attach it to whatever agent you care about. Speaker 1: It seems like we're all finally gonna consolidate around MCP or something like well, this I mean, I hope, you know, AI, you could right? But, you know, AI I'm making a bet that MCP is gonna work out Sai that I can develop this tool that enables us to better understand our code basis. It's it's for the human and the agent to understand the code base. So that's a pretty strong claim. How do you do that? Speaker 1: How do you beat bash? The way I did that is by separating this into 2 showing. I made a harness, and I made a tool. The tool specifically has 2 knobs that we can fiddle with Pun let the agent optimize over. It has fuzzy search, and it has look ahead. Speaker 1: I'm sure most of you know what fuzzy search is. I'm not gonna waste the time explaining that. Look ahead maybe needs a little bit more elaboration. So look ahead is essentially Jun, I'm already representing this code base as a graph, as a lot of structured codebase tools do. All I'm doing is I'm just looking ahead, like, literally, like, they query for this symbol, this function in the graph. Speaker 1: I'm gonna look 4 steps ahead or whatever. But here's the key part. That decision about how far ahead it's looking is made by the agent and is encoded into just a Python file. That's it. So I'm gonna start this thing running. Speaker 1: This is gonna run my harness, and we are gonna reproduce a result that I had earlier. So I just installed the Cerebras key, so AI don't know if this Workflow. Jun gonna focus for a 2nd. Speaker 0: I Speaker 1: want you to reproduce the ablation results we had Presenter with g Coding much. Only run a slice of phone, only run 3 games. You can borrow completely from the previous ablation rounds, but I would like you to create new bundles. Well, before I do that, I should probably really let this thing go because I need to talk. Yeah. Speaker 1: I'm doing this live. Don't do this. Okay. So oh, make sure to run the pre AI that will tell us how much this will cost. So I'm gonna show you what the bundle looks like while this runs, and I'm also gonna show you what some previous cost metrics look like. Speaker 1: I also have another unique constraint alongside, you know, wanting to beat BJSS, which is the thing that everybody else uses. I don't make a lot of money, so I don't have a lot of money for tokens. This is probably the stupidest run I ever ran, and I was like, I'm just gonna let it go AI, like, 50. And I was like, holy LRASPP. Stop. Speaker 1: Stop. Stop. And I couldn't make it stop fast enough. So in response, I was like, I'm gonna make as many guardrails as I run, and then derive as much information as I can from as many traces as I can. And I'm gonna put it in 1 bundle from a format that's as easy to read as possible and as checked as possible in a pure way. Speaker 1: So I found a language called Pkl. Pkl comes from Apple, and Pkl is really pleasant. I really like Pkl. Can you guys see the screen? But the nice thing about Pkl is that I can have an Event, gross Go AI that is performing all sorts of unpure calls and doing things that scare me, but I can have this nice little world over here about what it's actually Coding, and it's typed. Speaker 1: Well, I mean, you have docs and stuff like that. So it means that every single run that I'm doing, I can very easily reproduce secure the whole world is modeled in this Schema. All of it. So we have scoring, we have the contention of the LRASPP so the scoring is deterministic too, and then I can go look at the evidence. So I Actually, because I knew I needed the data, Jun committed all of these logs. Speaker 1: This is AI millions of lines, but I am a big fan of Git, and I think that Git is a better data store than you think it is. But there are interesting things that this static bundle format can give us. I'm not gonna talk about bug. It allows us to really easily visualize over these results. Let's see how it's doing actually. Speaker 1: Okay. It's AI some rounds. So what I'm working on right now is a nice visualization of what it looks like, but for now, you get some some graphs on my block. So I'm gonna show you some results, some actual graphs, and this is not rendering correctly secure my other Linux version has a different resolution, but that's okay. So I have some really strong results already. Speaker 1: This is a comparison against another tool that is paid called JCodeMunch. J Code Munch advertises a 95% token reduction on certain cases for agent code search. The what it's not telling you is that if you go to the ROI calculator, it's like, you know, not as good as it's saying it is, but real results. So I was like, okay. I want to compare myself against this tool secure it's already quite strong. Speaker 1: Let's see. How's it doing? Speaker 0: Uh-oh. Speaker 1: I'm kinda glad things are failing. I'd rather it fail than spend money. So it's already quite Street, so I wanted to compare against this already strong tool. So what I did was I gave them the same Workflow tree interface, the same scoring interface, and the same everything. And the way that I'm scoring them is pretty interesting. Speaker 1: Inside the harness, we have basically 3 tree sitters. So all of these tools that do structured codebase search, they use tree sitter, which is a tool that gives us the nodes and edges of a code base. I have my own version of of tree sitter in the harness, and then Checode Munch has their own, and I don't care about their life cycle. And then I have my own, and I don't care about my life cycle. But the reason why I have a TreeSitter instantiation of the graph inside the harness is because I can literally see how many hops away the three result is from the actual file. Speaker 1: So the way that this works is I'm using a Date from JetBrains called Cell c I don't have my search history on this Mike. But, it's it's a Date that is pretty simple. They have an issue, and they have a golden file that the issue is supposed to find, and that's it. So it's a complete end to end run. The the agent is just Speaker 0: AI, Speaker 1: go figure it out. The only thing that I'm bounding is it's it's number of tool calls and the tool that it's exposed to. And I can actually show you the system prompts for those Tools, if I can go find it. They're in Neatsmile, which I thought was kinda clever. If introduces interest oh, no. Speaker 1: I don't have the selection. Speaker 0: Alright. Speaker 1: So, you know, here's the prompt. You are an evaluator agent. Use the available Tools to localize the bug. And then I have Jun then, again, this is very AI. It's because most of it is defined in Pkl. Speaker 1: So I'm now gonna stop API, and I'm gonna see how this thing is doing. Speaker 0: What are we doing? The Jeffery was an l LSP? Speaker 1: Yeah. They make coding Tools. Some really nice ones. They actually have a new agent they released that looks interesting. Sometimes it can take a little bit because my codebase is kind of AI. Speaker 1: Oh, yep. That's the problem. So every Winkle, file inside my code base codebase has a AI, and the way that we do these pre Live calculations that tell me how much things cost is that we do an aggregate over all of the traces and all of the tokens spent, and we literally calculate how much we expect this to cost based off the bounded number of tool calls that we defined in our Pkl config. And because of the fact that I Agent told it to make a completely new bundle, it's AI, there are literally no runs that matches my exact configuration. Uh-oh. Speaker 1: So Sai don't know. But the point of this Select, and Dental lot of the systems choices I model, were to make things as reproducible and as safe as possible. I would much, much rather this thing fail a lot than run like like this. So as you can see, it's gotten a little bit Center. We Ariso spending less tokens, but the point is that I'm trying to catch as many things that Folder the out of these calls as possible so that I can have these nice graphs. Speaker 1: Now, I started this talk with a pretty strong claim sharing that I wanna beat BJSS. We have not beat bash, and it's probably gonna be a while before I think that I don't even know if it's possible. That's not Event really the Pun. Point. The point is to design a tool to learn about how agents search codebase, and maybe help teams learn about how their agents are searching code bases. Speaker 1: So if we go look at the BJSS results, yes, I perform much more efficiently against J Coding Munch, and the heuristics are nicer. J code Munch also just packs a lot of crap into their Mike server, which is silly, but they and they also just have so many more Tools. But Vue know, it's fine. So if we look at BJSS though, though I'm more consistent, I use more tokens. Where's that graph? Speaker 1: That's still j code Jun. Speaker 0: Blah blah blah blah blah blah. Speaker 1: So here's BJSS. So here's the completion Date, I LLM blue. We consistently get more correct results than BJSS, which is interesting. Bash is often usually very sharp, and because it's doing often just Street Searching, so it will literally just like, is this exact file right? No? Speaker 1: Okay. You know? And then Family, fuzzy search, unless the agent deliberately uses it, it doesn't use it. But I still don't perform much better than BJSS when it comes to efficiency, which was my target. But the interesting thing here is that I originally thought, based off how well I performed against J Coding Jun, that I would crush BJSS. Speaker 1: That we would be amazing. Because look at this advertised Presenter on their homepage, 95%. Wow. But the point is that the harness proved me wrong. By running just 10 runs of my thing, I was able to prove that BJSS is a very formidable opponent, which honestly just makes me wanna keep going. Speaker 1: So there's my tool. I don't know if this thing is still oh, hey. It produced it. Do you guys wanna run this live? Speaker 0: Yep. Okay. Speaker 1: Alright. I Route you to run this run this showing. Produce a report with the results afterward. And we can watch the search OpenAI when file this is showing. So if you guys don't know what cerebras is, most of my loop is just doing tool calls. Speaker 1: I don't really care much about reasoning because I'm testing the interface, not really the model. Cerebrus, what they do is they give you a, like, really cheap and fast not cheap. Fast model, that's dumb. That's really fast. They AI, like, 30 times, and they're right. Speaker 1: Like, I don't expect this to take very long at all lead it spends a bunch of time. I don't know. It's AI Sai need to add a bug AI. But the nice thing about Cerebras is that it means that my developer loop is very fast because all I'm testing is tools. So let's just watch this thing and see how it does. Speaker 0: And that Cerebras, Is it that LLM's behind the API Cell, so we're basically trying to Speaker 1: do that? They're just another OpenAI compatible inference provider. They're just actually, OpenAI recently bought them because they, are so much faster and perform so much better. They make a custom chip that is really big and does a lot of compute. Tools like it's running. Speaker 1: Can you Taylor the talk, please? Speaker 0: Phone teeth you mentioned it, but what's the interface AI to the LLM? Like, the shape of the actual Tools? Speaker 1: Do you wanna see like the MCP server? Speaker 0: Yeah. Speaker 1: Yeah. I can show you that. So if I go into AI it's literally just 4 so compared to J Coding much, it is a much smaller tool surface. It's Mike 4 tools. So if we go over to the AI. Speaker 1: It's yelling at me. Speaker 0: So Speaker 1: here we go. So we have Real. So resolve assemble to a graph node is just given a substring, look for the thing in the code graph that has that thing. And then here's the important fancy bit, we have expands, which is given a symbol in the code graph, expand Route. That's that look lead thing I was talking about earlier. Speaker 1: Resolve is actually secretly also doing a fuzzy Cell, and then resolve and expand, which does both. Right? And that's it. And then I have some admin tool calls I USA to, like, install the policy Jun, like, you guys don't need to care about that. I care about that, but, you know, I mean, maybe you care about it. Speaker 1: It's it's a pretty fun way of sandboxing. Instead of having I previously had, like, a work tree based Meetup where I would it was centered around that, but I moved to just I spin up 2 MCP servers, and I deterministically install the showing, and then iterative context actually just keeps it in memory. So instead of having to manage another file, it's just in memory a Python thing that we just exec on. And then I I should probably run there are some, like, sandboxing Python tools to, parse over your exec and make sure it's not doing anything X-rays, but Speaker 0: I suppose a whole lot of containers will take Speaker 1: care of that. Yeah. I should probably do that. When you run the Speaker 0: hops, did you run those, like, in parallel? Speaker 1: What was that? Speaker 0: When you run the hops, did Speaker 1: They're in parallel. Speaker 0: In parallel? Speaker 1: Yep. Yeah. So while the rounds run-in parallel. So I didn't leave City. I left it up to the Agent discretion about whether or not it wants to do parallel. Speaker 1: Pun, there's evaluate Render, and then there's, a matrix version which runs in parallel. And that's part of the reason why I switched to Go is because their concurrency model is really nice. So there's no red or blue coloring or anything like that. Speaker 0: So AI I see 1 of the Package files again? Speaker 1: Yeah. Pkl, by the way, is Jun, like, kind of a nice thing that I like. I could model this in in the Go version too. The point is not Pkl. The the point is, that's the schema. Speaker 1: The point is the files. This could be a markdown file, you OpenAI, although you do lose a lot of the determinism there. But currently, what I'm exploring to make things cheaper for myself is property testing. So if any of you guys know anything about that and would love to talk to me about a really nerdy programming thing, you can come talk to me about property testing secure that really excites Meetup. Jun that's where I think this is gonna go. Speaker 0: Or if you've already talked to me. Speaker 1: Or if if Speaker 0: you, you Speaker 1: know, you know me. Three is on GitHub right now. Although, honestly, I think I'm gonna probably make some of this not public, unfortunately, because AI really believe in this tool. Speaker 0: So getting bills. Speaker 1: Yeah. I Real, well, I did lead to find some people who care, but Jun better to be safe to be sorry than sorry. You know? Let's see how it did. It's still running. Speaker 1: How much did you spend, buddy? Speaker 0: Oh, how long does it Route? I I I can go fast. Speaker 1: You can go. Thank you, guys.