What Is Your Coding Agent Actually Searching? | Columbus .
Cities (231) ▾

Members-Only

Recent Talks & Demos are for members only

Exclusive feed

You must be an AI Tinkerers active member to view these talks and demos.

Sign in View FAQ
June 01, 2026 · Columbus

What Is Your Coding Agent Actually Searching?

Learn how a new harness evaluates coding agent search behavior, comparing strategies and revealing factors like anchor quality and lookahead for better bug localization.

Python tree-sitter bash git grep Codex
Overview

I built SearchBench, a harness for running controlled evaluation rounds over coding-agent search behavior.

A SearchBench round takes real bug-localization tasks, gives an agent access to a repository, and checks whether it found the files that were actually changed in the human fix. The harness compares an incumbent search strategy against one or more challengers, then writes a static evidence bundle with exact-hit, hop-distance, token-usage, failure, and report artifacts.

For the demo, I’ll run a small live ablation round over three cases. I’ll reuse cached Bash/native-search results as the incumbent, then run a few IC challenger policies in parallel. The question is: when IC succeeds or fails, was the important factor anchor quality or graph lookahead?

I’ll show the actual workflow: preflight cost prediction, run planning, parallel execution, actual spend, cost-prediction error, generated bundle artifacts, and a short report explaining what changed. One thing I want to show explicitly is how much information the harness can derive from a small, bounded run: exact hits, hop distance, token usage, failure modes, and the next optimization target.

Links
Tech stack
  • Python
    Python: The high-level, general-purpose language built for readability, powering everything from web backends to advanced machine learning models.
    Python is the high-level, general-purpose language prioritizing clear, readable syntax (via significant indentation), ensuring rapid development for any team . Its ecosystem is massive: use it for robust web development with frameworks like Django and Flask, or leverage its power in data science with libraries such as Pandas and NumPy . The Python Package Index (PyPI) provides thousands of community-contributed modules, offering immediate solutions for tasks from network programming to GUI creation . The language is actively maintained by the Python Software Foundation (PSF), with the stable release currently at Python 3.14.0 (as of November 2025) .
  • tree-sitter
    An incremental parsing library that builds concrete syntax trees and updates them efficiently during live editing.
    Tree-sitter generates robust syntax trees using a C-based runtime and language-specific grammars. It supports over 40 languages (including Rust, JavaScript, and Python) to power features like syntax highlighting and code navigation in Neovim and GitHub. The engine handles syntax errors gracefully and re-parses modified code in O(log n) time. This efficiency ensures immediate feedback without blocking the main editor thread.
  • bash
    Bash (Bourne-Again SHell): The ubiquitous command language interpreter and scripting engine for GNU/Linux and macOS systems.
    Bash is the GNU Project's powerful shell, serving as the default command interpreter for most Linux distributions and a widely used option on macOS. Developed by Brian Fox in 1989, its name is an acronym for 'Bourne-Again SHell,' signifying its compatibility with the original Bourne Shell (sh) while incorporating advanced features from shells like ksh and csh. The technology provides a robust environment for interactive command execution and non-interactive shell scripting, supporting critical functions like command-line editing, job control, and complex programming constructs (loops, conditionals, functions) for system automation.
  • git grep
    A lightning-fast command-line utility built directly into Git to search tracked files, the index, and historical commit trees for matching patterns.
    When you need to find a specific string or regex pattern across your codebase, skip the slow filesystem searches and rely on git grep. Because it targets only files tracked by Git (or specific commits and branches), it bypasses node_modules and other untracked noise automatically. You can pinpoint patterns with flags like -n for line numbers, -p to show the containing function, or target historical snapshots by passing a commit hash (such as git grep 'TODO' HEAD~5). It is a highly optimized, native alternative to standard grep that respects your repository structure out of the box.
  • Codex
    Codex is OpenAI's autonomous AI software engineering agent: it executes full development tasks in a sandboxed cloud environment.
    Codex is the advanced, cloud-based software engineering agent from OpenAI, built on a specialized model like `codex-1` (a fine-tuned version of `o3`). It operates on an asynchronous delegation model, allowing developers to assign complete tasks—not just receive suggestions—via the ChatGPT interface. The agent works independently in a secure, isolated cloud container provisioned with the user's GitHub repository and environment. It reads code, writes new features, fixes bugs, runs tests, and drafts Pull Requests (PRs) for review, significantly accelerating the development lifecycle. Access is provided through ChatGPT Plus, Pro, and Enterprise plans.
Events Gallery FAQ Sponsor