The Problem
This project started as a CodePath team exercise: an intentionally broken Streamlit number-guessing game with seeded bugs — a secret number that reset on every rerun, higher/lower hints that pointed the wrong way, and a difficulty selector that changed its on-screen label but not the actual number range the game played against. The assignment was to practice AI-native debugging: use an AI tool as a reasoning partner, then verify every fix with pytest instead of trusting the diff.
That phase ended when the game was fixed. The repository kept going. The same codebase became Applied AI System Project, and the number-guessing game was replaced entirely with a real agentic system: paste in any buggy Python snippet, and a 4-step AI agent — analyze, identify, fix, verify — investigates it using the Google Gemini API instead of hardcoded game rules.
What I Built
Project Overview
Applied AI Code Investigator began as a three-person CodePath team exercise (with Andrew Burke and Jamila Cocchiola) to fix seeded bugs in a number-guessing game starter. Every commit from March 2026 onward — the entire agentic debugging system this case study describes — was written solely by me.
My Contribution
- Designed and built the 4-step agentic loop in agent.py — analyze, identify, fix, verify — with a dedicated system prompt per step and each step's output feeding the next.
- Wrote guardrails.py for input validation and structured API-call/error logging, and logic_utils.py for parsing the agent's free-text responses into structured data.
- Wrote 23 mocked pytest tests, generated the architecture diagram, and authored the README's Design Decisions, Testing Summary, and Reflection and Ethics sections.
Four-step agentic debugging loop
User code passes through guardrails before a four-step Gemini-powered agent analyzes, identifies, fixes, and verifies it, then returns a structured report.
Each step sends its own system prompt to the Google Gemini API and receives the previous step's output as context — a sequential loop, not a single prompt.
Sample Interaction
Investigation report sample
A recreated view of the report the app produces, built from one of the README's real documented sample inputs.
4 agent steps
23/23 tests passed
gemini-2.5-flash
Visual uses safe sample data created for this portfolio page.
Technical Decisions
Split the investigation into four sequential Gemini calls instead of one large prompt.
- A single prompt asking a model to simultaneously understand, find, fix, and verify bugs tends to produce shallow answers because each task competes for the model's attention.
- Four calls per investigation instead of one, which costs latency and quota.
- Each step gets a focused system prompt and the previous step's output as context, which measurably improved the quality of the bug reports and fixes.
Validate input locally in guardrails.py before making any Gemini API call.
- Rejecting empty, too-short, or oversized input before the network call keeps API usage low and prevents wasting a response on unusable input.
- The 200-line cap excludes larger real-world files — an intentional scope limit for a demo tool, not a production constraint.
- Bad input never reaches the model, and every accepted investigation is logged with its step, status, and token counts.
How I Work
Engineering Challenge
- An AI-suggested refactor moved difficulty-range logic into logic_utils.py as a clean separation of concerns, and the diff looked correct — the function moved cleanly, the import was added, and the app ran without errors.
- Reading the diff alone gave no signal that anything was wrong; the bug only surfaced when the existing test suite was run against the refactored code.
- Pytest failed with a direct message that Hard mode's range (1, 50) was narrower than Normal's (1, 100) — a bug the refactor had silently preserved instead of fixing.
- AI-assisted refactors need the same test coverage as manual ones. The model optimizes for code that runs, not code that is correct, and only the test suite caught the gap that code review missed.
Quality & Testing
Quality Evidence
Product engineering evidence
- 23 of 23 pytest tests pass in tests/test_agent.py: 6 input-validation edge cases, 3 response-parsing formats, 5 bug-counting formats, 5 verdict-parsing cases, and 4 full-loop tests covering return shape, value types, exact call count, and error propagation.
- No hosted CI is published for this project; the suite runs locally via `pytest tests/ -v` in under 3 seconds with all Gemini calls mocked, so no API key is required.
- Verified scope covers guardrails, response-parsing helpers, and the full 4-step agentic loop's structure — not live prompt-quality regressions, which the README notes require manual end-to-end review with a real key.
Public page quality
- The architecture diagram and sample report use text alternatives and captions consistent with the rest of the case-study template.
- The diagram scrolls horizontally on small screens, matching the other case studies' architecture visuals.
- Every claim on this page traces to the repository's README (Design Decisions, Testing Summary, Reflection and Ethics) or its committed source files.
Results & Impact
The honest result claim is implementation completion, not production usage: a public repository documents a working 4-step agentic debugging loop built on the Google Gemini API, with input guardrails, structured logging, 23 passing pytest tests, and a generated architecture diagram — evolved from, and a direct sequel to, a team debugging exercise rather than a from-scratch rebuild.
What I Learned
Breaking a hard problem — debug this code — into ordered sub-problems that mirror the scientific method (observe, hypothesize, test, confirm) made each failure mode easier to isolate, for both the model and me. Testing AI-powered code is also fundamentally different from testing pure functions: mocks prove the loop's structure runs correctly, but they cannot prove the prompts produce useful output, and the project's own reflection work surfaced a real limit on that front — bug-report accuracy was highly sensitive to variable naming, with descriptive names producing far more accurate reports than single-letter ones. The biggest takeaway carried over from both phases of this project: AI is most useful with a well-defined role, a specific task, and the prior step's reasoning as context — and its suggestions still need the same tests a human's would.