A multi-agent AI system that orchestrates vision and natural language processing agents, enabling context-aware input routing and structured output validation for personalized recipe discovery.
Upload a photo of your fridge and it will identify your ingredients, intelligently format them, and find recipes that match what you already have.
The core design of Sous is its multi-agent orchestration system built with Pydantic AI. Rather than a monolithic AI attempting to handle everything, I implemented specialized agents that each excel at specific tasks.
The orchestrator agent serves as the systems brain, implementing intelligent intent classification to route requests to the appropriate specialized agent. It uses LLM-based classification with a keyword-based fallback for robustness, determining whether a request involves image analysis, recipe search, or general cooking questions.
The Fridge Agent handles the complete computer vision workflow. Using Google's Vision AI, it analyzes uploaded fridge images, extracts visible ingredients with validation, and manages the entire pipeline from image processing to recipe retrieval. Each step streams updates to provide real-time feedback to users.
This agent specializes in natural language query processing. It extracts structured parameters from user queries like "quick pasta recipes under 30 minutes" and maps them to the Spoonacular API's search parameters, handling complex queries with dietary restrictions, cooking times, and specific cuisines.
A general-purpose agent for cooking questions that don't fit into the other categories. It provides context-aware answers to questions about techniques, substitutions, and general culinary knowledge.
One of the key technical challenges was managing state across multiple asynchronous operations. I implemented a dependency injection pattern using a "Deps" dataclass that maintains workflow state:
Progressive State Accumulation
Each agent step builds on the previous one's results, stored in the shared dependency context. This pattern enables clean separation of concerns, testable components, and error recovery at any workflow stage. If a step fails, the system can gracefully handle it without losing progress from earlier stages.
Dependency Injection Benefits
By using dependency injection, services are instantiated lazily and reused throughout the workflow. API clients are created once and maintained for the request lifecycle, improving performance and reducing overhead. The pattern also makes testing significantly easier by allowing mock dependencies.
I think 35% of my time spent on this project was trying to make this bento grid look cool lol