Ajeenckya Mahadik

The problem

Football outcomes are low-count, high-variance events: a 3-way classification (home / draw / away) where even the best team loses often. The interesting challenge isn't predicting a winner — it's producing an honest probability, one you could bet on, plan inventory around, or price risk with. That's why the whole system is built around calibration rather than raw accuracy.

Features

Each fixture is described by engineered team-pair features: rolling form and attack/defense rates with time-decay weighting (the same idea as EWMA in volatility modeling), ranking deltas, rest and travel context, and host advantage. On top of that sit two live signals: an NLP pipeline that pulls RSS news, scores sentiment per team and per key player, and folds it into the features — and a weather block (temperature, humidity, precipitation, wind, cloud cover, pressure) for the actual stadium and kickoff window, which you can toggle in the Match center to see its real effect per fixture.

Architecture — three models, one ensemble

SNN: a PyTorch policy-gradient neural network over the engineered features, trained with a policy-gradient objective and emitting a 3-way softmax. It's the fast, deterministic baseline — one forward pass, one answer. BNN: the same backbone extended with a Bayesian output head. Instead of fixed weights, the head learns weight distributions; at inference I draw hundreds of Monte Carlo weight samples, each producing a slightly different network and prediction. The spread of those predictions decomposes into total uncertainty (predictive entropy), epistemic uncertainty (mutual information — what the model doesn't know because it hasn't seen enough data) and the win-probability standard deviation you can watch being sampled in the "Inside the model" tab. XGBoost weather sub-model: a gradient-boosted tree model that learns how weather shifts outcomes, feeding the ensemble. The final calibrated ensemble blends SNN and BNN with the weather signal and applies temperature scaling so that "70%" means 70%. On top sits a Poisson scoreline layer: it inverts the calibrated win/draw/loss probabilities into goal rates (λ_home, λ_away) and reports the most likely scoreline consistent with the pick — score predictions are a pure function of the same probabilities, not a separate guess.

Training & validation

Validated with 5-fold cross-validation and historical backtests; every variant comparison (e.g. weather on vs off) goes through McNemar tests in an A/B framework, so a feature only survives if it helps beyond noise. Calibration diagnostics — reliability diagrams, expected calibration error, Brier decomposition — plus experiment metadata are logged to PostgreSQL. Results: Brier score 0.24 → 0.176 (−26.7% vs baseline), recall 84%, scenario variance −17%.

The Monte Carlo tournament layer

Match probabilities become title odds by simulation: sample all 72 group fixtures from the ensemble's distributions, resolve standings (including the 8 best third-placed teams in the 48-team format), play the bracket to the final, repeat 5,000 times, and count champions. Uncertainty propagates from single matches all the way to the trophy — exactly what the simulator tab does in your browser with the same numbers.

Serving & feedback

The real system runs behind FastAPI: REST endpoints for predictions, team data and result uploads, a post-match feedback loop that updates the model as results land, and the news pipeline refreshing sentiment features continuously. It's deployable software, not a notebook.

Why it generalizes

Calibrated low-count forecasting is the same problem as demand planning for slow-moving inventory, credit-risk PDs, and insurance claim frequency. The football is the demo; the method is the product.

Zero-framework by design

CodeCraft talks to the LLM with raw HTTPS calls to an OpenAI-compatible chat-completions endpoint (xAI by default) using only the Python standard library — no LangChain, no CrewAI, no AutoGen. The point isn't framework allergy; it's that an agent is fundamentally a while loop around an API, and owning that loop means owning latency, cost, and failure modes.

How the loop works

The model receives the conversation plus JSON schemas for the local tools (native function calling). Each turn it either answers or returns tool_calls; the agent executes them in the workspace, appends the observations, and calls the model again — ReAct's think → act → observe cycle until the task is done. Session memory keeps the project map and prior decisions across requests, so follow-ups skip re-discovery.

The tool set

Eleven workspace-scoped tools: project_context, list_files, read_file, search_files, write_file, replace_in_file, make_directory, git_status, git_diff, run_tests, and run_command. Every file write shows a diff preview and every shell command requires approval by default (with an explicit --auto-approve escape hatch) — guardrails are part of the architecture, not an afterthought. A doctor command diagnoses provider/tool-call issues.

RLAIF scoring

Completed trajectories are scored by Grok 4 acting as an AI judge — task success, efficiency, and safety of the action sequence — giving a reward signal used to refine the agent's prompting policy over time (reinforcement learning from AI feedback, without human labeling).

The benchmark

15 identical tool-calling coding tasks, same model, same tools, run through CodeCraft and through a LangChain implementation. Result: 15/15 completed, 7.5× lower latency. The gap is pure orchestration overhead — chain abstractions, callback layers, and serialization the raw loop simply doesn't have. Packaged with Docker for reproducibility.

The hypothesis

Failure memory compounds. Inspired by SELF-REFINE (Madaan et al., 2023): if an agent analyzes why it failed and stores the lesson, retrieval of those lessons should make every future attempt smarter — improvement without touching the model weights.

Three agents, one benchmark

The system implements three strategies for controlled comparison: ReAct (think-act-observe baseline), Plan-and-Act (upfront planning baseline), and Strategy-Guided — the contribution, which augments planning with retrieved strategies from memory.

The learning loop

Every execution is traced step by step. When a task fails, a failure-analysis pass (heuristic rules plus LLM reasoning over the trace) produces a corrective strategy — a short, generalizable instruction like "sort by total price including fees before selecting." The strategy is embedded and stored in ChromaDB. On a new task, the agent retrieves the top-k semantically similar strategies and injects them into its planning prompt. The demo above is this exact loop: fail → analyze → store → retrieve → pass.

Distillation to a 1B model

To make the agent cheap to run, Grok 4 teacher trajectories are distilled into LLaMA-3.2-1B via QLoRA — 4-bit NF4 quantization with low-rank adapters — so a 1-billion-parameter model inherits behavior from a frontier teacher at a fraction of the inference cost.

Evaluation

94% task completion on Tau Bench, with additional OS-level and web-navigation task suites and an ablation study isolating each component's contribution (memory off vs on, heuristic vs LLM failure analysis). The success curve in the demo is illustrative in shape; the endpoint is the measured result.

Why a graph at all

Flat vector RAG retrieves chunks that look like the query. But "who approved the invoice Sarah flagged?" needs facts from documents that never mention each other in the same paragraph. A knowledge graph stores entities (people, vendors, documents, projects) and their relations explicitly, so retrieval can walk from fact to fact — multi-hop reasoning vector similarity structurally cannot do.

Ingestion & graph construction

10K emails are parsed (.eml, extendable to IMAP), chunked, and passed through LightRAG, which extracts entities and relations to build the knowledge graph alongside the text index. Every chunk is embedded with intfloat/e5-small-v2 (384 dimensions — deliberately CPU-friendly) and stored in Postgres 16 + pgvector with cosine-distance indexing.

Hybrid retrieval

A query first hits pgvector for candidate entry points, then expands along graph edges to pull in connected entities and their supporting chunks — vector search for recall, graph traversal for reasoning. The assembled context goes to a fully local Mistral 7B Instruct (GGUF via llama.cpp), which answers with referenceable /emails/{id} source links. Nothing leaves the machine — by design, since email is about as private as data gets.

Multi-user isolation

Authentication is JWT-based, and isolation is enforced where it can't be bypassed: every retrieval query is filtered by user_id at the SQL layer. One database, many users, zero cross-user leakage.

Performance

E5 embedding caching cut end-to-end retrieval latency from 1.8s to 1.0s (−44%). The FastAPI backend ships with seed scripts, a benchmark harness, and an evaluation report comparing graph-RAG against flat vector baselines on multi-hop questions.