M.S. Data Science, University at Buffalo (May 2026) · before that, Solutions Engineer at Flipkart (Walmart) · now fine-tuning and evaluating LLMs at Cardio AI

_{Sources: 38.6% · 30.4% · 17.6% · 17 of 30 · tau2-bench #574 · +401/−18 across 9 files}

• 🔭 Now: fine-tuning and evaluating LLMs at Cardio AI
• 🎯 Looking for: AI Engineer, Forward Deployed Engineer and ML Engineer roles in the US
• 🧪 How I work: lock the ruler before training, run the control before the claim, publish the null result
• 💬 Ask me about: post-training (SFT, DPO, GRPO), agent evaluation, RAG retrieval, LLM integrations
• ⚡ Fun fact: a random-reward control showed my own RL gain was noise, and it's my favourite result

• A fine-tuned 1.5B model scores 38.6% pass@1 on HumanEvalFix (3-seed mean; worst seed 34.5%), above OctoCoder's 30.4%, a 15.5B model, under the paper's own frozen protocol.
• The data won. $0.34 of API calls produced 2,551 certified training bugs; each had to compile and fail a real test.
• The RL didn't. A GRPO twin trained on a random reward matched the real one: real minus random came to +0.61 pass@1.
• Scaffold or fine-tune? In a verify-and-repair loop the untrained base reaches 65.2%, and training adds only +2.5 points.

Repo · Case study · Lab notebook · Every result

flowchart TB
    subgraph plan [" "]
        direction LR
        A["Frozen ruler<br/>OctoPack protocol,<br/>committed before training"] --> B["Data factory<br/>an LLM breaks clean functions;<br/>each bug must compile<br/>and fail a real test"] --> C["Three arms<br/>SFT · DPO · GRPO<br/>matched compute, paired seeds"]
    end
    subgraph test [" "]
        direction LR
        D["Controls<br/>random-reward GRPO,<br/>three seeds per arm"] --> E["Held-out exam<br/>HumanEvalFix, touched<br/>only at milestones"] --> F["Scaffold test<br/>sample, run tests, repair;<br/>base vs trained"]
    end
    plan --> test
    style plan fill:none,stroke:none
    style test fill:none,stroke:none

• Found a leak in the benchmark itself. In 17 of 30 dev tasks, one listing tool exposed answer-key data to the agent. I reported it as tau2-bench #574, with a reproducer that needs no model calls.
• Measured before claiming. 13 measurement and harness defects found and fixed, each with a regression test. False "transfer under way" replies fell from 26 of 27 → 2 of 27 in replay.
• Reported the null. Harness vs standard agent: 3 vs 3 of 30 tasks completed. No gain shown, and the report says so.
• Try it: make demo-provenance reproduces a verification bug and its fix, offline and at no cost.

Repo · Case study · Technical report · Every result

flowchart TB
    subgraph B["tau2-bench, pinned and sandboxed"]
        U["Simulated customer"] <--> AG["Agent under test"]
        AG --> TL["Bank tools + database"]
    end
    S["Spend admission control<br/>reserve, refuse, settle"] -. gates every model call .-> AG
    H["Runtime harness<br/>tool adapter, checks before actions,<br/>verification evidence, disclosure check"] -. wraps .-> AG
    TL --> G["Official grader<br/>final database state"]
    G --> E["Evaluation layer<br/>frozen decision rules, paired batches,<br/>replay probes, answer-independence check"]
    E --> R["Review<br/>blind readers + adjudicator"]
    E -. found .-> L["Benchmark leak,<br/>reported upstream"]

Code-review LoRA · fine-tuning Reasoning-trace SFT on Qwen2.5-Coder-7B that wins 86.0% of head-to-head reviews against the base model (CI [81.5%, 91%]) and cuts hallucinations 14.5% → 7.0%. Three rounds of RL on top didn't beat it, and the repo says so. Repo · Results	ProofCart · agents and integrations Hackathon agent across Slack, Notion and Stripe (test mode): reads supplier threads, waits for the owner's approval in Slack, pays, records. Exactly one charge under crash or duplicate (2/2 vs 0/2 against an ablation); 14/14 deterministic regression cases. Repo · Demo video
Sokratic tutor · group project, my parts Multi-agent Socratic tutor (UB). I built the first RAG pipeline (MRR 0.493 → 0.616), K-of-N partial reach in the reach gate, Haiku behaviour classifiers (29/30 on a check set) and an end-to-end regression harness. Reach rate: 38.9% → 66.1% across 59 conversations. My commits · My branch	Open source · merged pull requests LangChain: OpenAI's apply_patch built-in tool for langchain-openai, with unit tests (+401/−18 across 9 files, merged 9 Jun 2026). Apache Airflow: a dependency fix in the AWS provider (+1/−1 across 1 file, merged 1 Apr 2026).

LLMs and post-training

Agents, RAG and evals

Backend and infrastructure

Data and ML

Integrations

Buffalo, NY (Eastern) · open to US-remote and to relocating to the SF Bay Area, NYC or Seattle.
US work authorization: F-1 OPT, eligible for the 24-month STEM extension (E-Verify employer). I'll need H-1B sponsorship long-term.

_{Every number on this page links to the line it comes from, in a public repo at a pinned commit. This README is generated from the same checked facts as nidhi1603.github.io.}