Hi, I'm Sarthak 👋🏻
I'm a career-long software engineer and a lifelong problem-solving addict, currently working as an Applied AI Engineer. I like to poke at the known boundaries of computer science and software engineering — dissecting my own curiosities and, where I can, pushing them a little further.
As I build recursively on my own work, I want this blog to be more than an archive of findings. I want it to be:
- a medium for discussion and collaboration,
- a channel for contributing open research,
- and a fun way to keep learning and growing.
What each post answers
To serve that spirit, every post tries to answer the same handful of questions:
- What motivated this?
- why it was worth the time
- What did I do?
- the approach, honestly
- What did I measure?
- the concrete results and artifacts
- What did I infer?
- what I think it means
- What am I uncertain about?
- where I could still be wrong
- Where do I go from here?
- the next thread to pull
Happy reading!
Writing
See all posts
Don't judge an agent by its pass@1
A single pass@1 score collapses runs that differ on three axes it can't show: how a failure was shaped, whether a success repeats, and what either one cost. One model concluded a secret-safety task in 4 turns; two others avoided the leak but never concluded, burning ~77× the tokens. Then the reliability metric I reached for turned out to move in coarse sixths, and inverted when I reran the identical matrix. What to log, and how far to trust it.
Is your harness driving your model crazy?
One model went from a 0.10 to a 0.75 success rate without anyone touching the model. The fault was a tool-schema bug in the harness, and the only reason I caught it was reading the agent's trajectory instead of the scoreboard. A case study on why agent evals score the whole system, and how to tell which part you're actually measuring.
When is an agent truly done?
Many agent loops still treat a model's completion message as terminal, but stopping isn't the same as being correct. On moving the authority for 'done' out of the model and into a verifier, and what that one move makes measurable.
Get in touch
Reach out if you're poking at the edges of software, systems, or AI, or just want to compare notes.