# AI user research for coding agents: user stories checked against your data and specs tested on simulated users, for growing startups in San Francisco, CA, USA.

**Know what your users will do before your agent builds it.**

> OnPuck turns user stories into specs checked against code, analytics and logs. The founders applied this workflow in Woku and shipped changes. We are building a repeatable platform for coding agents, alongside a separate research prototype that tests preview builds with simulated users.

User stories checked against your data, turned into specs your coding agent can ship. Applied in Woku; simulated users are a separate research prototype.

OnPuck light paper, version 1.0, by P. Riquelme and D. Orrego. https://www.onpuck.com/ · Contact: paula@onpuck.com

## How OnPuck works

Building got cheap. Building the wrong thing didn’t. OnPuck finds out what your users really need, and hands your coding agent a spec it can trust.

1. **Plug in your stack.** Read-only access to your repo and data. Revoke it anytime.
2. **We talk to your users.** Short interviews about one real moment, every detail checked against your data.
3. **Your agent ships the right fix.** A spec with the evidence attached, ready for your coding agent.

| Today | With OnPuck |
| --- | --- |
| Guessing from a handful of interviews | Every story checked against your data |
| Vague tickets your agent misreads | Specs with the evidence attached |
| Building on what users say | Building on what they actually did |
| Research eating your team’s week | Run for you, free during the partnership |

Works with the tools you already use: GitHub, PostHog, Sentry, Axiom, CloudWatch, Postgres, MongoDB, Claude Code, Codex, Cursor, Copilot, OpenCode.

## What OnPuck is: AI user research for coding agents

- Interviews explain why.
- Data proves how many.
- Trackers join the two.
- Simulation tests the change before it is built.

**Figure 1. The loop.** The founders ran the evidence workflow in Woku and shipped resulting changes. Simulation is a separate research prototype.

| Step | Output | Status |
| --- | --- | --- |
| 1 Connect | Trackers verified against live data | Applied in Woku |
| 2 Listen | One concrete story per interview | Human-led in Woku |
| 3 Anchor | Each detail confirmed or contradicted | Applied in Woku |
| 4 Spec | A spec with the evidence attached | Applied in Woku |
| 5 Simulate | A predicted change, with an interval | Research prototype |
| 6 Ship | The change in production | Shipped in Woku; MCP planned |
| 7 Calibrate | An accuracy record | By hand in exp-01 to exp-04 |

## §1 Why user interviews and product analytics disagree

Interviews tell you why. Analytics tell you how many. Neither tells you if the story is true.

Across 52,007 people, self-reports about software use rarely matched the logs [32]. So OnPuck checks every **tracker**: a detail your system could have recorded, like a time, a device, a feature or an error.

**Figure 2. One story, resolved.** Synthetic example. One sentence, four trackers, one contradiction, and the contradiction changes the spec.

> "Last night I tried to export the monthly report from my phone, and it never downloaded."

| Tracker | User said | System recorded | Resolution |
| --- | --- | --- | --- |
| Time (anchor) | last night | `export_clicked` at 21:42 | confirmed |
| Device (dimension) | my phone | iOS Safari, 390 x 844 | confirmed |
| Feature (dimension) | export the monthly report | `POST /reports/export` returned 200 | confirmed |
| Outcome (dimension) | it never downloaded | File served, 1.2 MB; never opened | contradicted |

Spec: `fix` Make the exported file findable on iOS Safari. The download worked. The user could not find the file. Evidence: 4 trackers, 3 queries, 1 story.

## §2 How OnPuck turns user stories into specs for your coding agent

Seven steps, each backed by research or by our own experiments.

### Step 1: Connect

Output: Trackers verified against live data. Status: Applied in Woku.

An agent reads your repo, analytics and logs, and keeps only the trackers that show up in live data.

Evidence: Agents solve 21.3% of real enterprise SQL tasks without schema context [23, 24], and can trace errors in logs [25, 26].

### Step 2: Listen

Output: One concrete story per interview. Status: Human-led in Woku.

Our Woku field work used human-led interviews. Automated interviews are specified; existing transcripts can feed the same evidence workflow.

Evidence: AI interviewers match human ones [28, 29, 30]; concrete incidents beat opinions [31, 16].

### Step 3: Anchor

Output: Each detail confirmed or contradicted. Status: Applied in Woku.

Every tracker is resolved against your system. Contradictions are kept: that is where the spec changes.

Evidence: Self-reports rarely match logs [32]; time landmarks improve recall [33].

### Step 4: Spec

Output: A spec with the evidence attached. Status: Applied in Woku.

Each spec carries an action type (`fix`, `build`, `refactor`, `remove`) and the queries behind it.

Evidence: Coding agents fail on underspecified tasks [21, 22, 34].

### Step 5: Simulate

Output: A predicted change, with an interval. Status: Research prototype.

Simulated users calibrated on your sessions run your preview build and predict what changes.

Evidence: LLM simulations predict the direction of experiments [5, 6] and find usability problems [7].

### Step 6: Ship

Output: The change in production. Status: Shipped in Woku; MCP planned.

Woku changes were implemented with agents. Repeatable delivery to your agent over MCP [27] is being built.

### Step 7: Calibrate

Output: An accuracy record. Status: By hand in exp-01 to exp-04.

We compare prediction and reality, then recalibrate. Model versions are pinned [13].

## §3 From customer interviews to shipped changes in Woku

Before digital users, we ran the evidence workflow on Woku and shipped changes from its customer stories.

Woku is the feedback product the founders operate. Its customer interviews were human-led; agents helped trace stories through code and read-only data into specifications.

One investigation catalogued **79 trackers**: **65 marked verified**, 7 stale and 7 unverifiable. Account analyses retained confirmations, contradictions and gaps.

- **Review reliability.** Customers described blank pages and uncertain submissions. Code and error records shaped the specification. The implementation added retries, pending drafts and explicit submission outcomes; its engineering plan records a production rollout.
- **Repeated account warnings.** An interview about repeated inactivity emails was traced to duplicate jobs and account records. Warning and account deduplication were implemented; the batch plan records a production rollout.
- **Reports by period.** Customers assembled reports manually. Checking report scope exposed missing date controls. Period filters and reports were implemented; the batch plan records their production rollout.

This field work demonstrates execution and delivery. It does not establish causal uplift or simulator accuracy. Woku is the founders’ lab, not an independent OnPuck customer.

We are turning that agent-assisted process into a repeatable platform. Human review and source links remain part of the workflow.

## §4 Simulated users that get tired, frustrated and quit

An LLM alone plays a user who is too efficient. We add the part that gets tired.

LLM agents take 2.5 times fewer actions than people [6], know too much [4] and flatten groups [15]. Each simulated user keeps an explicit state: frustration **R**, fatigue **F**, boredom **B**, progress **P** and motivation **M**.

Chance of quitting at each step:

```
hazard = σ(h0 + 3R + 1.5F + 1.5B − 3M − 2P)
```

h0 is fitted on real sessions ([exp-02](https://www.onpuck.com/#exp-02)).

**Figure 3. Simulated run.** Illustration replaying our registered exp-04 prediction for a live A/B on woku. Drawn, not recorded.

| Arm | Registered prediction: first action is writing |
| --- | --- |
| Rating first | 7% |
| Rating after the comment | 20% |

Result pending.

## §5 How accurate are simulated users? Our validation

Four experiments on woku, misses included, and one blind prediction waiting for its result.

| Experiment | Question | n | Predicted | Real | Verdict |
| --- | --- | --- | --- | --- | --- |
| exp-01 | Can a simulated user predict how submissions changed between two past versions of woku? | 560 simulated vs 1,499 real sessions | Δ +0.028 [−0.016, +0.072] | Δ −0.022 [−0.084, +0.040] | Partial. Inside the real interval; the generic prior was not. But "nothing changes" was too, and the model missed real abandonment. |
| exp-02 | Recalibrated on real sessions, does it reproduce abandonment out of sample? | 280 simulated sessions | Abandonment after starting 0.24; Δ +0.061 | Abandonment after starting 0.20; Δ −0.022 [−0.084, +0.040] | Miss on Δ, bug found. Abandonment reproduced; the Δ missed. A simulated user found a real bug: tapping a tab submitted the form. Fixed in production. |
| exp-03 | Same model, better estimator: was the exp-02 miss just noise? | 120 paired simulated sessions | Δ −0.011 [−0.046, +0.025] | Δ −0.022 [−0.084, +0.040] | Hit. Yes. Inside the real interval, 62% narrower, and it excludes the generic prior. It can’t yet beat "nothing changes". |
| exp-04 | Blind, before a live A/B: what changes when the rating moves after the comment? | 540 simulated; up to 600 real sessions per arm | Writing first goes from 7% to 20% (Δ +0.131 [+0.059, +0.202]) | Result pending | Registered, result pending. Registered (git tag and SHA-256) before any real data. It can clearly fail, and we will publish it either way. |

**Figure 4. Prediction vs reality.** Change in review submissions between two versions of woku, on mobile. Bars are 95% intervals. Real data.

| Source | Change in submission rate [95% interval] |
| --- | --- |
| Real change | -0.022 [-0.084, 0.04] |
| exp-01 simulator | 0.028 [-0.016, 0.072] |
| exp-02 simulator | 0.061 [-0.018, 0.147] |
| exp-03 simulator | -0.011 [-0.046, 0.025] |
| Generic latency prior | 0.062 |
| "Nothing changes" | 0 |

Best result so far: a simulated user found a real bug on woku’s review form ([exp-02](https://www.onpuck.com/#exp-02)). It was fixed in production.

## §6 The research behind AI interviews and simulated users

What the research found, and what OnPuck does with it.

- Agents grounded in interviews beat demographic ones [1, 2, 3]. Today: Our prototype uses sessions; interview-grounded digital users remain planned.
- LLM agents act too efficient and know too much [4, 6]. Today: A state layer decides when someone gets tired or quits.
- Simulations get the direction right and the size wrong [5]. Today: We report a direction and an interval, never exact magnitudes.
- Verbalized sampling restores diversity lost to tuning [17, 18, 19]. Today: The model proposes actions with probabilities; code samples them.
- Keystroke-level models predict task time [20]. Today: Time per action is fitted on real sessions.
- World models learn how apps respond [8, 9, 10, 11]. Today: We run your real preview build instead.
- Cognitive models, identity labels and twins distort people [12, 14, 15]. Today: Every prediction is scored against "nothing changes". Misses are published.
- AI interviewers match human ones [28, 29, 30]. Today: Human-led Woku interviews; automated interviewing is specified.
- Agents on real data need its schema, values and code [23, 24, 25, 26]. Today: We checked Woku trackers against live data and resolved existing interview stories.
- Clear specs and good tools make coding agents succeed [21, 22, 34]. Today: Specs carry their evidence, designed to reach your agent over MCP [27].

## §7 What simulated users cannot predict

What we can’t promise, and what we do about it.

### Exact magnitudes

Why not: Simulations overestimate effect sizes [5].

What we do: We give a direction and an interval, scored against "nothing changes".

### Brand-new features

Why not: Hypothetical answers flatter [16]; twins transfer poorly to new behaviour [14].

What we do: We test changes to flows people already use.

### One person’s exact session

Why not: Next-action accuracy is 11 to 21% [35, 36].

What we do: We predict rates across a population, never one person.

### Replacing interviews

Why not: Without real input, simulated people are generic [15].

What we do: Interviews feed every step.

### Stable results across models

Why not: Results shift with prompts and models [13].

What we do: Model versions are pinned and baselines re-run.

### Transfer beyond Woku

Why not: The evidence workflow has been applied in the founders’ product.

What we do: Design partners test repeatability, cost and usefulness in other teams.

## §8 Design partner program for startups in San Francisco

We’re building OnPuck with 5 teams. The founders run it for you while the platform is built.

- Length: 3 months, renewable
- Check-in: Every two weeks
- Cost: Free
- Cohort: 5 teams in San Francisco

What you get:

- Trackers your system can verify
- Interviews with your users, checked against your data
- Specs your coding agent can ship
- Simulated runs before you build
- Every prediction scored against reality

What we ask:

- Read-only access to your repo and one data source
- A check-in every two weeks
- Intros to a few of your users
- Honest feedback

A good fit if:

- You ship weekly, ideally with Claude Code, Codex or Cursor
- You have users in production
- You run PostHog, Sentry, Axiom or CloudWatch
- Seed to Series A, in San Francisco

## Apply to be a design partner or book a call

Apply in five fields, or just book a 30-minute call with the founders.

Apply at https://www.onpuck.com/#apply. The form has five fields: Work email, Company URL, Your role, A recent change that did not do what you expected, Stack. A person reviews and submits it; then they pick a time for a 30-minute intro call. Or write to paula@onpuck.com.

Access and data:

- Read-only, revocable access
- Nothing we run writes to your systems
- We only interview users you introduce

## Questions about access, data, time and cost

What CTOs ask us first.

### What access do you need, and can you write to our systems?

Read-only access to your repo and one data source: PostHog, Sentry, logs or a database replica. You grant it and can revoke it any time. Nothing we run writes to your systems.

### How much of our time does it take?

Two weeks to connect, a check-in every two weeks, and introductions to a few users for 10-minute interviews.

### Is the simulator ready?

It is a research prototype: four experiments on woku, misses published, one blind prediction pending. Every prediction we make for you comes with an interval and gets scored.

### Does this replace talking to our users?

No. Interviews are the input. OnPuck checks each one against your data and turns it into a spec.

### How is this different from AI interview tools or synthetic users?

Interview tools don’t check stories against your data. Synthetic users simulate opinions and publish no accuracy. Analytics shows how many, never why. OnPuck joins the three on your stack.

### What does it cost, and what happens after three months?

Nothing during the partnership. After three months we decide together whether to renew. Either side can stop any time.

### Who runs it?

The founders, P. Riquelme and D. Orrego: two engineers who research and build the models behind OnPuck, and run woku, our lab. Write to paula@onpuck.com.

### What have you already applied in Woku?

Human-led interviews, agent-assisted tracker discovery and story checks produced specifications and shipped changes. The [field work](https://www.onpuck.com/#field-work) is separate from simulator validation. Woku is the founders’ product, not an independent OnPuck customer.

## References

[1] Park et al.. "Generative Agents: Interactive Simulacra of Human Behavior." UIST, 2023. https://arxiv.org/abs/2304.03442

[2] Park et al.. "LLM Agents Grounded in Self-Reports Enable General-Purpose Simulation of Individuals." arXiv preprint (first titled "Generative Agent Simulations of 1,000 People"), 2024. https://arxiv.org/abs/2411.10109

[3] Argyle et al.. "Out of One, Many: Using Language Models to Simulate Human Samples." Political Analysis, 2023. https://arxiv.org/abs/2209.06899

[4] Aher, Arriaga, Kalai. "Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies." ICML, 2023. https://arxiv.org/abs/2208.10264

[5] Ashokkumar, Hewitt, Ghezae, Willer. "Large language models can predict results of social science experiments." Nature, 2026. https://www.gsb.stanford.edu/faculty-research/publications/large-language-models-can-predict-results-social-science-experiments

[6] Lu et al.. "AgentA/B: Automated and Scalable Web A/B Testing with Interactive LLM Agents." arXiv preprint, 2025. https://arxiv.org/abs/2504.09723

[7] Lu et al.. "UXAgent: An LLM Agent-Based Usability Testing Framework for Web Design." CHI Extended Abstracts, 2025. https://arxiv.org/abs/2502.12561

[8] Ha, Schmidhuber. "World Models." arXiv preprint, 2018. https://arxiv.org/abs/1803.10122

[9] Hafner et al.. "Mastering Diverse Domains through World Models (DreamerV3)." Nature, 2025. https://arxiv.org/abs/2301.04104

[10] Chae et al.. "Web Agents with World Models: Learning and Leveraging Environment Dynamics in Web Navigation." ICLR, 2025. https://arxiv.org/abs/2410.13232

[11] Zheng et al.. "Code2World: A GUI World Model via Renderable Code Generation." arXiv preprint, 2026. https://arxiv.org/abs/2602.09856

[12] Binz et al.. "Centaur: a foundation model of human cognition." Nature, 2025. https://arxiv.org/abs/2410.20268

[13] Anthis et al.. "LLM Social Simulations Are a Promising Research Method." ICML, 2025. https://arxiv.org/abs/2504.02234

[14] Peng et al.. "Digital Twins as Funhouse Mirrors: Five Key Distortions." arXiv preprint, 2025. https://arxiv.org/abs/2509.19088

[15] Wang, Morgenstern, Dickerson. "Large language models that replace human participants can harmfully misportray and flatten identity groups." Nature Machine Intelligence, 2025. https://arxiv.org/abs/2402.01908

[16] Nielsen Norman Group. "Synthetic Users." nngroup.com, 2024. https://www.nngroup.com/articles/synthetic-users/

[17] Kirk et al.. "Understanding the Effects of RLHF on LLM Generalisation and Diversity." ICLR, 2024. https://arxiv.org/abs/2310.06452

[18] Hu et al.. "SimBench: Benchmarking the Ability of Large Language Models to Simulate Human Behaviors." ICLR, 2026. https://arxiv.org/abs/2510.17516

[19] Zhang et al.. "Verbalized Sampling: How to Mitigate Mode Collapse and Unlock LLM Diversity." arXiv preprint, 2025. https://arxiv.org/abs/2510.01171

[20] Card, Moran, Newell. "The Keystroke-Level Model for User Performance Time with Interactive Systems." Communications of the ACM 23(7), 1980. https://doi.org/10.1145/358886.358895

[21] Jimenez et al.. "SWE-bench: Can Language Models Resolve Real-World GitHub Issues?." ICLR, 2024. https://arxiv.org/abs/2310.06770

[22] Yang et al.. "SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering." NeurIPS, 2024. https://arxiv.org/abs/2405.15793

[23] Lei et al.. "Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows." ICLR, 2025. https://arxiv.org/abs/2411.07763

[24] Li et al.. "Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs (BIRD)." NeurIPS, 2023. https://arxiv.org/abs/2305.03111

[25] Chen et al.. "Automatic Root Cause Analysis via Large Language Models for Cloud Incidents (RCACopilot)." EuroSys, 2024. https://arxiv.org/abs/2305.15778

[26] Wang et al.. "RCAgent: Cloud Root Cause Analysis by Autonomous Agents with Tool-Augmented Large Language Models." arXiv preprint, 2023. https://arxiv.org/abs/2310.16340

[27] Model Context Protocol. "Model Context Protocol specification." modelcontextprotocol.io, 2025. https://modelcontextprotocol.io

[28] Xiao et al.. "Tell Me About Yourself: Using an AI-Powered Chatbot to Conduct Conversational Surveys with Open-ended Questions." ACM TOCHI, 2020. https://arxiv.org/abs/1905.10700

[29] Wuttke et al.. "AI Conversational Interviewing: Transforming Surveys with LLMs as Adaptive Interviewers." LaTeCH-CLfL, 2025. https://arxiv.org/abs/2410.01824

[30] Geiecke, Jaravel. "Conversations at Scale: Robust AI-led Interviews." CEPR Discussion Paper 19705, 2024. https://papers.ssrn.com/abstract=4974382

[31] Flanagan. "The critical incident technique." Psychological Bulletin 51(4), 327–358, 1954. https://doi.org/10.1037/h0061470

[32] Parry et al.. "A systematic review and meta-analysis of discrepancies between logged and self-reported digital media use." Nature Human Behaviour 5, 1535–1547, 2021. https://doi.org/10.1038/s41562-021-01117-5

[33] Belli et al.. "Event History Calendars and Question List Surveys: A Direct Comparison of Interviewing Methods." Public Opinion Quarterly 65(1), 45–74, 2001. https://doi.org/10.1086/320037

[34] OpenAI. "Introducing SWE-bench Verified." openai.com, 2024. https://openai.com/index/introducing-swe-bench-verified/

[35] Wang et al.. "OPeRA: A Dataset of Observation, Persona, Rationale, and Action for Evaluating LLMs on Human Online Shopping Behavior Simulation." arXiv preprint, 2025. https://arxiv.org/abs/2506.05606

[36] Lu et al.. "Can LLM Agents Simulate Multi-Turn Human Behavior? Evidence from Real Online Customer Behavior Data." arXiv preprint, 2025. https://arxiv.org/abs/2503.20749

## About

We exist to turn stories into evidence, and evidence into better decisions. Founders: Paula Riquelme (co-founder, engineer), Diego Orrego (co-founder, engineer). Lab: woku (https://woku.app/en). Contact: paula@onpuck.com.