AI Agents Fail for 2 Reasons. Crowdsourcing Solved Both. | Julia Dalton, SVP Product (Capacity)
Capacity's Head of Product explains how a decade of managing thousands of crowdsourced workers gave her the playbook most teams are still missing for building AI agents that actually work.
Our guest today is Julia Dalton, the SVP of Product at Capacity, an AI-powered support automation platform. Before that, we spent years at OneSpace, formerly known as Crowdsource, a crowdsourcing company where thousands of freelancers executed microtasks for major retailers. Routing rules, task chains, instruction validation, and more.
Today, that’s known as multi-agent orchestration. And Julia was doing it before it was cool.
In today’s episode, Julia shares:
How a PRP (product request prioritization) system she designed herself in one weekend transformed Capacity’s CS feedback by replacing the chaotic “firehose” of requests with a ranked, data-backed list
What running a human API layer taught her about prompt design, long before LLMs existed
And the truth most teams skip — that AI agents are only as good as their instructions. AI doesn’t fix bad data; it amplifies it, and teams need to audit their data before writing a single prompt
1. The two reasons AI agents fail (22:45)
At OneSpace, Julia’s team managed thousands of freelancers doing microtasks for large retailers at scale. The lessons learned were hard and expensive: if you send out 500 product descriptions with unclear instructions, you’ve paid for 500 things you can’t use.
“You could have the best instructions on the planet, the best prompt, but if your data is wrong, you’re going to get really, really terrible results.”
The two culprits? Bad instructions and bad data.
The product takeaway: You’re not the one doing the task — you’re architecting it. That distinction changes everything about how you design agent workflows.
2. Validate your prompts before scaling (11:39)
One of the most underrated moves at OneSpace: before deploying a task to thousands of workers, they’d run a separate mini-workflow with workers whose only job was to evaluate the instructions — not execute them.
Why?
Julia says:
“What seems clear to you and what you’ve communicated is oftentimes very unclear or not as clear as you thought to the audience or to the recipients.”
Her fix? Use a separate agent (or person) whose only job is to evaluate the instructions — not execute them.
Julia does the same thing now with agents: agent-to-agent evaluation runs, logging and scoring conversations, and humans doing test passes. Recursive validation before you ever go live.
The product takeaway: What seems clear to you is often unclear to your recipient, so make sure to build a feedback mechanism for your instructions before you scale them.
3. The PRP: A weekend project that untangled the feature request firehose (22:30)
Julia’s product team was drowning in requests from CS and revenue teams. Each submitter was convinced their ask was the #1 priority.
So, she built a structured intake system herself over a single weekend.
The result?
Structured inputs, auto-classification, ARR and retention impact weighting, and a triage layer within Customer Success before anything ever reached Product. The same signals that prioritize incoming work also let the team communicate the ROI of what they shipped.
“AI only amplifies the data — so if your data is wrong, it’s going to amplify its wrongness in a major way.”
The product takeaway: Data doesn’t just help you prioritize what to build — it helps you prove the impact of what you’ve already built.
Links
Chapters
00:00 Introduction
02:10 Julia's career path to Capacity
04:18 Microtasking at scale
05:59 Jula explains her workflow chains
08:36 Designing routing rules
13:15 Two failure modes
13:56 Simulating and scoring agents
17:22 Recursive prompting in practice
20:27 Data and knowledge orchestration
23:55 PRP Feedback triage system
27:02 Impact and ROI from signals
30:15 Conclusion
What does LogRocket do?
LogRocket’s Galileo AI watches user sessions for you and surfaces the technical and usability issues holding back your web and mobile apps. Understand where your users are struggling by trying it for free at LogRocket.com.