Ask a CEO how their AI rollout is going, and 81% will tell you they’re highly confident. Ask the people doing the work, and that number drops to 56%.
Today’s guest sits right in the middle of that gap. Annan Patel is VP of Product at Tricentis, a leader in enterprise software testing, where he builds AI products for companies where one bad release can cost tens of millions of dollars. Before Tricentis, he spent more than 15 years in product across ad tech, Microsoft, and WP Engine.
Tricentis recently surveyed more than 2,500 people across the tech industry. 83% of companies said they trust AI agents to make release decisions, yet 60% admitted they ship untested code. The top reason? Pressure from leadership to move faster.
Annan’s view: AI gets sold as a tool you plug in. In practice, it’s a new operating system for how work gets done.
In this episode, Annan shares:
Why plug-and-play AI only gets you about 60% of the way, and why the rest is context your team has never written down
Why Tricentis’ first AI products stalled after a strong start, and how the team rebuilt them around one principle: the AI is never the one held accountable
Why tokens spent and lines of code shipped tell you nothing about whether your AI initiatives are adding value
1. AI isn’t a new tool. It’s a new operating system.
When AI companies pitch to executives, they speak the language executives already use: ROI, efficiency, cost reduction. This pitch is compelling, and the vision is real. But Annan thinks it misses what makes AI different from every SaaS product that came before it: it changes the work of the humans doing the work.
“It’s not as straightforward as just adopting a new tool. In some ways, it’s adopting a new operating system. It’s a new way of working.”
Teams that have worked together for years build up habits that don’t map neatly onto an agent. Annan says,
“People develop ceremonies, cultures, practices in the way they work that don’t always easily translate to agentic or AI-driven systems.”
That’s where the gap comes from. It’s not that anyone’s acting in bad faith. When you’re leading from 10,000 feet, you just don’t see the dozens of little fixes your team makes every day to keep things on track.
Product takeaway: If your AI rollout plan looks a lot like a plan for buying new software, you’re probably underestimating it. You’re changing how people actually do their jobs, so give yourself time for the messy parts: rethinking processes, helping people adjust, and earning their trust. And when the folks closest to the work tell you something isn’t working, take them seriously. Because they spot the gaps long before you do.
2. The missing 40% is context nobody wrote down
On paper, this sounds easy: write down your process, hand it to the AI, and let it do its thing. But the stuff that really matters on most teams was never written down in the first place.
When you work with a team for a long time, there are shorthands you develop. Annan saw this firsthand with Tricentis’ own products. Customers said the AI was doing good work, but it only got them part of the way there:
“What the AI is doing is good, but it maybe solves for 60% of what I need. Because that remaining 40%…comes from information about my business, my team, my industry, the way that we work, the history we’ve built up over time.”
You can’t fix that by pointing an agent at a repo or a SharePoint folder. That context lives in people’s heads, so the system has to learn it from them.
Product takeaway: Before you scale an AI workflow, sit down with your team and get the unwritten stuff out of their heads. This includes the edge cases, the incidents everyone still remembers, the compliance rules, and all the other habits. If your AI output keeps coming back generic or almost right, it’s probably missing context, not a bad model.
3. Start with one job to be done, and treat it like CapEx
Annan has noticed a pretty clear pattern in which companies make AI work and which ones end up frustrated.
“Where organizations get a bit frustrated is when they try to do too much at once… Because that’s when you start to identify the skeletons in the closet.”
What works better is starting small. Pick one job, figure out how an agent can help speed it up, and then put the time you get back toward something else. Tricentis did this internally with security fixes, which let the team move fast in one area without blowing up its whole development process.
The catch? Those early wins usually won’t match the big ROI numbers executives hear in sales pitches. You only get those by changing your core business processes, and that takes more money, more time, and a lot more care.
“Sometimes you have to slow down to move fast.”
Product takeaway: Keep your first AI projects small enough that they can work without overhauling everything. For the bigger transformation, think of it as a long-term investment. You probably won’t see the full payoff in the first year, but you’re laying groundwork that pays off later. Just make sure leadership knows that going in.
4. Tokens and lines of code are the wrong scoreboard
When teams felt pressure to prove their AI spend was worth it, a lot of them reached for whatever numbers were easiest to count.
“We’re not always measuring the right thing… What you saw a lot of a year ago was the tokenmaxing, the ‘Hey, how many lines of code am I shipping?’”
The problem shows up later, in code that gets rolled back, ships without being tested, or never makes it to production at all. Testing teams are feeling it most. They have fewer people and, in some cases, 10x more code to check.
And as Annan points out, leadership doesn’t really care about those numbers anyway:
“At the end of the day, the executives don’t care how much AI was used, they care, ‘Did I get to the outcome that I needed?’”
Product takeaway: Counting tokens or lines of code tells you how busy everyone is, not whether anything got better. Measure AI against the things your business already cares about, like quality, revenue, and whether customers actually use what you ship. As Annan puts it, building faster makes good product judgment matter more, not less, because you can build the wrong thing really fast.
Chapters
00:00 Introduction
02:17 Annan's path from ad tech to enterprise testing at Tricentis
03:44 Why executives and teams see AI's capabilities differently
09:59 From coding assistants to agents to fully unattended workflows
13:07 AI hype on LinkedIn vs. the real risks enterprises face
18:10 Why tokens and lines of code are the wrong ROI metrics
19:59 How Tricentis adopts AI internally, from security fixes to PM work
25:43 What "think like a CEO" really means for product leaders
30:16 When velocity matters and when quality wins, depending on your market
32:47 Conclusion
Links
What does LogRocket do?
LogRocket’s Galileo AI watches user sessions for you and surfaces the technical and usability issues holding back your web and mobile apps. Understand where your users are struggling by trying it for free at LogRocket.com.