Leader Spotlight: Truth seekers don't confirm — they discover, with Jason Giles
Jason Giles is VP of Customer Intelligence at UserTesting, where he connects customer insight, market intelligence, and product strategy to help shape innovation and growth. Over more than two decades, he has held executive leadership roles in design, product development, research, and customer intelligence across Microsoft, AT&T, DIRECTV, and Ticketmaster. He is based in Barcelona, Spain.
In this conversation, Jason examines what it really means to democratize customer research — and where democratization goes wrong. He makes the case that access to tools is only the beginning: the harder work is building the rigor, the frameworks, and the truth-seeking mindset that turn cheap insights into good decisions. Along the way, he draws a sharp line between AI synthesis and the fidelity of a real human face, explains why digital twins are best understood as a rearview mirror, and names the single worst thing product teams do with research: run it after the decision has already been made.
The democratization gap
Has customer research genuinely become democratized, or have we simply democratized access to tools while leaving interpretation as the real bottleneck?
We’ve been democratizing research for a while — it’s been a key part of something I’ve been involved in for probably 15 years. What’s happened is that AI is accelerating it, reducing the friction and access to some of these tools. And one of the dangers — you touched on this with interpretation as the bottleneck — is that, like anything else, when you unleash all these tools, the maturity applies in when do you apply the tools, and how do you do it in a structured, scalable way that ensures you’re not over-deluged with a bunch of inactionable content. Traditionally, companies that have research teams are gravitating more and more to: “How do we build environments so that, as we make these tools more accessible, there’s a little bit more sense of control and quality, making sure that businesses don’t make bad decisions?”
The teams that are doing this well are going about it in a more structured way. They’ve built frameworks for how to do it right, and now what they’re doing is building those frameworks into the process — setting either gates or recommendations, or building their own internal tools that say, “For this type of question, here’s the right type of methodology to use. Here is something that is great for you to do by yourself, and here are some templates and guidelines. This is actually something where you’d want to partner, because maybe it’s more complex or it has higher risk.”
Isn’t it also important for someone using these tools to be able to validate that what they’re seeing is accurate?
That’s exactly it. Some of the best tools ensure that you have visibility into the actual source. My team is AI-enabled, but they’ve developed practices to ensure they know how to look for hallucinations — that when they see something that doesn’t look right, they drill in and get an understanding of the core signal, that it’s actually true. I love the fact that we’re reducing the friction. Now it’s just about developing the skillset to ensure we have the rigor to apply them properly. And depending on the maturity of your organization, that rigor is a sliding scale.
Research literacy: What product teams really need
As more customer research moves toward product teams, is there a minimum research literacy you look for — or are the frameworks that companies are building often clear enough that someone with a solid product background can jump in and be effective?
A lot of it is common sense. One of the frameworks a team will use is: when should I do research? That’s not rocket science. On a spectrum of low risk versus high risk — if it’s low risk, those are things that maybe you don’t need deep research or any research at all.
The other practice is taking a step back — and this is where researchers are really good. These teams will start building stuff and a researcher comes in and is very quickly able to identify: what assumptions are we making around the users? That practice really helps reinforce opportunities to go get more confidence. Let’s actually close out this assumption — whether it’s about user behavior, attitudes, or anything. That helps define what we should get feedback on.
The next question is: what’s the right method? This is where AI can help. Even on general-purpose AI, you say, “Here are the questions I’m wanting to ask,” and it’ll say, “Oh, for this type of thing, you’d want to do a survey.” Matching the question you have to the right methodology is going to be important.
But if you’re at high risk — let’s say you’re trying to think through a pricing strategy, something that is quite complex — that’s where you’re going to want professional help. High-level concept validation, usability, that type of validation: having early conversations with customers around maybe product-market fit or concept feedback, if done well, anybody can do that. It always helps if you’ve got a researcher to guide you along the way. But the tools themselves are getting better and providing that assistance, and even just general-purpose AI will give you at least the best practices around how to approach doing it yourself.
The confirmation bias trap
One of the risks of these tools is that they may make getting insights seem so easy. What are some of the common ways that product managers might accidentally convince themselves they’re being customer-centric when really they’re just confirming what they already believe?
That is not product-manager specific. I know this as a designer.
What you love about a researcher is that their whole job is to check your thinking. They’ve got this critical mentality — they’re truth seekers. Whether I have a PM or a designer who wants to get feedback on something or do their own research, the key thing I’m advising them is: you need to realize that you’re putting on a different hat. By day, maybe you’re designing concepts or figuring out a PRD. But when you go into the activity of research, be really clear about what your role is — because if you go in with that mindset, it helps with some of that internal bias. We all just want to know that our idea’s great. We all hope our prototype is super usable and delightful. We have a vested interest. You just have to acknowledge that upfront.
Don’t confuse the activity with the rigor. I do the activities, but when I test, I need to make sure that — I’ve been working on this, I’ve got a vested interest in this, I’m excited about this concept — I’m listening, I’m being very thoughtful. The function of research is to find truth. It’s a little bit of a different shift.
Human moderators vs. AI: Where the fidelity matters
Are there certain parts of research that should be left to humans versus AI? Where do you really need human judgment?
Obviously anything sensitive — if you’re in healthcare, if it’s emotional or relationship-based, forget the tools. And if it’s something where you want genuine reaction, it’s a question of fidelity. Today we do unmoderated testing — you write a little script, it answers all the questions.
There was a case a few months ago — just a validation test of usability. My instinct was that there was going to be some friction in this flow. When I got back the results, just from the AI synthesis, it wasn’t flagged. But then when I went back and watched the video, I saw this scrunched-up face when this person was trying to complete the task. In multiple cases, there was just this pause — they figured it out and moved on. The system didn’t track it. That’s the fidelity I’m talking about. If you want overall signal, if you need numbers from 200 people, that’s awesome. But when fidelity matters, when some of the subtlety matters, that’s going to be pretty important.
Do you have an example of a time where results seemed plausible but turned out to be directionally very wrong?
The mistakes I see most often are in the framing of the tests. We were talking to people around usability and features and functionality — less around: is this really even solving a problem that you have? How would you actually use it? You’re focusing on the wrong thing. You skip the step of: are we actually solving a problem? I see that frequently because I review a lot of products in flight and I’m like, “So what’s the problem this is solving for them? Did you validate that that’s actually a problem somebody is looking for a solution on?” Less around finding a hallucination in the results — but that’s something I see often.
Some teams are experimenting with AI moderators that can conduct interviews, probe responses, and adapt in real time. Are there categories of research where removing the human moderator actually produces better results?
Unmoderated studies have been around for a long time. What’s cool now is that with AI moderators, there’s more flexibility. How I like to think of them is: a survey on steroids. Typically, you write a survey with some branching logic, but with these AI moderators you can build in flexibility where the moderator might say, “Can you please say that in a different way? I didn’t understand your response.”
There are very distinct, to me, pretty narrow places where I think they make sense. Scale and volume — getting feedback across timezones, that’s pretty awesome. There are also a few documented cases showing that sometimes talking to a chatbot can be more effective. This came up with folks dealing with PTSD: they found that people were more likely to disclose how they were feeling to a chatbot versus a human.
The other place: researcher drift. A human moderator will do their first few sessions really, really well, but as you ask the same questions over and over, the purity of the test starts to decline, because they’re human. An agent never gets that — it asks the same way every time.
You’re never going to replicate the insight and fidelity that you get from talking to somebody in the first person. And there’s a trade-off in work: to do it well, you have to spend a lot of time upfront making sure that the questions are right, tuning how far out of band you’re allowing your AI moderation to go, doing a lot of pre-trial tests. Then you can set it free. Versus before — you write a screener guide, you start talking to people, you can adjust as you go. So there are certainly benefits, but you’re trading some work in one space for another.
Digital twins: Looking in the rearview mirror
There’s growing excitement around digital twins and synthetic personas. Could synthetic users eventually replace portions of exploratory research, or will they always be constrained by what we already know?
Where I really like digital twins is in making the quality of the research you do with real humans more powerful. Let’s say I want to get feedback from real users around a concept I just finished. I’d write up a little test script and then ask the digital twin, “Pretend you’re this persona and give me your feedback.” I can see where maybe some of my questions are confusing, what type of general feedback I’m going to get, and I’ll think, “Actually, I realize it’s going into this other area — I want to pull it back.” So it helps me refine my discussion guide before I actually go in.
The other thing that’s valuable — whether it’s a synthetic persona or just mining your research repository — is understanding: have we already tested this? Do we already know the answer? In large organizations, you’ve got all these teams and PMs asking very similar questions. You can easily go and pull that together: “What do we know today on this topic?” It might be six months out of date, but then I can focus my new insight work on stuff we don’t already know.
Do you find that there’s a lot of repetitive work across different groups or departments?
Oh my God, yes. I’ve managed research teams — so much of a researcher’s job is “we already know that, we already know this.” It’s been a challenge to know where all these little research studies have gone and pull them together. But yes, the same questions get asked all the time. And the answer can change over time, which is where we need to be clear about what we know today and add a timestamp. Oh, we did a study on mobile behaviors a year ago — my intuition tells me attitudes might’ve changed, behaviors might’ve changed. I’m going to retest that, which is totally fine. But having these synthetic personas or AI-mined repositories is just another way to get more value out of the activity you’re actually going to do.
Truth seekers, bias, and what to stop doing
Many great product leaders develop customer-informed intuition — that ability to recognize patterns across many interactions. Can AI help accelerate the development of that intuition, or does it risk preventing people from building it in the first place?
The things that really drive my intuition are those high-fidelity moments. I still to this day remember my first usability study and watching that poor woman break down and cry when she was using my prototype. That drove it. When you see somebody and talk to them and see the look on their face — to me, that is a critical part of developing that intuition.
That said, everybody is so busy. On the other side of the spectrum, what’s the alternative? You go by your gut anyway, informed by nothing — “Oh, I’m designing for myself,” or “my peer group thinks this is awesome.” So if we’ve got more signal in the mix, I think that’s a net plus.
The other thing, and this goes back to democratization: people across the company are making decisions constantly. The more people in an organization whose decisions are influenced by any type of customer signal, the better. It can be little stuff — a CEO glances at an insight and thinks, “Oh, interesting, that could inform a debate we’re having.” That’s what customer-centric really means: that people, in their day-to-day as they’re making decisions, are asking, “Well, what would the customer reaction be to this? What do I know about my customer that is informing this decision?” Net-net, more signal is better.
The caveat is that intuition also introduces bias. The newest strategy gets influenced by the last five customers you talked to, because it’s so fresh — and now we’re off chasing this other sparkly strategy. Were those really representative of your customer base? It does open up some risk. But I’d still rather take that risk and have people thinking about the customer.
When models are being developed for user research, does bias end up in there?
Oh, for sure. It’s a dumb example, but I’m doing a talk for our interns and I’m having images generated — all men, all White, just consistent. If I don’t actively tell it to diversify, it won’t, just because of what it’s been trained on. When it comes to building research repositories or digital twins, it has to be highly managed — “I want information based on this corpus of information. Let me know if you’re going out and referencing extraneous stuff.” The more you can tighten the context and scope of what it does, the less chance of hallucinations and inherent bias. This is why judgment and a critical eye are so important.
What’s one customer research practice that most product organizations should stop doing?
Quit resisting research democratization, if you haven’t crossed that chasm yet. But representing my research team, I’d ask for just a little sanity check: the biggest mistake I see is people running research when the decision has already been made. They’re basically just trying to validate proof for a decision that’s already been made. We call that research theater.
If you know yourself that you’re going to do this anyway, don’t play theater. If you are honestly just using it as a checkbox — don’t waste your time. Save your tokens, save your researchers. Just be really clear: “I want to inform a decision.” If that’s it, awesome. But if the decision’s already being made, don’t bother.
What does LogRocket do?
LogRocket’s Galileo AI watches user sessions for you and surfaces the technical and usability issues holding back your web and mobile apps. Understand where your users are struggling by trying it for free at LogRocket.com.