Leader Spotlight: Why understanding the customer journey matters more than the technology, with Shri Nandan
Shri Nandan is a VP-level AI and digital transformation executive with more than 20 years of experience across telecommunications, healthcare, financial services, and insurance. She most recently served as VP of AI Products and Experiences at Comcast, and previously held digital products and customer experience leadership roles at Momentum Financial Services Group and Main Line Health. She currently leads DahliaCX, LLC, providing executive advisory services on AI-powered customer experience strategy, and serving as a CXPA Professional Member.
In this conversation, Shri talks about why the biggest lesson from her AI initiatives has been to start with the customer journey rather than the technology, and how a simple request, like a bill summary, can mask a much deeper problem. She discusses why vanity metrics can still be directionally useful, how regulated industries change the calculus on AI pilots, and why AI systems can never simply be launched and left alone. She also shares her own habit for understanding unmet customer needs: going to wherever the customer is and just listening.
Start with the customer journey, not the technology
You led an effort that improved a chatbot’s resolution rate from 65% to 95%. What did that teach you about identifying customer needs and pain points?
One of the biggest lessons learned was don’t start with technology, start with the customer journey. When we looked at all of the agentic frameworks and evaluated vendors, it was very addictive, and it was almost like drinking the Kool-Aid and saying, “Wow, I really want this.” It’s shiny, and it’s lovely, and it’s beautiful, and it’s fun. But why do you need it? I think we should always start with that question. Don’t start with the technology or the architecture or the vendor; start with the why about who asked for this.
As you went through this initiative, was there a point where you had to pivot or change direction?
Possibly every five seconds. The nice thing about AI is we’re all in this together for the first time. We built a lot of things in-house at Comcast, at Momentum, and at other places, so a lot of it was learning as we went along.
It’s not so much pivoting as making adjustments. It’s saying, “OK, if I do this, I’m creating a ton of tech debt, and I’d really like to rethink my architecture and see if I can course correct.” That’s the sort of decision that AI leaders need to be able to make, and not be afraid to say, “I think this may not be working. I need to make a change,” and make that change. If you are not able to move at that pace, the technology is going to outpace you and create a ton of problems down the road.
What customers ask for isn’t always what they need
Is there an example of a time when customers said they wanted something but it turned out to be different from the problem they were actually trying to solve?
This example is from a few years ago, at Momentum: The customer kept asking for a bill summary. Momentum has both retail and digital services, and the customer wanted a summary of their loan transactions throughout the course of the loan. So we said, “Sure, let’s do that,” and we spent a lot of time building it out, but no one asked what happens before the bill summary and after the bill summary. What is the journey? What is the pain point that the customer is trying to solve? They may ask you for a transaction summary, but what they’re trying to tell you is, “I would like to lower my interest rate. I would like them to make it easier to pay the loan back. Or, I want to turn it into a bigger loan because I wasn’t able to pay my electricity bill this month, and I need more money.”
There are many things that happen before and after that one moment. So moving away from just building out a product feature because a customer asked for it and, instead, asking what happened before and after and looking at the entire journey from end to end, makes a big difference in actually solving the real problem.
How do you coach teams to prevent them from optimizing around symptoms instead of needs?
This has actually become really easy in the world of AI. Let’s say we designed an agentic framework and decided to launch a bunch of product features. We are no longer talking about just features. We’re talking about constantly evaluating the agent, almost in real time, and feeding the evaluation of that agent into your product roadmap.
Even before you reach the customer, you’re starting to think about how you think your customer will behave. You can use historical data, you can use synthetic data, you can do all kinds of things to do these evaluations. So the way you build your product has moved away from, “First, let me look at customer experience, then find the pain point, then identify these features, and maybe it will work.” Evaluations are no longer a matter of weeks, they are a matter of hours.
Using all the technology at our disposal today, we’re able to populate the product roadmap in a way that is more meaningful.
Metrics that actually reveal unmet needs
What signals have you found to be the most reliable indicators of unmet customer needs? And, are there metrics that teams undervalue?
As a customer experience professional, I try really hard to move away from vanity metrics. We love dashboards, right? We like to print dashboards and show them to our C-suite and say, “Look, we have so much adoption. We built 10 features, and the adoption of those features has gone up.” But what did that really do? Did that translate into a business outcome? Did that translate into a more meaningful metric, like resolution?
If somebody logs in and clicks on a button, that doesn’t mean they went on to resolve the problem. It’s sort of like following the customer’s journey to look at the business outcome and saying “They were able to resolve the problem, and therefore we prevent a churn and maintain loyalty.” That’s the sort of insight you need to derive from your metrics. Otherwise, you end up chasing fake loyalty and metrics that look nice on paper but don’t really translate into anything that is meaningful for the customer or the business. It just means somebody clicked on a button.
What’s your process for figuring out whether a problem is really impactful to solve or more of an edge case?
There are two ways to go about this.
Let’s say you have a chatbot. You have all your transcripts to lean on. Throw all these transcripts into a transcript analysis tool and see what insights it comes up with — it will surface all the pain points where you see the maximum amount of problems. You can create a heat map of your customer journey and then identify which areas to go after. The other way is using agent evaluations, not just to see if your agent is performing, but also to see if your agent is solving a customer experience problem.
And then you can rely on traditional CX metrics. Review your dashboard every week. You’re looking for trends, making sure everything is moving in the right direction. I know that we shouldn’t be focusing on NPS or tNPS as one metric, but it does give you some indication of where things are currently. You can put it into the context of other metrics and other data and see what the trends are. That’s the traditional way of doing CX, and we can still apply those principles today.
Navigating regulated environments
How does identifying and solving customer pain points differ between highly regulated environments and industries that are less regulated?
The use cases are different, obviously. When you are looking at something like insurance or a bank or even healthcare, the first thing you have to be sure of is that any technology you build, whether it uses AI or not, protects customer data. You do not want to be leaking PII information, especially if you’re using OpenAI or Claude or any public LLM. You want to be careful with customer data, otherwise you’re going to lose trust. These kinds of industries are also very prone to fraud. Probably some of the lowest-hanging fruit when I was working in fintech was centered around fraud and how to mitigate it. We built a ton of technology to protect ourselves from what we call fraud rings — a group of people who just try to break the system to steal money and data.
So the use cases tend to be more around risk mitigation, fraud reduction, and customer data protection. It becomes front and center when you’re in a highly regulated industry. It’s also difficult to build pilots and scale them if you have a ton of regulation. It’s best to have good governance in place before you even launch a pilot.
Are there instances when regulatory constraints can actually lead to better products and customer trust?
Well, yes and no. If you allow the friction to seep into the customer experience, then it becomes a problem. If you’re trying to approve somebody for a loan or schedule a healthcare appointment, but the technology gets in the way of that and the customer is just looping around because you’re so afraid to let them progress into the next step, then that’s a problem. It’s really important that you evaluate your technology, whether it’s an AI agent or something else, and test it out before you launch it.
The quality assessment, especially now that we are in the agentic world, has become so critical. We had a huge quality team at Comcast running end-to-end evaluations and tests to ensure that the agents were performing. Running all of those evaluations, the quality metrics, the observability, and having the ability to move fast and test your technology before it reaches the customer is helpful. But if you’re in a highly regulated industry, I would recommend that you bring in your legal, compliance, and regulatory folks very early on so that you’re not reacting. That way, you’re building with their input as opposed to against the tide.
AI systems need constant human oversight
Can you share a time when customer behavior changed through a launch process and you had to rethink the experience?
When we launched some of our AI-based IVR systems at some of the organizations, we realized it solved some problems — it automated some things and made some things faster — but that also made the customer more savvy. Say I’m on a call and solved my problem very easily without going to an agent, so now what else can I do? Now that I have managed to reduce my bill, for example, can I get an extra piece of equipment without even talking to an agent? That gets fed into the model, and the model starts learning and training itself and giving other recommendations. We realized that what we need to do is not just build an agent and throw it out there — we need humans in the loop who can watch what’s happening. We need people to sit with the human agent, and see what’s going well, and what’s not going well.
So some of it is evaluation, some of it is customer metrics, but a lot of it is just humans being in the loop, constantly evaluating.
What is your approach to bringing customer-facing teams into product development?
From a technology perspective, we built customer data platforms that took the events that the customers were going through and fed them into a feedback loop. We also instituted listening sessions. You sit down next to a human agent and look at what’s going on. This is a business process improvement team that has customer experience and engineers sitting together with the agent, solving problems in real time. But you’re listening, and the customer’s struggling, and you’re saying, “I’m going to improve the agent and do this on the fly.” This is the kind of thing that bypasses bureaucracy and solves customer problems in real time.
We also created a summarization tool — it’s like a little clock that says whether the customer is happy or not. When the clock goes to the left, the agent sees that. It gives recommendations like, “Looks like the customer’s not happy. Maybe you should try saying this.” It happens in real time, and the AI takes care of that.
Where do customers still require human judgment, and where would it be hard to train that into an agent?
In a telecom industry, for example, or an internet company, or even healthcare. Let’s say a tree falls on my house. Do I want to talk to a bot? Probably not. I would be so stressed out and so scared that I would want to speak to a human who can help me through it. If I’m going through a medical crisis, I don’t know that I would trust a bot to take care of me. I just don’t think we’re there yet. That’s why it’s important to design your AI agentic systems in a way that you know exactly when the human should be in the loop.
If you can identify situations where you think it will not make any sense for the bot to be here, then you just completely escalate it to the human agent. In some way, that restores customer trust. In moments of extreme stress, you can hear the customer saying, “Customer service, customer service, customer service, human agent...” You can hear the frustration and the fear, so it’s important that we look at all the flows in our business and say, “Where does it really make sense for us to leave the humans in there?” It’s important to have empathy for the customer when you design your AI systems.
What separates teams that truly understand customers
Are there customer problems you’ve found that are just inherently difficult for AI to recognize?
It happens all the time. I think we make a lot of assumptions about what a bot can solve. Repetitive tasks, like changing the password or having better pricing recommendations, are the things that we can do using ecosystems of technology. But what happens afterward? It is entirely possible that a customer makes a decision to change their plan, but then abandons it. You’re able to ask the question along the journey of what happened as a customer enters the funnel: “They did this, and then they did this, but then they didn’t do that.” Being able to ask those kinds of questions, and being honest about what the customer interactions are, will really help alleviate some of those concerns of whether a bot can really solve the problem.
But there are fringe edge cases where the bot will just not be able to handle it, and maybe we can fix that by changing the agent, evaluating the agent, or by building a new agent. There will always be situations where the bot hallucinates or fails to solve the problem.
What do you think will distinguish product teams that genuinely understand customers from those who simply have access to more data?
I think all product teams should have access to the data, but product teams who can use the data to derive insights will make the difference. My product team in my past role built an analysis tool. Having no experience in data whatsoever, and not being part of their job description, they built an analysis tool with a frontend, a backend, an LLM, and everything. It simply takes a bunch of transcripts and analyzes the heck out of it, and comes up with brilliant recommendations that before would have taken us days and weeks. They were thinking outside the box, and that’s the kind of thinking that you need.
I had a product manager who took all of our product artifacts and threw them into ChatGPT to create a custom GPT product catalog, so that now anyone can search through our product catalog as if they’re talking to a person. That’s the kind of out-of-the-box thinking that product managers need to do, instead of just saying, “I’m a product manager. I’m going to wait for someone to tell me what they need, and then I’ll be looking for features.” Getting ahead of the data is going to be the winning solution.
Is there one habit you’d suggest product leaders adopt to better understand customer pain points?
What has always helped me as a product professional is going to where the customer is. I’ll just go sit down in the retail store and listen to what’s going on, or I’ll look at the systems that the retail employees are using, or I’ll go to a contact center and sit next to the agent and just listen for a day. Those are the things where you can really understand what’s going on when you’re not looking at the data. So, stay curious, and just go meet the customer where they want to be met — that helps.
What does LogRocket do?
LogRocket’s Galileo AI watches user sessions for you and surfaces the technical and usability issues holding back your web and mobile apps. Understand where your users are struggling by trying it for free at LogRocket.com.