Leader Spotlight: Where finance, psychology, and product judgment meet, with James Falzone
James Falzone is Director of Product at Kargo, where he leads machine learning and marketplace initiatives at the intersection of finance, psychology, and product judgment. A University of Pennsylvania psychology graduate who focused on behavioral economics and game theory, he began his career in institutional equity sales and trading at Morgan Stanley before joining Kargo’s finance team in 2016 and transitioning into product management — a path that’s since taken him through a decade of building marketplaces at scale on a machine learning backbone. Today, he leads Kargo’s Outcomes and Auction Pod.
In this conversation, James talks about how his unlikely path from psychology coursework and Wall Street trading floors to ad tech shapes the way he thinks about risk, value, and machine learning at scale. He walks through the three-way tension every marketplace has to resolve between suppliers, the business, and buyers, how his team organizes itself around that tension, and what separates an ML model that scores well in a lab from one that survives real production traffic. He also gets into how his team’s architecture mirrors Conway’s law, and how he expects product judgment to change as AI compresses the cost of generating new ideas.
From Wall Street to ad tech: An unorthodox path into product management
You took an unusual path into product management. How has it shaped your approach?
My career trajectory, both into product management and machine learning product management, is definitely a little bit unorthodox. But I think it’s helped a lot and brought a lot of creativity and decision-making, as well as frameworks, to what I do.
It started when I was in college — I was a psychology major at Penn. There were many clinical classes, but I was always mostly interested in the behavioral economics and game theory components of it, which is where I did a lot of my coursework. My first job out of college was on Wall Street in the sales and trading division of a large, global investment bank. While that’s very different from what I do today, a lot of things have carried over.
That was my first exposure into what a marketplace is, because in sales and trading, you are a marketplace. Back when I started, it was when things like high-frequency trading were starting to pick up a lot of steam. So I was able to learn a lot about how a marketplace works, especially as tech gets more involved. But I knew that the trading culture wasn’t for me, and I had this kind of entrepreneurial bug. I wanted to be closer to technology. I wanted to build stuff.
I ended up finding this small company called Kargo back in 2016. I joined their finance team and learned a lot about how the ad tech industry worked. When I joined, we were building this new marketplace technology called an SSP, a supply-side platform, and I was able to transition from the finance team into the product management team. That’s where I’ve been ever since. Over the last decade or so, I’ve been focused on building marketplaces at scale, specifically with a machine learning backbone.
A lot of what we do today, when we connect buyers and sellers, is driven by some sort of machine learning optimization or algorithm. There’s always a financial analysis involved in how you scale a marketplace — we’re very P&L driven — but the fundamentals of a marketplace always stand true regardless of what industry or sector you’re in.
Did that finance background teach you when to hedge your bets, or when to cut your losses?
It taught me to be a bit less conservative when it comes to failure, and to embrace failure. My background in finance is helpful, because at this scale — in just assets under management, we’re talking about billions — a little mistake can cost a lot. And in ad tech, the scale of the data is actually in that same magnitude, when we’re talking about tens of billions of ad requests happening at any given day.
But the difference is that I was able to see that that scale should encourage your experimentation philosophy, because there is capability to have so much data that even the smallest test can have a really big impact. It can be quite frightening at times to see how big things get. That was a learning lesson that took some time: If there’s a lot to work with, figure out how to work within those guardrails.
Defining value in a two-sided marketplace
How do you define customer value when each side of a two-sided marketplace wants something different?
It’s constantly evolving. If you go back to the marketplace model, as far back as ancient times, the game theory behind it is always the same. In order to be a successful marketplace, you have to accomplish three things: you have to provide revenue to your suppliers, you have to provide profit to yourself, and you have to provide value to your buyers.
What’s really interesting is that profit and revenue to your suppliers are quantitative financial metrics that you can calculate very quickly. Value, on the other hand, is constantly changing, and it is up to you — via research, experimentation, and the occasional bet — to really decide and define what that value is. The equilibrium of achieving those three things, which are technically always pulling at each other, starts with defining what that value is. If you do that first and drive value to your customer, then the rest will follow.
Is defining that value where teams struggle most?
Absolutely. It changes, and you end up in a lot of places where the customer is not really going to tell you what that value is. In our industry, for example, even though we might sell to a customer, we’re not actually selling to the customer directly. Our machine learning model is interacting with their machine learning model. So even though you’ve done what the customer has asked, or established the marketplace fundamentals to drive those three things, you might end up in a world where the models don’t agree with each other on their own optimization goals.
What ends up happening is that you have to react quickly, but you also have to understand that there is technical complexity to how you set up what you’re offering. The contextual understanding of the business model, and what that buyer is trying to achieve via your marketplace, should always come at the forefront of whatever decision you’re making.
Organizing teams around a marketplace’s north star
How do you translate marketplace complexity into product decisions the whole team can align around?
That’s kind of the secret sauce of the job sometimes. Something that pushes too much in one direction can make it very difficult for the team that represents the other side of the marketplace. So first and foremost, establish a north star goal: to always try to hit that marketplace equilibrium.
There’s going to be some push and pull, but as long as you’re driving that value, the rest will follow. There’s a cultural component to it — this is the direction we’re going in order to drive value and scale our marketplace. But it ultimately comes down to how the team is organized and how you manage communication within that team.
For example , my team is called the Outcomes and Auction Pod. We’re a group of product managers, data scientists, machine learning engineers, and software engineers, and our tentacles spread and interact with a lot of data engineers and analytics engineers. We’re super close to product marketing as well, and sales, of course.
We accomplish building the system and the proper infrastructure, while also having business context at scale, by verticalizing every stakeholder into various squads. We organize around who each squad is ultimately serving — some squads are aligned to a specific customer relationship, and we also have a squad focused on our own internal marketplace needs. It’s about making sure everyone has a north star in terms of who they’re trying to serve, and then there’s an orchestration layer to make sure we’re all on the same page.
One of the things I talk about a lot is the concept of failing. Every two weeks our team meets, we do a retro — what went well, what didn’t, where are the blockers. But we recently decided to add “Where did you fail this week?” Because we’re really of the belief that if you’re not failing, you’re not trying something new. Why did you fail? Was it a disconnect with the north star fundamentals, something technical, or contextual information that was missing? Even though we’re organized in a squad format, the knowledge sharing has to stay collaborative, because the best ideas can come from different types of people.
What happens when different squads have conflicting priorities?
I know this sounds weird, but those are some of my favorite problems to solve, because it means we’re doing something right. Imagine there’s a yield issue that could impact a customer outcome, but that yield happens to be with a very important supplier, or vice versa: you might have a customer that’s super important who needs a certain access to supply.
Those problems are fine — that’s when you get into the heart of what we’re trying to do. The solution is usually machine learning-driven to some degree because we need a prediction and optimization layer that can help balance the tugging forces between those two squads. But everything starts with the conversation of: What problem are we trying to solve? As long as we’re focused on the problem space, building that ML algo is going to be much more successful than if we hadn’t.
From clicks to real-world outcomes
How do you build products that connect digital marketplace dynamics to offline behavior?
This is something that anyone who’s ever tried to run their own advertising campaign has felt — am I getting the return on my ad spend and driving those sales? You have to consider your goal: do you want foot traffic, or do you just want to make people more aware of your brand? There’s the measurement component, which is a really important factor in ad tech in general: make sure you are measuring the true impact of your ad strategy.
The second factor comes down to how we position ourselves as a marketplace: the differentiation of supply and the differentiation of data. Value is constantly changing for the advertiser, and so is their user journey. Just a few years ago, the way we researched what product would best fit our needs was very different. ChatGPT, or any other AI chat service, has really compressed that user journey. Kargo is one of the first ad tech companies to partner with ChatGPT to get access to its supply for advertising services. It’s really about understanding your measurement capabilities, and whether you’re meeting the customer at the correct point of the user journey with the appropriate messaging.
The next component is the differentiated data. Commoditization is a risk a marketplace always faces. Machine learning capabilities are a layer that add value to the customer, but much of that information is open source, so the true differentiation is the data you have in your training set. Having access to that data enables you to reach the user at the right point in the journey with the right messaging. This is what ultimately leads to things like brand lift, foot traffic, and eventually sales.
Building ML that survives production scale
What separates a machine learning model that works well in experimentation from one that survives at production scale?
Assuming the technical implementation is the same in both scenarios, I think there are two answers. The first is solid ML engineering and infrastructure — you can run your machine learning model in a training environment and get specific scores that show you how strong it is. But if the infrastructure isn’t there to serve it at a very high scale — one of our models is actually called 500,000 times a second — a training environment doesn’t usually go to that level of scale. So that’s the first thing: solid ML infrastructure engineering.
There’s also the contextual business understanding of the problem that model is trying to solve. One of the things that’s beneficial for ad tech is that we’ve been using machine learning for the last 15–20 years. The more specific the problem you’re trying to solve, the better the output of that model will be. If you try to use ML as a magic bullet without the contextual understanding of how market forces react when you introduce these changes, or without the specific value proposition you’re trying to solve for, that becomes the difference between getting good scores in training and deploying a model in production that doesn’t do what you expect.
Where do ML-driven marketplace products usually break first?
That’s part of the investigation. There will always be technical issues, maybe a pipeline issue, maybe a scale issue. But the most common reason for something not going right is that you’re trying something new, and you can’t define the value upfront, because that’s the point of the experiment.
The contextual understanding is usually where we start: we deployed this model in production, we’re not seeing the results we expected — why? We start following the flow, and if everything is technically set up correctly, then it’s a market force we can’t predict that has to be studied.
This ties back to what I said earlier: a two-sided marketplace has three parties, and all three are tugging at each other to get what they want. If the equilibrium isn’t met, we go back to those three pillars: was revenue lost, was profit lost, or was value not delivered to the customer? That’s usually where we start our investigation.
How do you design a system to change as fast as customer value does?
From a technical perspective, it always comes down to trying to avoid monoliths and breaking things down into microservices. The more specific you are with a problem you’re trying to solve, the more modular your system becomes, which means the more adaptable you become to reacting to those market forces without building up tech debt — which is, of course, inevitable in every system.
It comes back to solid infrastructure and systems engineering that allows you to be malleable in the future. But it also comes down to how you communicate as an org and get everyone on the same page. As long as you’re aligned in the mission, the people making those changes can become more adaptable too.
When something stops working, most people ask, “What do we do? How do we fix it?” If you take a step back and ask instead, “How has the value changed to the advertiser?” — that already gives you a framework for how to solve the problem.
Product judgment for an AI-accelerated era
How does organizational design show up in marketplace product architecture?
As Conway’s law suggests, organizations and systems are built following the same communication model that’s used within the company. This is something that goes back to the 1960s, and the genius of that statement cannot be overstated. It’s even more powerful today than ever before.
The way we organize ourselves, and the way our products fall under that umbrella, is that we have the supply team, the demand team, and the internal marketplace team, and underneath that a horizontal team that serves those internal customers.
Whiteboarding is one of my favorite activities to do, both with new hires and people who’ve been at the company a long time, because even if everyone is a subject-matter expert, as you start drawing things out and you start to uncover logic in a very complex system that you didn’t realize was connected to something else. We’re the marketplace, we’re connecting buyers and sellers. Through a whiteboarding exercise, we may realize we have a piece of code on the supply side that’s actually impacting a decision happening on the demand side. That’s normal as marketplaces scale.
How you pitch the value to your buyer is how you define it internally, and that’s how your org is going to be structured. Your tech follows that same shape, and your logic starts to work under those same rules. It’s almost like a game of Operation at times, because while you’re trying to maintain your current line of business, you’re also moving as fast as you can just to stay still. It’s that kind of Red Queen hypothesis thing.
How should PMs evaluate ideas now that AI makes it inexpensive to produce more of them?
One of the things that’s really exciting about the advent of chat-based tech, or LLMs in general, is that it has lowered the barrier for accessibility to code contributions and understanding. As a research-driven product manager, what that does is open the floodgates to new ideas. People who were otherwise unable to understand the code or how our systems worked can now ask a chatbot questions and get those answers — which means ideas can come from other places.
I’m a big believer that in any organization you’re a person first — you’re not an engineer first, or a product manager, or a data scientist, or a seller. The best ideas can come from outside sources. This is an amazing time for a PM, because now we have even more access to new ideas at our fingertips. But with every great power, you have the responsibility of filtering those ideas, and that’s where it becomes very difficult. The fallback has to become: How does it fit within that framework of delivering value? How specific is the problem you’re trying to solve?
There are two things that matter most. The first is the infrastructure it’s built on. In our industry, our auctions run within a second, and we’re talking about events that happen within milliseconds. LLMs cannot operate at that speed just yet, but LLMs can be orchestrators — they can run their own experiments and help us come up with ideas, but only if your infrastructure is in a strong place. So having new ideas means you have to invest much more in infrastructure to capitalize on them.
The second thing is that it becomes even more important to define the level of effort, the business criticality, and the risk of not doing something, and to prioritize initiatives based on those definitions — always keeping in mind what value it brings to the marketplace.
How will things change once LLMs can operate at latency speeds fast enough for real-time auctions?
When they do, I think there’s going to be more emphasis on things getting better — not just bigger. Quality will become even more important. Right now, it’s much easier to write documents or create new videos, but it’s not the quantity of the documents or the video — it’s their quality that’s most important. We’ll see how things change from an organizational perspective, or what it means for product managers to focus on better, not bigger. You can only do that if the fundamentals of your understanding of the problems within the marketplace stay true.
What does LogRocket do?
LogRocket’s Galileo AI watches user sessions for you and surfaces the technical and usability issues holding back your web and mobile apps. Understand where your users are struggling by trying it for free at LogRocket.com.