Working with Product Data: Different Types, Different Strengths

Dave Feldman
10 min readOct 10, 2024

--

Originally published in Berlin’s ProductLab Community as part of their Data-Driven PM series.

As PMs, we want to ground our decisions in data. But what does that mean? We hear about companies like Netflix, using huge volumes of data to guide product strategy — but our realities are often very different. We have far fewer data points than Netflix, Amazon, Meta, or Google — especially once we start slicing and dicing our audience. And simply having data isn’t helpful on its own: we need to turn it into meaningful insights.

Data: A Broader Definition

When we talk about data, we often think of quantitative data. It’s relatively easy to collect, but many of us lack the scale — the statistical significance — to gain true quantitative insight from it. But qualitative sources are great, too: we can derive a lot of value from a low number of data points.

In fact, even at scale we need both qualitative and quantitative data to give us a complete picture. Quantitative data tell us what is happening while qualitative help us understand why. And investigating the why can lead us to new what questions that deepen our understanding and help us make better decisions.

Example: My last startup, Miter, built software to help people run better meetings. Our app had a pre-meeting screen to build your agenda, an in-meeting screen for note-taking, and a big Start button to move from one to the other.

We could see in our product analytics that a lot of users weren’t clicking Start. They were engaging with the product, but not getting to the important bit. These data were anything but statistically significant: we were a pre-seed startup with tens of users per week. But we didn’t want to ignore the signal, either.

We hypothesized a why for the what: maybe the agenda looked too much like the note-taking experience. Maybe people were taking notes on the agenda, not realizing they were in the wrong place. To test that hypothesis, we did two things:

  • We looked at session replays in Fullstory. For privacy reasons we couldn’t see what people were actually typing, but we could see how much they were typing, and it was an awful lot for an agenda.
  • We looked at the product database and, indeed, saw agenda items that looked like notes.

That was more than enough to confirm our hypothesis. I updated the design, implemented it, and the problem evaporated: people began clicking Start.

Types of Data

With a broader definition of data in mind, what are all the things we can use for robust decision-making?

Product Analytics

This is the “clickstream” data that tells you what your users are doing — retention, usage over time, conversion rates, funnel analyses, and so on. It’s great for understanding behavior in aggregate and over time.

Example: I worked with a medium-sized startup to redesign their marketing website, including the sign-up form. A few weeks later the marketing team came to me, concerned: they could see in the product-analytics tool that the form’s conversion rate had dropped by 90%!

They had a what — the much lower conversion rate. They’d formulated a why, too — the redesign. But I was skeptical, and turned to product analytics to test their hypothesis. Conversion rate is a fraction, so I checked out its two parts. The numerator (# conversions) hadn’t changed, but the denominator (# visitors) had gone up 10x. Same number of conversions, but far more people arriving without converting.

From there, I dug into the traffic sources and solved the mystery: we’d created a new source of traffic around the same time as the redesign, and none of the new people were converting.

The initial what in that story is important: something was off with our sign-ups. The right why was even more important: if we’d accepted the first why, we’d have invested time and energy in fixing a problem that wasn’t there, while continuing to pay for a source of traffic that wasn’t helping us.

Tools: PostHog, Pendo, Amplitude, Heap (disclosure: former employer), and others.

Product Database

Product analytics capture one sort of user data: user behavior (events) over time. They can answer, “How many users visited the shopping cart?” but not, “How many non-empty shopping carts do we have?” or “How many people are in the average user’s contact list?” For that, turn to your product database — the thing that stores your app data.

Tools: The database itself is part of your app’s architecture and isn’t an analytics tool in itself. Larger companies may pipe its data into a data warehouse and point a Business Intelligence tool at it like Looker, PowerBI, Mode, or Tableau. Otherwise, you can answer straightforward questions like the ones above with database queries.

Using Quantitative Data Qualitatively

When you’re small and don’t have the scale for statistical significance, product-analytics and product-database data are still useful. You can use them qualitatively to point the way to further investigation, just as we did in my Start button example. There, this pseudo-quantitative data still served as the what (too few people clicking Start) to help us generate a why.

Session-Replay Data

Session replay tools let you watch an individual user session like a video. Because you’re looking at individual sessions, this is definitely not quantitative. It can be useful as a way to uncover opportunities, build hypotheses, investigate bugs, or discover the why behind a what. In my Miter example, session replay helped us validate our why.

Tools: Fullstory, Hotjar, PostHog, Pendo, Amplitude, Heap. (As you can see, product analytics packages often offer this as a feature.) Note that there are privacy concerns with session replay; choose and configure carefully.

User / Customer Interviews

Structured conversations with users, customers, prospective customers, or even users of your competition can help you understand how they see the world, how they perceive your software and the problems it solves (or doesn’t solve), and how their mental models differ from your own.

These conversations can be more tactical (e.g., learning whether users have need of particular functionality) or foundational (e.g., learning about their workflows, perceptions, and overall needs to discover potential product directions and inform existing ones).

It’s worth educating yourself and your team about user-research techniques if you haven’t already. Otherwise it’s easy to ask questions — and hear answers — that can lead you astray. For instance, if you ask a user, “Would you pay for this?” and treat the answer as ground truth, you may end up killing a great product or building a bad one.

Example: When I first joined Heap (a product analytics platform), I conducted interviews with data teams at a variety of companies — some customers, some not. I asked about their day-to-day responsibilities and challenges. I asked what made them love their jobs and what frustrated them. I gave them a giant piece of paper and had them sketch out the network of platforms and tools they used to collect and analyze data. Only at the end, when it wouldn’t bias their prior answers, did I ask about Heap and its functionality.

The results were illuminating. I did learn about their perceptions of product analytics; but I also heard how it fit into their broader worlds. And even more interesting, I learned about the human side of data work: the ways in which data teams frame their job in terms of impact on the team and the people in it. Had I simply asked for feedback on our product and roadmap, I’d never have heard any of that.

User Testing

Many teams use usability testing techniques to gather data: asking participants to evaluate a prototype or mockup in various ways. This can be valuable in determining whether you’ve built something users will understand, and can provide a little insight into how appealing it is. But again, you’ll want to be careful about how you structure the test and what conclusions you draw. In particular, user tests can help you validate a solution but aren’t the best way to validate a problem.

Tools: UserTesting for conducting remote tests, Respondent for recruiting, Dovetail…but if you’re strapped for budget you can get by with Google Meet and a note-taking app.

Surveys

Surveys bring qualitative interviews closer to the quantitative realm: you trade a back-and-forth conversation for more responses. I like surveys for (a) getting a broad sense of how people perceive a topic, (b) double-checking what I thinkI’m seeing in interviews or other qualitative techniques, or (c) getting the lay of the land before doing more time-intensive interviews.

Example: We were considering the creation of a new product — let’s call it Widgets. It seemed like a promising direction, but it was a new domain for us. We wanted some quick initial validation of the problem before even investing in prototypes. So, we did a survey — in this case, a simple Google Form.

And the volume of responses was data in its own right: people were excited to talk to us!

In constructing the survey I was careful to avoid “leading the witness”. I didn’t ask, “Would you use Widgets?” or even, “How badly do you have this problem?” but rather, “What’s your biggest challenge working in this domain?” And it seemed we were onto something: without prompting 30% of respondents cited the problem we’d identified, and complained about the inadequacy of existing solutions.

Tools: For standalone surveys, Google Forms, TypeForm, or Tally; in product, PostHog or Pendo. And plenty of others, too.

Aggregate Go-to-Market Data

Customer-facing teams are a great resource for understanding how your customers perceive and use your product, and where their pain points are. The trick is to aggregate it: you want a bird’s-eye view of what these folks are seeing, not the thing that got in the way on the last sales call.

I’ve found it valuable to set up a monthly meeting between Product and go-to-market teams (Sales, Customer Success, Account Management, etc.) in which they present the top issues they see in the field, and any change since the last meeting. Product, meanwhile, presents its perspective and progress on the prior meeting’s list and any updates to the roadmap. It’s valuable to do this synchronously and not on Slack.

If you use a CRM (e.g., Salesforce), it can be used to identify and aggregate these things in real time — provided your go-to-market teams are up for it.

Competitive and Market Analysis

Look at what’s going on in the market. What are your competitors doing? How do they seem to view the world? What are people blogging and posting about? What does it say about your users / customers and their needs? Remember, though, that you can’t see into your competitors’ minds. If competitor A copied competitor B copied competitor C without further analysis of their own, you may have one data point that looks like three.

Gut Feeling

Yes, it’s data too! I’m not suggesting you simply trust your gut. But it’s telling you something, and that’s worth understanding. Be curious: where does that feeling coming from? What prior experience and knowledge may be feeding it? Is anyone else feeling the same way? I’ve learned, over the years, that when I have a sense of, “Something’s not right here,” — or to quote Han Solo, “I’ve got a bad feeling about this” — it’s something I should unpack.

Example: At Miter, our product synced with Google Calendar. That code was complicated — calendars are complicated, integrations are complicated, and our data model was different from theirs. I couldn’t shake the feeling that we were investing too much time in it, given how small we were (at that point, just my co-founder and me). We talked through it periodically, and every time we concluded there wasn’t a simpler way to implement it.

Much later — during our retrospective as we shut down the company — we realized: there was a way to test our hypotheses without a calendar integration at all. My gut wasn’t telling me it was too complicated…it was giving me a way to focus on validating over building.

I came out of my Miter experience with this as a personal area for growth: listening constructively to my gut, and building teams that excel at doing that together.

Final thought: build clarity with confidence scores

Not all data are created equal. They vary in how much you can trust them, and in what you can trust them for. As you work with data, it can be useful to apply a confidence score — it frames and contextualizes the data, and anticipates pushback. Saying, “The team feels like this is a good strategy,” opens you up to reasonable objections. If instead you say, “The team has an initial, low-confidence sense this is the right direction and we’re moving to increase that confidence,” that’s more robust.

Here’s a chart from Itamar Gilad, author of Evidence Guided (and a former Google colleague of mine):

As you can see, he’s using confidence not only to contextualize individual data sources, but to frame the entire product lifecycle: from initial idea through testing and iteration all the way to launch. At each step, we increase the investment in testing our hypotheses, commensurate with the confidence level of the previous step — until we’re sure enough to launch the real thing.

How have you used data effectively? When have you seen it misused? What are your success stories and challenges? What types of data did I forget here?

--

--

Dave Feldman

Multidisciplinary product leader in Berlin. Founder of Miter, Emu. Alum of Heap, Google, Meta, Yahoo, Harvard. I bring great products & teams to life.