Why Don't Product Leaders Read Data Books?
Six books that every product leader should read
A few weeks ago, I recommended several product management books for data leaders in this article:
And just after hitting publish, I realized I could also write the opposite article, recommending data books to product leaders.
As I mention in my own book, Data as a Product Driver 🚚, product management books don’t teach product leaders how to actually work with data. The data function is either taken for granted or not taken into account at all in such books (and I don’t know what is worse!).
If we think about common product challenges like defining success metrics, you need to understand how they are created and calculated, how to interpret them in A/B test results, or know when you can actually set OKRs using those metrics.
So let’s go through six books that, as a data person, I've often started recommending to my product counterparts:
Measure What Matters (2018) by John Doerr
The Tyranny of Metrics (2019) by Jerry Muller
Fundamentals of Data Engineering (2022) by Joe Reis and Matt Housley
Data Mesh (2023) by Zhamak Dehghani
Experimentation Works (2020) by Stefan Thomke
Trustworthy Online Controlled Experiments (2020) by Ron Kohavi, Diane Tang, and Ya Xu
I’ve grouped them into three different sections. Let’s dive deep!
Understanding What You’re Actually Measuring
I’ve found that product leaders in companies where data is treated as a support function often struggle to work with metrics because they delegate that type of work to the Analyst. However, product leaders need to own the business definition of the metrics their team is responsible for if they want to grow into data-driven machines.
Two books taught me that choosing the right metrics is hard and dangerous in equal parts.
John Doerr’s “Measure What Matters” was my proper introduction to OKRs. I was already using OKRs at my company before reading the book, but I was using them wrongly. Something the book says that we didn’t do well: “It’s not a key result unless it has a number.”
Doerr also emphasizes that OKRs should be “public and transparent”. And I would add, easy to find! There are so many ways of doing OKRs wrong that I cannot recommend the book enough.
Jerry Muller’s “The Tyranny of Metrics”, on the other hand, provides the necessary counterbalance. It’s the book that makes you skeptical of your own metrics, so the challenge is to find a balance between both book ideas.
For example, the book catalogs the ways metrics go wrong:
Gaming: “Whenever reward is tied to measured performance, metric fixation invites gaming.”
Goal displacement: Effort shifts to what gets measured, away from what actually matters.
Measurability bias: Preferring options simply because they’re easier to measure.
Product leaders need this perspective because you’ll create metrics that your team will game (not maliciously, but we optimize for what they’re measured on). If you measure feature adoption, teams may push notifications to inflate the metric. If you measure retention, teams will make it harder to unsubscribe or delete accounts. Beware of that.
Understanding How Data Actually Works
Two books taught me why “just a simple dashboard” is never simple, and why architectural decisions matter for product strategy.
“Fundamentals of Data Engineering” by Joe Reis and Matt Housley is the book to go to for understanding the underlying architecture of “just a simple dashboard.”
The book explains the data engineering lifecycle: data generation, storage, ingestion, transformation, and serving. You’ll understand why layering your data warehouse is important. It will also help you understand why it is important for product leaders to have a say in event tracking, data contracts, standards, and governance. These topics are complex, so you want to nail them early.
The book, for example, distinguishes between operational data (what makes your product work) and analytical data (the copy you send to your data warehouse). As a product leader of a cross-functional team working in a specific domain, you own both and are responsible for making sure data is sent to the data warehouse for Analytics. That is a non-negotiable if you want to become data-driven.
Zhamak Dehghani’s “Data Mesh” is important to read to understand the bottlenecks of the centralized data warehouse (and data team) model. Dehghani proposes that cross-functional domain teams own their data as products. The analytics team doesn’t “own” customer data; the customer-facing product team does, and they provide it as a product to internal consumers.
This has profound implications for product leaders, which I develop further in my book Data as a Product Driver 🚚, but mainly:
Your product team should be providing high-quality analytical data about your domain, not just be focused on running the operations. If you own the checkout flow, you own the analytical data product that describes what happens in checkout.
“Data as a product” means that just like your API has uptime guarantees, your analytical data should have quality guarantees. Completeness, accuracy, and timeliness, to name a few, become necessary product requirements for data assets.
Sometimes you need data transformed specifically for your use case. That’s okay, building a data product that serves your needs, rather than forcing everyone to use the same centralized model, is allowed, but governance is needed.
Understanding A/B Testing
Last but not least, two books taught me that running A/B tests well is more complex than one assumes.
Stefan Thomke’s “Experimentation Works” makes the business case for experimentation and identifies the attributes of a successful culture of experimentation.
An interesting insight from the book is that: “Microsoft has found that only one-third of its experiments prove effective, one-third have neutral results, and one-third have negative results.” The takeaway you need to get used to is that two-thirds of the experiments generated knowledge and weren’t a waste. They helped the organization to learn, which helps refine and further optimize the product by better understanding your users at scale.
”Trustworthy Online Controlled Experiments” by Kohavi, Tang, and Xu is the technical manual that makes you realize how much you didn’t know about A/B testing.
The book introduces Twyman’s Law, “The more unusual or interesting the data, the more likely they are to have been the result of an error”, which is depressing by the way; deep dives into topics like Sample Ratio Mismatch (SRM), novelty effects, Overall Evaluation Criterion (OEC), and much more.
The book is important because most product organizations lack robust experimentation programs that help everyone run tests consistently.
After reading the book, I placed much more emphasis on pre-test preparation (writing a good hypothesis, aligning everyone on the decision criteria, etc.) and post-test analysis: no changing the success metric mid-experiment, or let’s run it and see what happens.
If any of the above resonates, choose the topic that presents the greater challenge to you, and I suggest reading one of the two recommended books in the section to start with. Any other books missing in the list?
If you’re interested in bridging the product and data worlds, you might like my book Data as a Product Driver 🚚.
Thank you! My reading list is growing but I just started Transformed based on your previous list.