It is difficult to prioritize product and growth work.
Should we write a new feature? Remove a feature? Fix a bug? Design a user interface? Remove a step in the activation funnel? Write a blog post? Offer an e-book for a lead nurturing campaign? Change pricing? Hire a customer support person?
How many people are getting authentic value from our product?
We say “authentic” value to avoid vanity metrics such as pageviews and sessions.
We don’t use the terms Daily Active Users (DAU) or Monthly Active Users (MAU). The North Star metaphor reminds us to use the metric as our guide. The moment when a user gets authentic value may not always be when the user acts.
For example, thoughtbot’s mission is to create better software. We believe a job to be done to create better software is to keep a project’s code in a consistent style.
Hound serves that job to be done by reviewing GitHub pull requests for style violations.
The moment Hound reviews is called a “build.” It is the moment the team gets authentic value because the product helps them keep their code in a consistent style. It is the moment the product’s job is done.
Hound’s North Star metric is therefore:
Teams with builds per week
We’ve often calculated the North Star metric using our application’s SQL database, hosted by Heroku Postgres. We run a SQL query on our production database’s “follower” (read-only slave) using Dataclips:
SELECT week, teams, ( teams::float / lag(teams) OVER (ORDER BY week) - 1 ) growth FROM ( SELECT date_trunc('week', builds.created_at) AS week, count(DISTINCT repos.id) AS teams FROM repos INNER JOIN builds ON builds.repo_id = repos.id WHERE repos.full_github_name NOT LIKE 'thoughtbot%' AND builds.created_at >= current_date - interval '14 weeks' GROUP BY week ) AS _;
The subquery aggregates our
builds data into
date_trunc function and
GROUP BY statement.
DISTINCT GitHub repositories
and alias them as
WHERE clause filters the results
in any appropriate ways for the particular product.
We usually remove employees of
the product’s company
and restrict the time window
to the previous quarter,
including this week in progress.
We have to be careful in the
understanding the domain and code
enough to know whether the state of the data
we are querying now
represents a good state of the data
for the time period.
The outer query
lags each week’s
OVER its previous
lag() is a window function that
compares rows of
teams by an offset of
The result set looks like this:
It is necessary to calculate both absolute numbers and growth rate in order to completely understand the product’s current health. We want to know Hound currently has “266 teams with builds per week”, and is growing at “9% per week” (the average of the trailing quarter, excluding the current week).
Paul Graham has written that Y Combinator startups should grow by 7% per week:
When I first meet founders and ask what their growth rate is, sometimes they tell me “we get about a hundred new customers a month.” That’s not a rate. What matters is not the absolute number of new customers, but the ratio of new customers to existing ones. If you’re really getting a constant number of new customers every month, you’re in trouble, because that means your growth rate is decreasing.
He also identifies our most-common follow-up question after learning the current North Star value and growth rate:
How many new versus retained users?
A second Dataclip calculates retained users:
SELECT date_trunc('week', builds.created_at) AS week, count(DISTINCT repos.id) AS retained_teams FROM repos INNER JOIN builds ON builds.repo_id = repos.id WHERE repos.full_github_name NOT LIKE 'thoughtbot%' AND builds.created_at >= current_date - interval '14 weeks' AND repos.created_at < date_trunc('week', builds.created_at) GROUP BY week;
It is almost exactly the same
as the North Star’s subquery
except for the
AND repos.created_at < date_trunc('week', builds.created_at),
which filters the data
to include only builds
from repos that were created
prior to the current week.
We can use
Heroku Dataclips’ CSV export
and Google Spreadsheets’
to get a live import
in Google Spreadsheets:
Which lets us chart the data and make other calculations, such as averaging the growth rate:
The new versus retained teams bottom chart is generated by importing the second Dataclip into a second Google Spreadsheet tab. On that tab, we calculate new teams by subtracting the retained teams from the total teams in the first Dataclip.
The “Goals” section is calculated by multiplying last week’s total teams, retained teams, and new teams by 1.07 (a 7% growth rate).
The “Current” section is calculated by dividing this week’s total teams, retained teams, and new teams by their corresponding goals.
The bottom chart and our current progress toward our goals helps us make decisions about whether to focus on customer acquisition (marketing) or customer retention (product).
The data is telling us that retention is good but acquisition is not.
The guidance from our North Star metric causes us to ask questions such as “which customer acquisition channels are working the best?”, or “do we have an awareness problem or an activation problem?”
Compare the Hound chart to another product’s chart:
In this example, the data is telling us that we’re still making something people want, not yet ready for marketing something people want.
The questions lead us to look deeper into our data in places such as Mixpanel funnels, which helps reveal where in the product people are getting blocked.
Here’s Hound’s activation funnel in Mixpanel:
While both steps could be improved, it seems more likely that improving the first step would result in a bigger improvement of our North Star:
Teams with builds per week
Numbers identify the “what.” Our team identifies the “why.”
To answer “why?”, we can critique the current home page for ways to improve the user interface design. For example, we might want to change the call to action from “Sign in with GitHub” to “Add Hound to your GitHub repo” or some other copy text.
Another way to answer “why?” is to look for qualitative data to compare to the quantitative data. We often use Intercom for user research.
In our conversations with users, we may discover that they are unhappy with the permissions we require in the GitHub OAuth step of the sign up process. That may lead us to change the OAuth permissions, removing functionality which relies on the more aggressive permissions.
We now have ideas for changes to make. In this example, the changes are to the product, but our process could have as easily led us to make changes to our marketing.
The product changes are only one part of the design of the growth experiment. We also need to define how we will know if the changes are successful.
Changing the CTA text should improve the conversion rate of from “Viewed Home Page” to a new “Clicked to Authenticate” event in the funnel.
Requiring fewer GitHub permissions should improve the conversion rate of from “Clicked to Authenticate” to “Signed Up.” That would represent the percent of users who have successfully authenticated with GitHub.
As sometimes happens in these iterations, we need to instrument the app with a more granular event. Then, we can compare the changes by A/B testing. Or, we could deploy the instrumentation and gather baseline data for a period of time before deploying changes.
After we feel good about our conversion rates, we might turn our attention away from product work, toward marketing work such as prioritizing customer acquisition channels.
Once we begin looking at the North Star metric on a daily and weekly basis, we’ve given ourselves an organizing mechanism for deciding where to focus our time.