The HandbookAbout This SiteQuestions & Feedback
×
At least this isn't a full screen popup.
If these guides are helpful and you want more, we can send you an update when we complete a new guide, have a new recommendation, or uncover a new deal.
We won't share your info. This is mainly another way for us to get feedback on our guides. You'll be able to unsubscribe at any time.

The Best Web Analytics Tool

Last Updated May 30, 2020
Reading Times:
Summary only: 1 minute
Full guide: 27 minutes

Introduction

Every YC batch, Michael Seibel gives a talk about building product. He'll ask: "How many of you are using Google Analytics as your primary source of metrics?" But it's a trick question. When the majority of the audience eventually raises their hands, he'll fake a sigh and tell everyone that they're doing it wrong, and that they should instead by relying on an event-based analytics tool.The linked video is from Startup School, but the same talk is given at YC Core as well.

It's a perennial favorite, but don't let the humor distract you. Michael's got a good point. Without using an event-based analytics tool, which tracks the interactions your users have with your product, you won't know how your users are using your product. This is arguably just as important as actually building out the product. There are a lot of event-based analytics tools out there, but we think that if you're an early-stage startup, Heap makes the best tradeoffs.

Our Recommendation
Heap
Highest benefit-to-effort ratio
Gave us auto-tracking functionality, which we think is an exceptionally good safety net for an early-stage startup with rapidly-changing features and limited engineering resources, although it lacks some of the extensibility and more advanced analysis capabilities of its competitors. That said, we think this tradeoff best suits an early-stage startup.
heap.io

Summary

  • "Setting up event-based metrics is something that's super important very early in your company, because it's how you know whether your product is being used or not. And it's the number one source of new product ideas and inspiration." - Michael Seibel
  • Google Analytics by itself is insufficient to figure out how your users are using your product, although it is useful to complement an event-based analytics tool.
  • Event-based analytics help you figure out what actions a user took on your website.
  • We are primarily focused on evaluating the three main event-based analytics tools: Mixpanel, Amplitude, and Heap.
  • We found that event-based analytics is a category of SaaS with some of the most feature parity.
  • The core analytics functionality of all three tools is more or less equivalent.
  • The key considerations for an early-stage startup are the amount of maintenance and discipline required to maintain useful analytics, and the ability to go far without spending a lot of money.
  • We found that defining events in code while having auto-track as a safety net is a near-ideal setup for an early stage startup.
  • We recommend Heap, because it is the only major analytics tool that offers this capability. The main downside for an early-stage startup is that you can't get as far on a free plan on Heap compared to Mixpanel or Amplitude.
  • If pricing is the main consideration, then we recommend Amplitude. Even though its product is oriented towards later-stage companies, Amplitude has the most generous free plan of the major event-based analytics providers.

Ratings Matrix

Event-Based Analytics Tools
Core Analytics Functionality
Tolerance for Mistakes
Ease of use / Non-Technical User Friendliness
Extensibility / Integrations
Advanced Analytics Functionality
How far you can get without paying
Paid Plan Affordability
Heap
Our take: Gave us auto-tracking functionality, which we think is an exceptionally good safety net for an early-stage startup with rapidly-changing features and limited engineering resources, although it lacks some of the extensibility and more advanced analysis capabilities of its competitors. That said, we think this tradeoff best suits an early-stage startup.
Amplitude
Our take: Amplitude's startup deal provided the most generous free plan of any analytics product we tested. Much of their product is aimed at later-stage companies, but their free plan and startup deals really stand out for an early-stage company.
Mixpanel
Our take: A jack-of-all-trades, but which we think is better for larger companies than early-stage startups. You need more discipline to maintain Mixpanel's tracking than Heap's, and its startup deal, while quite good, isn't as generous as Amplitude's. Its main differentiators come from extensions and advanced analytics, but this mainly benefits larger companies.

Context

What is event-based analytics?

Pageview-based analytics tools like Google Analytics and event-based analytics tools such as Mixpanel, Amplitude, and Heap are typically grouped together under the heading of analytics tools. However, we think that lumping them all under the same heading is misleading, because they fundamentally do different things. To better understand what event-based analytics are, we need to understand what exactly Google Analytics does, figure out where its gaps are, and then use that context to motivate event-based analytics tools.

Google Analytics uses a pageview-driven paradigm, a holdover from what was important in the early 2000s. Its focus on pageviews helps answer questions such as how many users came to your website, what pages they visited, and how they found your website. Unfortunately, it isn't able to help you figure out which specific actions a user performed on any given page of your website. For example, it doesn't answer whether a user clicked on a CTA button, if they abandoned their cart, how far they got into signing up, or what part of the page they were looking at before they converted. Answering these kinds of questions can be quite informative to early-stage startups' decision-making, and can be accomplished quite easily by using an event-based analytics tool.

An event-based tool will provide two things: an interface to collect "events", and an interface to analyze those events. You can define and implement your events (commonly referred to as instrumenting) by calling the API of the event-based analytics tool you're using.

To illustrate, let's say you run an ecommerce store. You want to figure out what types of items people checkout without hesitation, and what types of items often result in abandoned carts. This isn't something that can be accomplished with pageview-based analytics tools such as Google Analytics. However, this can be performed quite easily with an event-based analytics tool. If you track two events — when someone adds something to their cart, and when someone checks out their cart — you can figure out which SKUs lead to a high checkout rate, and which SKUs lead to abandoned carts. This is implemented by calling two functions:

  1. a function track('add-to-cart', {<metadata>}), including metadata about the item added (e.g. SKU, name, price) in your code that executes when someone adds something to cart, and

  2. a similar function track('checkout', {<metadata>}) in your code that runs when someone checkouts. Data from these events will be sent to your analytics provider, which will provide an interface to analyze and draw conclusions from your data.

This is a simple example, but is already quite a powerful tool to understand your users, and can be easily extended. The same power generalizes beyond an ecommerce store to any startup's web product, helping one answer essential questions such as: Is your CTA convincing? What are the characteristics of products that customers are the most hesitant about buying? What part of the signup process needs to be improved to prevent potential users from dropping off? Which portion of your product page was the most engaging and persuaded customers to take the next step?

Early-stage startups and their analytics dilemma

If event-based analytics are so important for early-stage startups, one might rightly wonder why so many YC startups, whose founders are quite sharp in aggregate, rely primarily on Google Analytics?

It turns out that this is a rather illuminating question. The main contributor to this phenomenon is easy to understand and empathize with. Startups are busy and overworked, and analytics are often ostensibly seen as orthogonal to the priorities of understanding their users and building productThe canonical priorities for early-stage startups.. Therefore, event-based analytics often fall to the wayside as they typically require engineering time and discipline to maintain. If you forget to update an event, or don't have time to implement analytics for a new feature, then you can't get any benefit out of it at all.

Yet, in reality, event-based analytics are one of the best sources of new product insights and inspiration and are a fantastic way to understand how your users are using your product. This is one of our main motivations for writing this particular guide. Early-stage startups have a unique set of challenges to face, particularly around prioritizing their time and engineering resources. That said, we think that there's an approach that can make event-based analytics require a lot less discipline, and therefore make it a lot more attainable for an early-stage startup.

Methodology

Based on a few years of first-hand experience using and testing event-based analytics tools, in addition to aggregating feedback from other founders, we have found the following factors matter most for early-stage startups choosing an analytics tool:

  • Core Analytics Functionality refers to the standard analytics features that are critical for an early-stage startup to understand their users. These include standard analysis tools, such as funnel analysis (i.e. figuring out when users drop out in each step of a process), retention analysis (i.e. figuring out how many users churn, and when), and cohort analysis (i.e. figuring out how different user segments interact with your product).

  • Robustness / Ease of Maintenance: Early-stage startups need to use analytics just as much as larger companies, but also have to be a lot more resourceful and focused with their time and energy. What happens if you change a component, and don't update the tracking code? What happens if you're under time pressure from a customer and make the conscious decision to launch a complex product without tracking? While large companies have the ability to implement strict QA processes and are okay with extended deadlines, early-stage startups don't have these options. Therefore, we've kept a close eye out for how each analytics tool performs under less-than-ideal situations.

  • Affordability: Web analytics tools are somewhat notable/notorious for giving generous free plans, attaining lock-in, and then making things more expensive once your company starts becoming successful This is one of the reasons why we suggest using Segment, since it helps you avoid vendor lock-in and gives you more negotiating power.. That said, several analytics tools have generous free tiers or deals that can last you quite a long time without paying. Therefore, we've split this criterion into two subcriteria: How far you can get without paying, and Paid Plan Affordability. The latter is actually somewhat non-trivial to evaluate, because most analytics providers hide their pricing behind a Contact Us gate, even for early-stage startups. That said, we got our hands on a decent amount of data on the actual cost of each service, and we have also compiled a list of typical discounts and deals that you can reasonably expect to get from each analytics provider.

  • Ease of Analysis is a composite criterion that rolls into one the three related metrics of speed of analysis, non-technical user friendliness, and overall ease of use. Speed of analysis refers less to the number of milliseconds it takes to return a query, and more to how much time and energy is required to answer a new product/user question. Non-technical user friendliness is an important consideration because every member in your organization should have the confidence to access and utilize analytics data. While we found that every analytics tool could be adopted quite readily by engineers, making it as accessible to non-technical team members was a much greater challenge.

  • Extensibility / Integrations are an important (but likely not critical) consideration, as there are lot of things that you might want to integrate with your core analytics stack. To give some examples: how do you measure the impact of an A/B test on KPIs? How can you engage users (e.g. through SMS, push notifications, etc.) who fall into a certain cohort? The best analytics tools either provide this functionality in-house or provide integrations with third-party providers. We took a close look at how well these extensions and integrations function overall.

  • Advanced Analytics Functionality: We've done a blanket grouping of this functionality to refer to any analytics functionality that requires a lot of data to work. A quintessential example is predictive analytics, which try to predict new user behavior from existing behavior. In general, we discount this criterion relative to other ones when we make our recommendation, as we think it's less relevant to early-stage startups. Because statistically-relevant and actionable predictions require a lot of data (typically data points from 100,000+ users), we don't think these are particularly relevant for a typical early-stage startupAlthough they can become very valuable to larger organizations.. Moreover, these advanced analytics features are usually only unlocked by expensive paid plans, as they're one of the features used for price segmentation.

Findings and Recommendations

General Findings

Web analytics tools are unique compared to most SaaS products in that there is a high degree of feature parity between the leading web analytics products — there are actually a number of things that aren't fundamentally different between the main analytics providers, which means that, while important, we didn't see them as a reason to choose one over the other.

In general, each of Amplitude, Mixpanel, and Heap can sufficiently answer most analytics questions for a typical early-stage startup. They all have more-or-less equivalent core analytics features, such as funnels, cohort analysis, retention tables, etc., and are more-or-less equal in visualization, intuitiveness, alerts, APIsThey become functionally identical if you use Segment or a similar wrapper library., and reliabilityAll have scalable infra, IAM, data security, GDPR compliance. Core analytics features and user experience have converged for all three providers over the past few years.

For early-stage startups, we found that out of all the differences between the three main analytics providers, there are two that are key: 1. pricing, especially with respect to startup deals, and 2. approach to tracking, specifically auto-tracking + programmatically-defined events vs. programmatically-defined events only. Immediately, we need to give two caveats. First, we think that auto-track functionality is misused by many startups, and we'll delve deeper into this topic when we discuss auto-tracked events in more detail. Second — there are other differences between these three analytics tools, but we don't think that they're particularly important for a typical early-stage startup compared to the two that we previously mentioned. Regardless we'll point these differences out in each analytics tool's own individual sections.

Our recommendation: Heap

We think that Heap is the best analytics tool for a typical early-stage startup. Out of the event-based analytics tools that we tested, we found Heap to be the best-suited for products with rapidly-changing feature sets, despite it lacking some of the extensibility and more advanced analysis capabilities of its competitors. We think that this tradeoff is best suited for early-stage startups.

Heap's primary differentiator is that it automatically tracks every interaction on your website without requiring you to instrument each event, i.e without needing you to write code defining which interactions to send. This functionality is where a lot of the startup-specific benefits come into play, but is also something that we think is often misused. Specifically, we think that most startups should rely on programmatically-defined events as a primary source of data and use auto-tracking functionality as a safety net, not a primary source of events.

The Primary Differentiator: Autocapture

Note: This section refers primarily to Heap's auto-capture functionality, but can be generalized to all analytics tool that have some variant of this featureHeap, PostHog, and 2016-2018 Mixpanel..

For most analytics tools, you define an event in code. This means that if you make changes to your product, you'll need to modify your codebase to track those changes.

Heap's main claim to fame is their so-called autocaptureIn other libraries, this might be known as auto-track. functionality, which automatically tracks every interaction within your website. This means that on top of offering a programmatic API, which allows you to use track(<event>, {<metadata>}), Heap will also automatically capture data about every user interaction, from which events can be defined after-the-fact. Instead of modifying all of your track calls every single time your product changes, you can instead define an event on Heap's online dashboard at the point in time you need to answer a question about your actions.

Nominally, this also enables you to save engineering time because you no longer have to change tracking code while changing and improving your product (you only have to update events at the time you want to perform analysis vs. keeping them updated on every code release), and also prevents the availability of up-to-date data from being bottlenecked by engineeringWe think this is an underappreciated consideration. Many of the CTOs we talked to admitted to letting analytics updates slip on a big release, simply due to the additional engineering work required..

We found that often founders who use an analytics tool with auto-tracking functionality tend to rely entirely on events being automatically captured without programmatically defining any events. After extensive testing for ourselves, as well as our own historical experiences with analytics tools, we don't think this is the best way to use these tools. Instead, we think the ideal approach is to rely primarily on programmatically-defined events and use auto-tracked events as a backup.

Specifically, we found that relying solely on auto-track functionality has two primary upsides and two primary downsides, but prioritizing programmatically-defined events does a good job of mitigating those downsides.

Here's a deeper dive into each upside/downside:

Upside 1/2 — Error Tolerance. Early-stage startups typically have rapidly-changing products as well as limited engineering resources. This is an environment that isn't conducive to keeping analytics updated or comprehensiveIn an ideal world, this wouldn't be the case, although an ideal world would also have all products launched on-time without any tech debt nor code smells.. While important events should still be manually instrumented in code, automated tracking can cover the bases for some failure cases of programmatically-defined events common to early-stage startups:

  1. Someone realizes that an event is important, and should've been tracked from the beginning.
  2. An event has accidentally changed without notice, or hasn't been updated to match a feature update.
  3. A reasoned decision was reached to launch a feature without code-defined analytics to dedicate engineering resources elsewhere.

In each scenario, automated tracking will still have captured data of this event even if it wasn't defined programmatically, although it may require additional data wrangling in your analytics tool's online dashboard. In this manner, automated tracking is a big safety net, as well as a way to give startups more flexibility in deciding how to prioritize tasks.

Upside 2/2 — Non-Technical User Friendliness. Automated tracking enables non-technical users to start looking at analytics data as soon as a new feature ships, without requiring an engineer to first write tracking code before a feature can be analyzed. It's easy for a non-technical user to use Heap's selectors on its dashboard and analyze a newly-released feature. Without writing tracking code for every feature, any team member, technical or not, can still ask and answer analytics questions about those features.

Downside 1/2 — Event Definition Disorder. If you update code on your website, you'll still need to update the event definition in your analytics tool at some point. This can get very unwieldy as changes accumulate. In a concrete example: if you change your selector for your all-important checkout button from a.checkout to a#checkout and then to div.purchase, you'll need to keep track of all three selectors in your event definitionThis can get particularly unwieldy if you're piping your raw data into a data warehouse, although this is a more relevant consideration for later-stage companies.. In practice, we found this to be the main drawback of relying solely on auto-capture. However, if you programmatically implement the events that you know are important, your analytics data will naturally be unified (e.g. maintaining a single track('checkout') call that can be referenced with a checkout event).

Downside 2/2: Potential Lack of Focus. Trying to monitor everything can cause one to lose focus of the metrics that matter the most. Metrics, like many other things, fall under a power law — that is, a few metrics will be overwhelmingly important for your startup. If you have too many metrics that you're actively tracking, it'll be easy to become overwhelmed by the less important metrics and lose focus on the important ones. We found that this is a common issue with startup founders who used automatic tracking tools — when you have a lot of data, your instinct is to find some way to use it all. Instead you should be focusing on the subset that is important for your product, while using the automatically captured data as a backup in case your main tracking setup has an issue. Programmatically-defined events help with this, as it naturally focuses your analytics on the user progressions that you have programmatically implemented.

One final tip: you should be aware of unintentional data dredgingp-hacking when you perform retroactive data analysis. When you look at all of your data retroactively, it probably won't be too hard to find some explanatory story that fits a subset of the data Relevant XKCD/. With auto-track-based analytics tools, you'll get access to a lot of data, but it then becomes your responsibility to use it responsibly. This is also something that can be mostly mitigated by prioritizing programmatically-defined events, as the act of choosing which metrics to track constitutes a hypothesis.

This is why we believe that auto-capture functionality in general is better-suited as a complement to programmatically-defining events, rather than stand-alone use. That said, we think that having autocapture running in the background is something that's particularly well-suited for the challenges typical for early-stage startups, and is the reason why we think Heap is the best-suited product for those companies.

Other Considerations

So far, we've focused our discussion on automated tracking functionality. Outside of Autocapture, however, we think that Heap has some shortcomings compared to some other analytics tools. We don't think that these are deal breakers for early-stage startups (otherwise we wouldn't be recommending Heap in the first place), but it'd be good to be aware of them. These shortcomings typically affect late-stage or enterprise companies more than early-stage ones.

As we mentioned in our General Findings section, in terms of core analytics capabilities, we've found that pretty much all analytics tools have feature parity. All of them have advanced cohort analysis, funnel analysis, nice visualizations, etc. We actually think that product analytics tools are a category of SaaS with some of the most feature parity with respect to their core functionality. The differences come primarily in pricing, with additional differences in extensibility and advanced functionality.

Regarding pricing: Heap has an okay free plan compared to the other three main analytics providers, capping off at 5k sessions per month. However, after you take startup deals into account, Heap does the worst job of attracting new users. The only publicly available deal is one in which this session cap gets increased to 20k sessions per month if you embed an image of Heap's logo somewhere in your website. We've heard multiple founders get this deal without embedding the image, and we've also heard of other founders getting higher session volume without paying anything. We have a pretty good idea of how to accomplish this, but don't want to publicly say anything without confirming its repeatability (although you can contact us if you want to try your hand at it).

The nice thing about Heap's pricing is that once you move onto a paid plan, it generally costs less than both Mixpanel and Amplitude for an equivalent amount of users. The cheapest paid plan is nominally $500 a month for 125k sessions, though there's room for negotiation. Furthermore, the price doesn't increase that much as your session threshold gets higher. We hear $1k for 2.5m sessions after negotiation is quite reasonable. One thing to be aware of here is that the session-based pricing model likely won't be the best for your company if you're building a product where there are many short, high-frequency visits.

Heap focuses almost entirely on core analytics and its approach to tracking, whereas some of its competitors (primarily Mixpanel) have focused on creating adjacent services to add to their core product. For example, Mixpanel will give you the ability to send notifications (e.g. SMS or in-app push notifications) to users who fall within a cohort, or perform A/B testing with their own A/B testing suite. Heap has some integrations with tools like Optimizely and Intercom, which are more powerful than comparable in-house alternatives, but require a little more work to tie together. We don't think one choice is necessarily better or worse — it's just two different approaches that we think work wellIf we had to choose, we'd give an edge to Mixpanel, because a typical early-stage startup would likely value more seamless integrations and might not be able to use the full capabilities of third-party services..

Second, Heap's advanced analytics functionality is not as fully-fledged as some of its competitors, particularly Amplitude. That said, compared to Mixpanel's ecosystem of integrations, advanced analytics is generally more of a later-stage feature. Advanced analytics require a pretty large chunk of data to have statistical credibility, around the order of 100k+ users. They also usually only unlock at an enterprise-level pricing tierAKA the "Contact Us for a Quote" page..

If you want to go far on a free plan: Amplitude

We found Amplitude to stand out for two reasons: its focus on product analytics (to the point that other founders often rave about how intuitive and powerful Amplitude's analytics dashboard is), and its generous free plan followed by a cliff in pricing. Both have benefits and drawbacks.

First off, Amplitude has invested more time and energy into improving the experience of answering questions about a product than Heap or Mixpanel. Heap and Mixpanel gear their user interfaces towards relatively simple questions, e.g. a chart that tracks signups over time, or a straightforward retention table. This makes sense from a historical perspective, as Mixpanel, which was one of the first event-based analytics tools, was originally focused on a tier of analytics questions one step more complex than those of pageview-driven analytics. Amplitude orients their product towards making it easier to answer more complex questions, such as those that involve conditionals. We readily saw this in the user experience, as Amplitude puts more complex templates and dashboards up-front in their user interface, while you have to dig around Heap and Mixpanel a bit to get those set up.

These more complex questions are useful to answer, but also require more data. This is why we see a de facto trend of Amplitude being primarily used by late-stage startups and larger enterprise companies, as they have the scale to both get meaningful returns from more complex product analytics. In our experience, both through testing and talking with other early-stage founders, we found that Amplitude's focus on complex event-based analytics usually isn't fully-utilized.

That said, we found that a lot of the founders we talked to were quite happy with their choice of Amplitude. When we dug into this, we discovered that while Amplitude's ease of use was a major factor for their satisfaction, Amplitude's pricing model was what convinced a lot of founders to initially sign up.

Put simply, Amplitude has the most generous free tier of any of the big three analytics products. Their free tier gives you a quota of 10 million events with no restrictions on retention and # of seats, although you aren't given access to a couple of advanced features such as predictive analyticsAs we previously discussed, however, a typical early-stage startup won't have the amount of data required to make good use of those features. Moreover, Amplitude provides a deal where you can double this quota, as well as get full access to its advanced analytics tools, for free for one year. This is an exceptional deal. From our discussions with other founders who use Amplitude, both this free tier and this startup deal are more than sufficient to let you not worry about paying for analytics until your company has become successful.

The big caveat is that pricing shoots up a cliff after you pass the limits of the free plan. This should be mostly expected, as Amplitude's generous free plan means that its cheapest paid plan will provide more MTU-equivalents than the cheapest paid plan from Mixpanel or Heap. That said, of the founders we talked to who were at this stage of their company, the general trend was that Amplitude's quotes after negotiation were usually the highest. In general, we've found that Amplitude's first paid plan will cost around $18,000 annually.

Remaining Competition

Mixpanel

Mixpanel was one of the earliest bona-fide event-based analytics tools out there, and the first big success in the category of event-based analytics.

To start, we'll reiterate one of our primary findings, which is that for an early stage startup, Mixpanel, Amplitude, and Heap will all be sufficient (and quite similar) in helping you answer core questions about user behavior. In our experience, Mixpanel had a powerful and intuitive implementation of the core analytics functionality that we identified in our Methodology section, but so did Heap and Amplitude.

If we had to summarize Mixpanel, it would be jack-of-all-trades. Mixpanel M.O. has been to build in-house, feature-parity implementations of products in or related to analytics. For example, Mixpanel implemented their own version of Heap's Autocapture after the latter was launched, although this was recently retired (more on this in a bit). Mixpanel has also implemented their own versions of products in adjacent verticals, such as A/B testing and support for sending notifications to a cohort of users.

Mixpanel typically implements functionality from both its competitors as well as companies from adjacent industries, and brings this functionality in-house (we'll discuss this in more detail further below).

One notable omission, however, is the lack of auto-tracking functionality in its current releases. There is an interesting backstory here. Mixpanel originally had an implementation of auto-track to reach tracking parity with Heap. However, it was leaked that Mixpanel was inadvertently tracking passwords, and this eventually led to the retirement of that feature. We believe that this was an overcorrection in response to pressure from its larger enterprise customers. This issue was avoided in other auto-tracking implementations by ignoring all input fields by default. This is the primary reason why we believe that Mixpanel is less well-suited for an early-stage startup. The only way to track events is to programmatically implement them, which means that there is no recourse if your product changes and you forget to update the tracking code. This also means that there is no option to make a deliberate choice to defer updating your tracking code while still getting some sort of usable data. Mixpanel has some ways to mitigate the effects of these occurrences, e.g. you can get alerts for abnormal shifts in metrics, which will come into play if some part of your tracking pipeline is broken. However, this is still a reactive feature, as any data that should've been captured during that time is permanently lost.

Apart from the additional work required to maintain your analytics setup with Mixpanel, everything else is quite good. The analytics interface was fairly fast and decently intuitive — our only nitpick was that defining conditionals was sometimes frustrating for non-engineering team members. Overall, however, once the analytics dashboards were set up, there was no issue accessing and interpreting the data by any team member. One benefit on this front is that Mixpanel is the only event-based analytics tool with a mobile app. This may seem redundant, but we found it to make dashboards, KPIs and common analysis much more accessible within our team, as it promoted checking metrics on a more routine cadence for team members who otherwise wouldn't.

We found that Mixpanel's emphasis on building lots of extensions in-house was overall a net positive for an early-stage startup. The ability to perform basic A/B testing as well as engage users within a cohort, directly from the Mixpanel interface, was something that greatly simplified the process. This functionality is less fully-featured than a startup dedicated to each feature (e.g. Optimizely for A/B testing, OneSignal for user reengagement), but the integrated nature of these features means that they are much easier to use for startups with relatively straightforward requirements. This does mean a later-stage company may actually not be able to derive as much benefit from these extensions due to the lack of advanced functionality, but Mixpanel still has relatively robust support for integration partners, although to a lesser extent than Amplitude.

Mixpanel has a free plan limited to 1k monthly tracked users (MTUs). However, Mixpanel has a semi-public startup deal (typically offered only to companies to well-known startup accelerators, although we found you can take advantage of it via a referral) of 250k MTUs for free, plus access to their more advanced analytics tools. This is quite a good deal, although it may be hard to take advantage of (you can contact us if you want help getting it). After the free deal ends, Mixpanel is typically a more expensive option than Heap, although not as much as Amplitude.

Tips, regardless of your choice

Consider using Segment as a wrapper

We talked with 10 YC founders about their experiences with analytics tools, and a majority of them independently mentioned the benefits of using Segment as a layer between their website and their analytics tools, although most had some qualms about pricing. This mirrors our own experience, so we think it'd be prudent to dedicate a section to talking about where Segment might help, and our general thoughts on using it.

We expect many of you to know what Segment does, but we'll provide a brief overview here. Segment acts as a wrapper around analytics tools. To use it, you implement event tracking with a single unified API, and then select which analytics tools to send the data to.

Segment has two main benefits and a concern around pricing:

  1. It allows user IDs and session data to be more easily shared across different tools.
  2. More importantly for our discussion of analytics — Segment makes it easier to add or swap analytics tools. With event instrumentation already implemented through Segment, switching or adding analytics tools comes down to toggling a switch on a dashboard. This makes switching analytics providers quite easy.

It also used to be the case that Segment, in its role as a data proxy, provided all users with the ability to "replay" historical data into new analytics tools. Unfortunately, this has since become an enterprise-only featureNominally, this is understandable, because long-existent enterprises with lots of data and high switching costs would benefit the most..

The general consensus is that Segment is a well-loved product with a well-disliked pricingAs waxed emotional throughout Hacker News..

We think you should use Segment if you are able to get a discount (or usage for free), whether it's through Segment's free-for-startups program, your investors, an accelerator, etc.If this is posing problems for you, feel free to reach out — we might be able to help you get a discount. Otherwise, it might be better to take a look at an alternative. Two years of free Segment usage should be sufficient for most startups, but it may not be a good choice for slower-growing businesses. For a while, Segment was the only option out there, but we've seen some alternatives pop up recently Freshpaint and Rudder come to mind. and we'll be evaluating potential alternatives in a future guide.

Want More Guides?
We'll send you an update when we complete a guide, have a new recommendation, or uncover a new deal.
We won't share your info. This is mainly another way for us to get feedback on our guides. You'll be able to unsubscribe at any time.

Add a comment

Required
Required
Optional
Your email is only used to send notifications on replies to your comment.
mike88
Have y’all considered post hog? Heard pretty good things about their auto capture
kevin
Hi @satchel team, great review! If we want to use Heap.io for server-side tracking, can we still use the AutoCapture feature? Thanks!
Andrew C. (Satchel)
@mike88 We're doing a second round of testing to round out the event-based analytics tools. This includes PostHog, Google Analytics (the event-based side of things), Pendo, etc. Re: PostHog specifically — the main benefit vs. Heap that it's open-source, which => free (excluding the cost of self-hosting, of course) + privacy implications (both good and bad). We have a pretty good idea of which situations it'd work well for, which we'll lay out in the update.
Andrew C. (Satchel)
@kevin in general, autotracking as a concept doesn't really work for backend systems — for it to work, it'd have to have visibility into your backend code, which is 1. quite technically difficult to accomplish consistently given the variety of backend systems out there, and 2. it (almost certainly) wouldn't be robust. However, if you're using Heap (or any other tool) in a web app, you can still implement events programmatically in the backend. But if you have no web presence and are looking for an analytics tool for just your server code, then there are better options than Heap.
peter
Good comparison of deals offered. Used amplitude at a previous employer and had a good experience but wasn’t considering it now because it seemed like an expensive and later stage tool. The startup deal looks great, inclined to try it when we switch. Kudos
Recursive
I am CTO for a startup that is conducting a private beta (1k active users) and, having read this guide, investigated Heap for tracking user behaviors. It would have existed as a module within a larger tracking strategy unpinned by Segment. Unfortunately, Heap with Segment ingestion starts at $12k/yr and there is no month-to-month offering for early-stage companies.
jm
Would you ever consider the other types of developer-event tracking platforms like Sentry, Rollbar, Bugsnag, Airbrake, etc.? As to respond to the gentleperson talking about instrumenting backends, would you ever consider tracing frameworks in a similar discussion such as Wavefront, New Relic, Splunk, etc.?
Bernard
I think Freshpaint does Heap + Segment but better.
© Send Reality Inc. 2020. All rights reserved.