How we enriched customer contact and organization data

We wanted to get more context for the data we collect, so we shopped around for data enrichment services. These companies can contextualize your data with more details. For example, given a domain name, a third-party enrichment service can give you the domain’s associated company name, its size, industry, and so on.

Data enrichment is a typical practice for growing companies, and certainly for large companies. But before you go shopping for enrichment services, you should have a known set of problems to solve (or goals to achieve), that enrichment can help with. For Metabase, we wanted to enrich our customer contacts for two reasons:

1. To keep in touch with our customers. Enrichment can alert us to job changes for contacts that may impact our relationship. For example, if a contact transitions to another job within or outside the company, we may want to reach out to congratulate them and make sure we’re in touch with whomever is taking over the relationship so that important product comms don’t get lost.
2. To gain a better understanding of organization size and industry, which can help us tailor our marketing and product efforts to make sure we’re building Metabase to solve the kinds of problems these segments face.

Evaluation of service providers

There are surprisingly many service providers for data enrichment, each with its own pros and cons. While we considered various LinkedIn data dump providers, we decided against using them, as we shouldn’t store that kind of data in our data warehouse. We also didn’t evaluate Clearbit, since it’s no longer available as a standalone service.

TL;DR: Apollo.io offered the best coverage, pricing, and features for our use cases.

Continuous enrichment

With the service provider selected, we used dlt and Apollo.io’s API to enrich new contacts hourly based on priority—prioritizing new contacts before updating existing ones, and so on.

Here’s a code snippet in Python that:

1. Gets a list of emails from a prioritized list of contacts from our data warehouse
2. Calls Apollo’s API to enrich those contacts.
3. Then saves the enriched information back in another table in our data warehouse.

def enrich_contact(self, postgres_connect_string, to_schema):
    pipeline = dlt.pipeline(pipeline_name='enrich_contact', destination='postgres', dataset_name=to_schema,
                            credentials=postgres_connect_string)

    @dlt.resource(write_disposition='merge', primary_key='email')
    def enriched_contact():
        with pipeline.sql_client() as psql:
            with psql.execute_query("select email from prioritized_contact") as cursor:
                emails = cursor.fetchall()

        for (email,) in emails:
            enriched = self.people_match(email)  # Call Apollo's "people/match" API
            yield enriched['person']

    print(pipeline.run(enriched_contact))

Modeling for self-service analytics

We added the enriched data, such as organization size and industry, to our customer and contact models. For organization size, we grouped the data into a few categories to simplify analysis for our teams.

, case
    when estimated_employees <= 50 then 'Micro'
    when estimated_employees <= 200 then 'Small'
    when estimated_employees <= 1000 then 'Medium'
    when estimated_employees <= 10000 then 'Enterprise'
    when estimated_employees > 10000 then 'Mega Enterprise'
  end as organization_size

In our contact model, we also added left_company_at to indicate when a contact left the customer organization. This field makes it easy to find out which organizations we should reach out to so that they can stay up to date on important product communications.

Final thoughts on data enrichment

Enriching our data has already proved valuable. One interesting insight from the enriched data is that our customers come from a mix of micro to mega organizations. Our teams have already used this data to better understand our customers and monitor job changes.

We’re now discussing how to use enrichment for additional purposes, such creating ideal customer profiles to make the most of our marketing efforts.

But enrichment isn’t free, so we’ll continue to evaluate the value we gain relative to the cost and iterate accordingly. And we hope this post can help you shop for an enrichment service that fits your needs.