Reverse ETL – The What, The Why, and The Who!

by | Feb 18, 2022 | Blog, Integration, Reverse ETL

The industry is making a lot of fuss about reverse ETL nowadays, and for a very good reason; Operationalizing your analytics makes the modern data stack much more powerful than ever before.

Modern no-code automated data stacks have been a dream for user-centric companies all over the world. In fact, it’s starting to gain pace, with more and more companies getting comfortable with the idea of fetching data from operational or transactional platforms such as Facebook and Salesforce into a central data warehouse for analytics and decision-making. 

But, just pushing data into a single place does not ensure that it gets used across the organization in their day-to-day work. So, how about pulling processed data from a data warehouse into these platforms so that your teams can use it in tools and settings already familiar to them?

In this blog, you’ll learn a little bit about Reverse ETL – the new entrant in the data space, and why you should be considering it while planning your data strategy.

What is reverse ETL?

Traditionally, ETL is about extracting data from a transactional database or platform and putting it into a data warehouse, where it can eventually be used to create reports and analytics.

So, what exactly is the reverse of this? 

Well, it’s kind of cool and mind-blowing. To put it simply, what we are doing with reverse ETL is this –  we’re taking data that has already been structured and organized for analytical reasons in a data warehouse and putting it back into operational applications such as Facebook or Google Sheets, where your teams can use this data for their day to day work.

Well, that’s interesting. 

However, as I went digging into this whole idea of reverse ETL, one thing kept bothering me over and over again — Since reverse ETL is as easy as extracting and storing data, Why didn’t it appear up until now?

Surely we could have just reverse-engineered traditional ETL or ELT and saved a lot of time & money that goes into ineffective manual processes?

To be honest, the issues were really fairly simple. 

  • Platforms or applications, in general, want to prevent sourcing from one big “SINGLE SOURCE OF TRUTH (SSOT).”

Okay! But Why would they do this?

It’s because it’s very rare that you can put all of your data into one place for usage, where you can also establish a common understanding of what different data elements mean for the various platforms that you are trying to put data into. Let that settle in.

  • Another big headache comes when you try to use analytical platforms that we use for reporting and analytics such as Looker, Power BI, or Tableau for reverse ETL. With these platforms, you need to use a power query to be able to put data in a format that makes sense for the transaction or user application platform that you are trying to put data into.

So, if you need an analyst every time you want to move a single line of data back into say Facebook to make it useful, it really isn’t seamless and it requires you to set context every single time. 

It is starting to look scary now, why would someone want to disturb the traditional ways of extracting and loading data when it is just too complicated?

The next sections cover just that. Read on!

Why use Reverse ETL?

So, now that we know a little bit about what reverse ETL is all about, let’s move on to ‘the why’. Why would you want to use reverse ETL? 

Let’s begin with two familiar platforms (or applications) that you are currently using in your daily business activities.

Salesforce, and Mailchimp. 

If you haven’t heard of them, Salesforce is a CRM tool, and Mailchimp is used for email marketing. 

Now think about this. 

As a marketer, you’re looking for ways to organize your email outreach for reaching out to potential customers in your target segment. To do this, you’ll pull records of potential customers into a cloud-based CRM – say something like Salesforce, then dance around with this data in an analytical fashion to make sense of things and then put it back into something like Mailchimp for running email marketing campaigns.

This is really what forward and reverse ETL looks like in practice. 

Here are some more common use cases where this part of the data stack really shines

  • Syncing key metrics and definitions that you built in your data warehouse into operational systems such as CRMs where your teams can actually consume them. These can include lifetime value, propensity score, and lead score, among others.
  • Fulfiling ad hoc data needs without the assistance of the data teams. “Can I get a list of *** in excel format please”?, Sure.
  • Personalising and hyper-targeting your market segment by syncing customer and audience lists with your marketing platforms.
  • Keeping product, sales, and support teams in sync (while using their respective platforms) about customer touchpoints, actions, and overall journey. 

I could go on and on, but I’m sure you catch my drift. 

The requirements are always evolving and may differ from one company to the next, but the pursuit remains unchanged.

You have the needful data in a data warehouse, and all you need now is a means to push it into the platforms you use every day with ease.

Now, it was alright if it was the 60s and the world did not keep changing overnight, sure you could achieve this manually. But, it’s 2022 and brands are competing against rising privacy restrictions, greater consumer expectations, rising volume and value of data, and so much more. 

Here is a screenshot of why manual reverse ETL is not sustainable in the long run. 

  • It is time-consuming, inefficient, expensive, and above all very SLOW!
  • It requires someone with a good amount of knowledge and skills to waste hours in a redundant and boring day in day out processing. 
  • One bad INSERT, UPDATE, DELETE can ruin the entire setup and force you to start again.
  • As you grow, scaling manual reverse ETL (or the usual ETL for that matter) becomes impossible. 
  • And, above all, if keeping TRACK of data in real-time was that easy, you would never have been struggling with data silos. 

Interestingly, a lot of Gartner and world-renowned research backs up the same idea and says that the difference between the leaders and laggards today is highly influenced by how automated, codeless, and real-time their data management practices are. 

But, still, we want to either keep it manual (read slow) or in-house (read expensive). 

Who offers Reverse ETL 

Until recently, companies wrote their own API connectors from various operational platforms such as Salesforce, Facebook, and Mailchimp to their data warehouse, and back from the data warehouse to these SaaS platforms, to pipe their data into an operational or analytical format. 

But, we have heard straight from the horse’s mouth – It is no fun!

  • Writing data connectors (forward or reverse) is time-consuming, inefficient, expensive, and above all extremely SLOW!
  • Most APIs are not built to handle real-time data transfers and hence need constant batching, retries, and checkpointing to bypass rate limits. 
  • Mapping fields from the data warehouse to the operational platform  and vice-versa takes a lot of time and precision.
  • API specs vary as they upgrade and it becomes a total nightmare to maintain connectors over time.
  • As your company grows, scaling connector volume becomes overwhelming and unsustainable.

No worries at all!

Enter 3rd party solutions. 

There are many companies and products available in the market today that are trying to solve the problem of writing and maintaining data extraction and storage connectors. 

DataChannel is one such solution. But, wait, we are far ahead from your usual ETL solutions. 

We offer both data ingestion (ELT/ETL) and reverse ETL in the same platform.

Is that awesome or what?

We believe given how similar the fundamental capabilities of piping data in and out are, having two separate setups (one for forward and one for reverse ETL) brings nothing but chaos, incoherence, and inconsistencies in the system. 

And, we already have a consistent reputation in the ETL space where we serve the likes of JUMP, Upgraid, iProspect, TheSouledStore, Clearlight, TheFaceShop, Fernish, Uable, WorkStore, and numerous others. 

So, as our user data stacks matured, warehousing data using fully managed ready-to-use zero-maintenance automated data connectors seemed “incomplete”. We wanted our users to push data into the systems and not just warehouse it. 

That’s why we decided to go a mile ahead and solve the “the last mile” problem. 

From making data-driven decisions to taking data-driven actions. 

Let’s discuss how you can implement the same. Talk to our experts and see DataChannel in action.

 

 

Request a Demo