Naresh Govindaraj, Author at Trifacta

Recent advances in data architectures including the modern data stack have led to the rapid adoption of storage repositories such as data warehouses, data lakes, and more recently a best of both worlds’ solution with lakehouses. These repositories not only store data but also process data at any scale. In parallel, there has been a transition of data architectures from traditional ETL (Extract, Transform, Load) to a more modern ELT (Extract, Load, Transform) approach. This shift to profile and transform data using an ELT approach has been fueled by the migration to the cloud as businesses can now benefit from large-scale compute, near-infinite storage, higher return on investment, and lower total cost of ownership using the cloud. The Traditional ETL Design Paradigm. Let’s take a step back and talk through this transition. Prior to the cloud computing era, the traditional data stack consisted of data warehouses located on-premises along with an ETL platform. The data landscape was vastly different back then with a limited number of data sources and very little data volumes to work with. ETL was introduced as an alternative to hand-coding approaches. Since it was primarily driven through metadata, ETL frameworks were easier to work with. However, they were predominantly targeted towards developers, limiting their use and adoption across other personas. Data practitioners who understood the data had to rely on IT teams to build the data extracts, leading to slow, inefficient processes preventing businesses from achieving timely insights. This limitation became more prominent over the years as data volumes grew in scale, sources and types of data became diverse, and the business demanded dexterity, accuracy, and speed. Over the years, the same ETL design paradigm has often been repurposed to address use cases for data integration requirements in cloud data warehouses as a workaround. But, it has become evident this solution is inefficient and cannot scale as data practitioners need a modern and agile design paradigm to understand their data better. This further emphasizes the need for a solution that is built from the ground up, open, interactive, and collaborative to enable self-service and boost productivity. The Modern ELT Approach. Fast forward, the cloud has now become table stakes for data architectures. This is the primary catalyst for the increased ELT adoption, an approach that is tending to be the default data architecture. The premise of ELT is the modern data stack that is an intelligent framework bringing together the elements of data ingestion, data warehousing (in the cloud), data transformation, and data visualization. Due to the scalability and versatility of the modern data stack, businesses can now work with massive amounts of data across a gamut of sources. However, this also implies that a lot of raw data needs to be cleaned, profiled, and transformed before becoming consumable by downstream applications. This is no trivial task. Thinking through this further, what else needs to be done to achieve efficiencies with the democratization of data that can lead to easy access for driving value and achieving advanced insights? Let’s analyze. Modern, Scalable Data Engineering, A Speed of Thought Design. As we push more data towards cloud data warehouses, transformations need to happen in almost real-time without compromising on performance or scale. Data practitioners need to understand the data better, profile the data for high quality, perform the required transformations, and drive informed decisions quickly and accurately. IT teams who own the data and Lines of Businesses (LOB) who understand the data need to work in parallel without waiting or depending on each other. IT teams focus on governance, re-use, and scale, and LOB teams prioritize the dimensions of ease-of-use, self-service, and go-to-market. Organizations often fall into the trap of executing data processes in series, leading to inefficiencies. The goal is to avoid falling into this trap by driving the business forward with existing data and exploring future business opportunities. The data practitioner needs to be in the driver’s seat within the data paradigm. The modern data stack needs to enable the data practitioner to provide real-time, accurate feedback on the data, achieve faster insights, and scale with heterogeneous data. Real-time feedback is critical with a design that can process at the speed of thought, especially when dealing with unknown data from different sources. How can we achieve this in the most efficient manner? Stay tuned for the second part of this blog post where we will take a closer look at Trifacta, the Data Engineering Cloud to achieve all this with a modern design at scale. Trifacta is the only open and interactive cloud platform for data practitioners to collaboratively profile, prepare, and pipeline data for analytics and machine learning, meeting the needs of today and setting you up for future success. Learn more as we unveil the details in a blog post near you very soon.

Resources

Support

Upcoming Events

Webcast

DIY Data

Event

Wrangle Summit 2022

Start Free

Templates

Industries

Use Cases

Platform

Naresh Govindaraj

Efficient Data Engineering for the Modern Data Stack

The Need for Scalable, Modern Data Engineering

Products

Industries

Trending

Company

Cloud Technologies

Use Cases

Popular

Resources