Next week Ralph Kimball is presenting a webinar titled “Building a Hadoop Data Warehouse”. For those of us who have spent a significant amount of time working with data solutions, this is a milestone for Big Data. Kimball is the “father of business intelligence” and the mind behind data marts, star-schemas, and snowflake data structures in the data warehouse. His books are all-time best sellers on data warehousing.
Kimball’s webinar is a sign of the times. A new, Hadoop-driven data stack is emerging as an enterprise standard alongside the data warehouse. And while some would claim that the data warehouse is dead, the death of the data warehouse has been exaggerated. However there is a transformational change happening in the world of data.
Hadoop Changes the Approach
Data infrastructure and how we model data is changing – embracing the agility of schema on read over schema on write. But more interestingly the analytic process itself is changing. I saw this at both Aster Data and ClearStory Data. The data scientist and increasingly the business analyst is taking a new, more exploratory approach to discovering insight. They are asking questions of the data rather than the traditional approach of force fitting a data set into a key performance indicator and star schema.
If you know me, then you know that how organizations adopt data as a decision-making platform fascinates me – potentially more than data technology itself. I might blame it on my social science background. Or the fact that I’ve spent my career trying to solve different problems at different layers of the data stack- databases, middleware, and applications – and always coming back to the same challenge of user adoption. But it has always been clear to me that technology is only one part of the equation when working with data.
Business Adoption Has Always Been the Issue
You can see this in adoption rates in the market. We’ve delivered highly scalable, low cost data storage with Hadoop. Adoption of highly scalable in-memory data processing is available through Spark. But less than 25% of the potential user base for data-driven decisions is actually using a business intelligence tool. And while beautiful visualizations will improve the data storytelling of that 25%, they’re not going to assist with the broad-based adoption of data analysis for decision-making.
The adoption gap that exists in today’s new world of Big Data is a human gap. And it is not new. It has existed since the beginning of the data warehousing era. It is a gap in the ability for users involved in the process of analyzing data to manipulate the underlying data to their liking. This solution to this gap lies in a new emerging market space of Data Transformation. Refreshingly, Trifacta has acknowledged this gap and identified that both people and technology need to come together to cross this adoption barrier. That’s the foundation of Trifacta’s Predictive Interaction™ technology. It celebrates the ability of humans and machine learning algorithms to work together and increase the accuracy, agility and productivity of teams working with data.
Why Trifacta Is Making a Difference
If I can’t see it, touch it, and feel it, I don’t believe it – even if an advanced machine learning algorithm told me that it was true. So Trifacta ensures that the machine learning algorithms are working with humans to tease out and incorporate business knowledge in the data manipulation process. Transparency in data, democratized data access, and ease of data transformaton is what makes the analytics accessible to a broader audience of decision makers. It is what will drive more adoption in the Big Data era than we’ve seen in the past. It is what gives business people the courage to make hard decisions – based on data.
Keeping an eye on the data is how this market is going to evolve. And at a start-up the key to success in a big market is focus. I’m excited to join of a team that is focused on solving a big problem with incremental steps that uniquely deliver true business value. A team with the humility to understand that both people and technology need to be part of the solution if we’re going to change how data is used in the enterprise. I’m convinced that Trifacta is going to be a keystone in closing the gap between technology capability and broad-based adoption of Big Data analysis for business benefit.