You often hear about being at the right place at the right time. But for the important things in life, the right people matter a lot more.
I’ve been thinking and writing about interacting with data ever since my graduate school days. But everything came to a head when I met the right people. I first teamed up with Jeff Heer and Sean Kandel five years ago. Jeff was starting up as a professor at Stanford, and was already one of the most influential thinkers and builders in data visualization, working on Protovis and D3.js. Sean had just made a last-second leap from the world of high finance to graduate school at Stanford. From a market standpoint, the world was converging on a broad-based understanding that data analysis was going to play an enormous role in the next stage of modern economy and society.
Sean, Jeff and I had reached the same conclusion: that the coming revolution of data was not just a matter of new backend engines and statistical algorithms—it had to involve people and put them in direct contact with their data. Much like the graphical user interface made computing ubiquitous, new interaction models would cement the Big Data revolution and drive the ubiquity of data analysis.
We also saw that this revolution would happen on the ground, by addressing real problems for the people working with data. We engaged in a series of discussions with data-driven workers of various stripes: from data scientists worried about statistical models, to business analysts trying to hit their numbers for the month, to IT professionals wrangling data for a living. Our clearest takeaway from these conversations—across the board—was that the greatest need for technology help was in the arduous and costly manual process of transforming data for use. We wanted to enter the market to solve a big problem—and this was it.
The analysts we spoke with told tales of spending up to 80 percent of their time “munging”, “wrangling” or “transforming” data. This included everything from discovering what the heck was actually inside an intriguing new data set, to structuring unruly data to match the needs of an analysis or visualization package, to cleansing the contents of a data set to deal with issues like standardized coding and outlier handling. We looked at the way users worked: typically by writing low-level code, or by repetitive mousing in spreadsheet software. Regardless of the approach, the work was mechanical and tedious, and cried out for a different approach. The challenge we saw was that each use case was different—the problem would not be solved automatically. People have to stay very much in the loop during data transformation: to iteratively assess their data, make decisions about how they want to use it, and validate their outcomes at every step.
After much iteration, we developed a new class of interface, designed to drastically enhance people’s abilities while they work with their data. We call our approach Predictive Interaction™. The idea is simple. First, the human side: ensure that the user always has eyeballs on the data—no matter its state—and can easily highlight what matters most for their purposes. Second, leverage technology: provide machine learning methods that anticipate a user’s desires and make recommendations based on what the users see and do with their data. Finally, close the loop: enable users to easily review machine recommendations and validate the outcomes.
The idea is simple, but achieving it requires a nuanced combination of design and technology. So we assembled what is without doubt the finest combination of engineers and designers ever seen in the world of data. And the results can now speak for themselves.
Today we announce the availability of the Trifacta Transformation Platform. It’s a milestone for Trifacta, and the whole team is excited to mark it with the customers who shared the road with us through our alpha and beta. I’ve been fascinated and delighted at what we’ve achieved already with Predictive Interaction: the ease, familiarity and power that it combines. And there’s much, much more to come.