Alex Chan, Author at Trifacta

As part of our expanded focus into Data Quality, Trifacta recently announced a new approach aimed at quickly and intuitively resolving data quality issues -- Smart Cleaning. This blog will cover two different features of Smart Cleaning -- Cluster Clean and Pattern Clean. Cluster Clean allows users to quickly and flexibly standardize and resolve similar values to a standard value, whether chosen from the existing data or entered by the user. Pattern Clean addresses issues with mismatched formats, for instance date or phone number formats. Within this framework, Trifacta Smart Cleaning brings scale and flexibility to the traditionally rigid processes of standardization. Smart Cleaning Cluster Clean We recently introduced a brand new way for users to standardize and clean misspelled or varied data in Trifacta, Cluster Clean. Many of our users struggle to deal with messy data that can be hard to reconcile. Sometimes it’s data that has been manually entered into systems, other times it’s data coming from multiple sources. We’ve found that traditional methods of clustering and standardizing similar values are slow and brittle. Users choose a clustering method and then spend lots of time manually moving values into the right cluster. When they get new data they need to painstakingly reconcile it with their existing clusters. Our approach is different, we know that no single clustering method does a great job catching every type of issue. We designed Cluster Clean with this in mind, allowing users to quickly explore multiple clustering options and catch new problems. It’s also resilient to new data, easily incorporating new values without being tied down to whatever clustering method happened to work best the first time. We approached Standardization with user flexibility in mind. Our initial model was centered around allowing users to take advantage of the clustering aspects of the key collision--comparing string similarity, and metaphone--comparing pronunciation, algorithms while being able to break out of these groupings if the user saw that the algorithms did not produce the desired groupings. To accomplish this, we allowed the user to not only specify a value for the cluster to resolve to, but also to specify the new value for any of the individual values within the cluster. Through user testing, we found that this model gave our users the flexibility that they were looking for, but it was missing a key aspect when the clusters didn’t match the user’s expectations: bulk editing. Users felt that it was tedious to pull multiple values out of a cluster one by one so we added the ability to select multiple values and edit them in bulk. Through this iteration, we discovered that selecting values was generally very useful and intuitive so we extended the pattern to work across clusters as well. Now, users are able to select a single cluster, multiple clusters, specific values within a single cluster, as well as specific values across multiple clusters and resolve them appropriately. Pattern Clean Another data quality issue that Trifacta can quickly address is issues of mismatched formatting. If you saw our blog on Active Profiling, you saw how Trifacta helps users identify and drill down on issues related to mismatched formatting. By interacting with that profiling information, Trifacta also gives users a powerful method to address those mismatched values, Pattern Clean. For those familiar with more traditional approaches to addressing these types of mismatched values, it often requires a heavy reliance on regular expressions and complex conditions. That can take enormous amounts of manual effort to identify those issues and patterns, and equal amounts of manual effort to replace them with the correct format. Trifacta identifies the patterns for you, and by interacting with those patterns, Pattern Clean will predict the best way to resolve them. What's Next This is just the first step in our focus on making data cleaning more intelligent and efficient, we have a lot of exciting extensions of this work coming soon! We’ll make it possible to use an existing set of ‘gold standard’ values to speed the process up even more with Reference Clean. We also have new clustering methods planned, allowing for more ways to explore how values may be related. We’ll be layering in more intelligence over time, giving users intelligent suggestions on how they can resolve their values and clusters. Try it out today and let us know what you think!

Resources

Support

Upcoming Events

Webcast

DIY Data

Podcast

Data Wranglers

Start Free

Templates

Industries

Use Cases

Platform

Alex Chan

Flows 2.0 – A New Flow Creation Experience

Trifacta for Data Quality: Introducing Smart Cleaning

Products

Industries

Trending

Company

Cloud Technologies

Use Cases

Popular

Resources

Resources

Support

Upcoming Events

Webcast

DIY Data

Podcast

Data Wranglers

Start Free

Templates

Industries

Use Cases

Platform

Alex Chan

Flows 2.0 – A New Flow Creation Experience

Trifacta for Data Quality: Introducing Smart Cleaning

Subscribe to our blog