Dan Woods is CTO and founder of CITO Research. He has written more than 20 books about the strategic intersection of business and technology. Dan writes about data science, cloud computing, mobility, and IT management in articles, books, and blogs, as well as in his popular column on Forbes.com. Check out his previous blog on why you should empower analysts to wrangle their own data here.
Have you ever sent someone else looking for something on your behalf? You know what it is, you know where it is, and yet, despite your detailed description, the person can’t find it. Ultimately, you have to go looking for it yourself.
Signal hunting—the process of finding meaning and structure in big data—is analogous in some ways. And as a business analyst, you have the domain knowledge: you know what you are looking for.
Signal hunting is the act of mining big data for useful nuggets or “signals.” It’s the process of exploring data for the purposes of discovery. You interact with data, identify unique elements such as value distributions and outliers, and use them to transform data on the fly. Signal hunting enables you to uncover data insights that you wouldn’t think to look for otherwise.
Signal hunting takes place during the data wrangling process, which has historically fallen to those in charge of Hadoop or the data warehouse. But big data has a low signal to noise ratio, and it takes business savvy to identify the signals. That’s why, as a business analyst, it makes sense to become empowered to signal hunt on your own. If you leave data wrangling to others, you miss the opportunity to hunt for the signal, identify outliers and dismiss the noise. In other words, you miss the valuable business insights promised by big data.
In order to successfully hunt for signals, you must be able to efficiently wrangle data and prepare it for analysis. If you spend all your time wrangling data, you won’t have any time left to hunt for signals. And that’s very often the problem: preparing data is the most time-consuming analytics task, taking up about 80% of an analyst’s time. The exhausting task of cleaning data leaves no time or energy for signal hunting. That’s why you might view this opportunity with a bit of skepticism: cleaning data is hard work. In our next blog post we explain how to overcome this challenge and empower yourself to become an effective signal hunter.