As part of my ongoing series of interviews with analytic practitioners I caught up recently with Matthew Kitching, Senior Data Scientist at Apption. I am giving a webinar with Matt Kitching, Senior Data Scientist at Apption at 10am Pacific, May 27 on The Value of Predictive Analytics and How Using Decision Modeling Helps You Succeed.
Apption is a predictive analytics development company founded on providing value for its clients through quick wins and in building a long-term data science services relationship as a trusted partner. Apption differentiates itself because of its unique combination of deep data science experience and strong enterprise software development with client collaboration. At Apption, Matt works with clients to identify analytic projects that not only address business problems, but that return the most value to their organization and then with Data Engineers at Apption to implement these project.
What’s your background, how did you come to be working in analytics?
I’m a computer scientist who began working in artificial intelligence fifteen years ago, focusing on combinatorial optimization problems. Four years ago, I moved into the analytics field, in large part due to the explosion of big data technologies which expanded the power of machine learning.
I began working at 360pi, a company focused on price intelligence, which scraped large collections of prices of consumer products. It was a very data rich environment, where we were collecting vast amounts of data for our clients. The clients obviously wanted more than just raw data – they wanted a variety of data analytics. Clients wanted products clustered or classified so they could make sense of the data. They wanted visualizations to interpret the data. They wanted anomaly detection to find anomalous prices. It was clear that data on its own had little value, but the analytics supplied a huge amount of useful insights. So we started really focusing on analytics – from clustering and classification to visualizations.
What are the primary kinds of analytics you build at the moment?
We try to focus on using the latest technologies like Hadoop and Spark for big data analytics.
A big part of our analytic engagements involve cleaning and storing raw, noisy data from a variety of different sources. After that we work with a wide variety of algorithms, depending on the needs of the client. We are doing a lot of anomaly detection for security, and we are doing classification and clustering work for churn analytics and, more generically, customer intelligence.
In your experience what are some of the top challenges for analytic professionals in terms of maximizing the business impact of what they do?
The first big challenge is buy-in from the organization. Particularly with government clients, we find there is a lot of red tape and reluctance to adopt new technologies. In addition, we find both public and private industry to be understandably concerned with privacy and security.
Once we have buy-in we may have trouble identifying what data is available to us. Often, data is coming from different parts of the organization, and there is no single employee that knows all the relevant data sources. In addition, data is often inaccessible. For example, data has been inside legacy systems in factories.
During the development of solutions, the top challenges is the noisiness of the data, particularly missing, duplicated, badly formatted, and garbage data. Another challenge is how to process unstructured data. A final challenge is how to integrate the solution into the existing architecture of the client.
What have you found that helps meet these challenges? How have you evolved your approach to analytics to maximize the business impact of what you do?
From an organization level, we found incremental buy-in to be very effective. Showing our expertise by completing low-cost pilot projects. By having these early successes, the vision of analytics can mature in an organization, and we can get greater buy-in.
During an initial design phase, we spend a lot of time finding out what data they have and how accessible it is. We do have to make assumptions about the quality and accessibility of data during the design phase, and revisit our assumptions after data is delivered and ingested, long after the initial project plan is completed.
To deal with the noisiness in the data, we have found 3rd party tools for data management and data transformation very useful. Finally, we meet with our clients’ system architects to understand how our solution can be integrated into their system in the most efficient way.
How, specifically, do you develop requirements for analytic projects?
First, a business identifies how analytics can offer an ROI for the company. Often, the companies we work with know they have data, and they have a notion that analytics can be useful to them, but they don’t know how to capitalize on this asset. We work with the client to identify their business needs, we help them to understand their data, and we then present options to leverage clients data through a new product or service offering.
We then look at the data to see its features and characteristics. Based on our evaluation of this data, we work with clients to refine business goals to make sure they are feasible given data availability, privacy concerns, and available resources. We continue iterating towards project requirements that meet our clients’ needs.
There’s a growing interest in rigorously modeling decisions as part of specifying the requirements for an analytic project. How do you see this approach adding value for analytics professionals?
Many companies have a belief that they have useful data and that analytics could be useful to their organization, but don’t know exactly how analytics can be used. They don’t know what data is useful, and what is noise. I think modeling decisions is a great way to evaluate how analytics can actually benefit an organization.
Anything else you would like to share?
We see the decision modeling process as useful not only for specifying the requirements, but as an evolving document as requirements change and as new data becomes available.
Last question – what advice would you give analytic professionals to help them maximize the value they create for their organization?
View data from the point of view of what is gained for the organization. For example, when exploring a data source, we can find a lot of interesting insights, but an equal part of the job is finding ways that these insights can be used by an organization.
I am giving a webinar with Matt Kitching, Senior Data Scientist at Apption at 10am Pacific, May 27 on The Value of Predictive Analytics and How Using Decision Modeling Helps You Succeed.