Is your company ready for Predictive Analytics?

 

Is your company ready for Predictive Analytics?

Every business leader has now become aware of the application of AI/ML to Predictive Analytics. They would like to harness prediction using ML for the benefit of the company. There are some applications at the top of their priority list- sales and demand forecast, failure prediction, probability of prospect conversion, prediction of employee attrition and so on. One can easily see that if a business can foresee even a few of these important parameters, it can gain a solid advantage.

But can every company gain from predictive analytics? Is there a classification or maturity level of organisations that indicates how well it can use ML for prediction? Let me put together a few ideas that will help you to analyse your company’s preparedness for predicting future.

The evolution of data maturity

In 1949, a scientist called Abraham Maslow put forward the theory of human motivation. Popularly called Maslow’s hierarchy of needs, his theory says the drivers of human behavior can be described as increasing levels of needs. At the lowest level of this hierarchy are the physical needs such a food and shelter. Once the basic needs are satisfied, higher needs such as safety, love and self-esteem drive people.

A similar hierarchy can be observed when you study the history of an organisation’s use of data. In order to put data to work, an organisation has to spend of resources to collect, store and distribute data. Thus there has to be sufficient motivation for the use of data if this spending has to be approved. Here comes the hierarchy of needs. An organisation’s data use typically grows over time, and we can identify some distinct phases by the motivation behind its use.

Compulsion: This phase is characterised by reasons that are mostly mandatory. The first of this type is usually memory and sharing. There are only a few people in the business at this time. They need to remember contact details of a prospective client, for example. They also need to share such information between themselves. They start noting information in a register, or in a spreadsheet. Thus the primitive data store is born. Another big driver is compliance. Statutory agencies force the businesses to submit records. Customers ask to send invoices, specifications, account details and so on. Most of the data collection in this phase is not conscious. The business people may even not be aware that they are collecting certain data.

Management: As the organization grows, it enters into the Management phase of using data. Now the data is being used for streamlining the operations of the company, mostly by improving processes. The data required now becomes more complex and bigger in size. The collection, storage and distribution becomes electronic, as other methods prove unmanageable.

There are two unmistakable signs of the management phase. The data is now centrally stored, as many people need access to it. And the other sign, linked to the first one, is that the users of data become data themselves. ‘Who needs access to what’ becomes important not only for security reasons, but for effectively designing the workflows of processes.

Insight: The management phase usually transitions into the Insight phase very soon. With large amount of data easily available in the organization, the desire to use it for understanding the goings on takes shape. Even during the management phase, some insights are already being extracted in form of reporting. But the insight phase is marked by an expansion of data not directly related to operations, usually called ‘Master’ or indexing data. Here the business leaders’ vision about their business starts getting mixed with day-to-day running of the operations.

Most large businesses have reached to this phase today. Companies have data analytics teams. Many have bought DIY tools for analytics so that subject matter experts can do the analytics themselves, without having to depend on technical people. In some cases, insights play such important part that a completely separate collection and storage system for data called Data Warehousing is implemented. These organisations have come a long way from the compulsion phase.

Prediction: As of now, this the final stage of maturity of an organisation’s data use. There is a realisation in the company that it has a lot of data of various kinds, and it can be used for predicting future outcomes. This stage however is not immediately adjacent, like the Management and Insight phases are. Most organisations have not been able to make this transition easily. But this phase is the topic of this article, so it deserves a separate section.

The Prediction phase

Though the prediction phase is next in the evolution of data maturity, it is situated far away from the insight phase. There are some peculiarities associated with prediction, which make it hard for companies to upgrade to prediction level. Let’s have a look at some:

Mathematics: Thus far, simpler varieties of logic could do the job of using data. In management phase, it was mostly simple queries on the stored data. Insight phase required more thinking and some advanced algorithms to present the data. But prediction requires you to think about an equation:

y = f(x)  

where-

x -> input parameters

y -> outcome

f -> the relation, or function that connects x and y

The tool used for Predictive Analytics is machine learning (ML), and its job is to take pairs of (x, y) and figure out f, commonly called Predictive Model. This means that you should be able to think of data in your company as an (x, y) pair. This is now much harder, because of the ‘cause and effect’ relation involved.

Outcome data: As we saw above, we not only need all the parameters that impact an outcome, but the outcome itself in order to create the predictive model. There are two problems that are commonly encountered – the outcomes of business processes are not always captured, and when they are captured, they are not always linkable to the inputs. Thus even if an organisation thinks it has a lot of data, it may not be useful for modelling.

Time: Machine Learning uses pattern in the data to figure out the model ‘f’. But it takes time for patterns to emerge. If the company or the business process is new, it has not stabilised enough for patterns to emerge. A startup experiencing a roller coaster ride will not find much pattern in any of its data, thus the model building will not be effective.

External data: In all previous phases, the data was all generated inside the organisation. In some cases, data was obtained from clients, suppliers and partners. But prediction requires data that is external to the company. For example, sales forecast may require data about climate, economic condition, holidays and even big sporting events. Such data need to be obtained from reliable sources and appended to organisation data.

Dark and unstructured data: With the evolution of data use, the collection of data also evolves. In the initial phases, a lot of data was on paper. Later, electronic systems are implemented to collect a majority of data. Prediction might require collection of data that is not easily readable. Such data includes visual and audio data, handwriting, parameters such as temperature and pressure, waveforms etc. A lot of useful data might be hidden in piles of text, which need to be extracted.

Using predictions

Making prediction is meaningful only if you know how to use it. In many cases, this is not so clear. Let’s see two examples:

Sales forecast: Any prediction progressively become simpler as you do it on aggregated outcomes. Predicting sales for the entire company is easier than predicting sales for a small city unit of that company. But aggregated predictions are not that useful. Effective interventions can be done only at more granular levels.

Failure prediction: Let’s say we are trying to predict failure of a machine in a factory. There can be two types of prediction. From the past data, it can be predicted relatively easily that ‘some’ machine in the factory is going to fail in the next two days. But this, as you can see, is not so useful. The other type of prediction actually points out that ‘this’ machine is going to fail today or tomorrow. This obviously is very useful, but it will require data that you may not be collecting today. Remember that you need both the input parameters and the outcome. Even if you start collecting data now, it will be some time before you can build a model. Again remember that it’s not the ‘time’ elapsed that matters, but enough incidences of failure. 

Can your company transition to prediction phase?

In his 1949 classic ‘The Hero with a Thousand Faces’, mythologist Joseph Campbell shows how most of of the world’s mythology follows a certain pattern, one that he calls a monomyth. In his words -

“A hero ventures forth from the world of common day into a region of supernatural wonder: fabulous forces are there encountered and a decisive victory is won: the hero comes back from this mysterious adventure with the power to bestow boons on his fellow man”

The progress of data in organisations is somewhat similar. We can probably create our own ‘data monomyth’ –

“An organisation begins with uses of data that are compulsory. It then begins to use data for managing its operations. While doing this, it realises that important insights can be obtained from its data. Finally, it overcomes the hurdles to cross to a higher level of using data for prediction.”

Will your company be able to use the power of prediction? Firstly, it is important to check what is the data maturity phase of your company. As we saw, the progress of data use is evolutionary in nature, so the easiest transition will be from one level to the next.

Then come the peculiar requirements of prediction phase. Some organizations will have better capabilities to deal with the mathematical formulation or gathering external data, may be because they have done it before. The companies which have more stabilized processes will stand a better chance at prediction.

The business leaders will have to think how to put to work the predictions they are able to make today. It will be important to begin with predictive analytics, as the data maturity is going to evolve, and will not be achieved overnight.

By Devesh Rajadhyax

Co-Founder, Cere Labs