Information Technology

Big data should support instinct and experience – not replace them

By Gaurav Gupta, Affiliate, Kotter
Information Technology
Published: 13 February 2018

Boards and management teams once relied on information carefully collated over days or weeks, designed to answer pre-defined questions and presenting a picture a few days, or weeks or even months in the past: now they can see data against a huge range of metrics, accurate to within hours or in real time, and even interrogate that data during meetings.

Information in the form of data and analytics should provide relevant and useful insight, and help people make evidence-based decisions. But just because we can measure, map and monitor the minutiae of an organisation’s activities, doesn’t mean that doing so is always helpful. Indeed, the abundance of data available to decision makers can be inhibiting. It can dictate rather than inform decision making, and ultimately can supress the kind of instinct and intuition which helps individuals and organisations explore new ideas and take giant leaps forward.

More and more research shows that what we call a “gut feeling” is much more than a feeling – it is a combination of experience, memories, values, and instincts that are hardwired into our brains. We should be using “gut feelings” in conjunction with data to make good decisions. However, often the abundance of data and analysis entirely overrides instincts and intuition leading to poorer decision making. It is safer for leaders to rely entirely on the analysis, than to be accountable for a decision based on their intuition.

Imagine you are a manager sitting in a strategy discussion where a 100-slide power point deck has been reviewed with reams and reams of analysis that suggest a particular course of action. Your instincts tell you that the team is missing something, but the analysis seems so thorough and you are not able to put your finger on the problem. Are you able to question and challenge the conclusion or do you wilt under the power of the analysis? The latter would not be unusual.

Yet, with the increasing complexity in today’s businesses and uncertainty in the broader environment, even the best analysis can often be highly sensitive to variations in the inputs. Respecting our gut feelings and challenging our conclusions will help identify where the analysis may be incomplete or inaccurate.

Data is particularly good at answering questions about what has happened in the past, and the advent of big data analytics is providing greater insight into what we can predict about the future from the patterns of the past. However, determining which questions we should be asking still requires the sort of abstract thinking and open debate that only comes from the sharing of opinions and hypotheses that are often based on – you guessed it – gut feelings.

Dashboards are another example of how the incorrect use of data can send organisations down the wrong path. The ability to measure and track everything has led to dashboards with an ever increasing number of metrics. When a management team looks at a dashboard of, say 20 metrics, all rated in a traffic light system, attention easily turns to the glaring red metrics. These “misses” present a problem to be fixed.

It is true, these missed targets may indicate a serious problem, but equally if we are setting challenging targets there should always be some missed in each period. By scrambling to address these metrics and focusing on them at the exclusion of others, businesses miss upside opportunities and managers are incentivised to set safer targets. When decision making is driven by responding to data rather than by using it as an input, management teams can become remarkably passive.

This easy access to data has also created a situation where it is not being reviewed at the appropriate levels of the organisation. I worked with an organisation where a corporate mandate was established to reduce inventory at manufacturing plants and distribution centres. While lower inventories are a way to create value, the managers at the individual plants got so focused on improving this metric, which was being tracked in real time at headquarters, that bad decisions were made about maintaining appropriate stock levels. This in turn reduced flexibility and created production problems. The plant managers’ ability to make local decisions about operating their plants had been compromised, creating negative impacts on the bottom line and reducing the incentive for managers to be proactive in contributing to the organisation’s overall success.

Data is essential for any organisation’s continued existence. But it has to be the right data, used in the right context, and at the right level. For organisations to really thrive they need to be able to take the kinds of giant leaps forward that require robust debates which include discussion of intuitive, gut feel about what’s right. Data can support and inform in this context, but it isn’t the driver. When data analysis is used inappropriately without question, it can stifle debate and rule out opportunities to voice dissenting opinions.

Data and gut instinct can be formidable partners, and healthy organisations allow data to be challenged and provide space for instinct, imagination and experience to play their part in decision-making. Questioning and challenging the conclusions reached through data analysis will be increasingly important as developments in AI and automation create the ability to make more and more decisions without human intervention. Leaders must not abdicate decision making to big data and AI, but instead use these as tools in their decision-making process.

For more information visit