Forrester: Why digitisation needs strong data engineering skills

Multiplying data

The first area to consider is that digital transformation generates more data – and that data is all over the place. The new normal drives more digital activity, which produces more data, both structured and unstructured – internal data generated by enterprise applications and customer experiences, as well as external data coming from partners, data marketplaces and social media.

The noble but idealistic vision that all enterprise data will eventually end up in a single – physical or virtual – analytical enterprise repository has come and gone. While this is still a recommended aspirational vision and strategy, the reality is sobering – anecdotal evidence shows that no more than 20% of all enterprise data that could be used to drive actionable insights is leveraged for that purpose. And with artificial intelligence (AI), this challenge will only grow as new models spin off more data.

In addition, new architectures and platforms will shift responsibilities. While the promise of a unified enterprise analytical data repository is always enticing, the current state of being able to find, access and analyse data wherever it is has driven the development of newer technologies such as data catalogs, data fabric, enterprise semantic layer platforms and knowledge graphs.

In turn, these technologies drive new questions for data and analytics leaders. Who is responsible for managing these new platforms – business, IT, or organisations reporting to chief data officers (CDOs) or chief analytics officers (CAOs)? Is it down to data professionals or analytics professionals to manage platforms that sit in the murky area between data and analytics in a typical enterprise analytical technology stack?

With the current state fragmented between on-premise legacy databases, cloud, lakehouses and the like, large enterprises will face complex decisions as they define roles and responsibilities.

Technology platforms

Another area IT leaders must assess is that modern insights platforms and solutions will multitask and overlap. Deciding what technology platform should support your data-driven business transformation is increasingly challenging.

Not only are new and emerging technologies driving shifts in responsibilities, but they also complicate the “who does what where?” questions due to their highly overlapping capabilities. For example, most leading enterprise business intelligence (BI), predictive analytics and machine learning (PAML) and digital decision platforms now include data preparation functionality. Further, BI platforms open their semantic layer to competitors’ BI platforms, acting as data catalogs, and the lines between automated machine learning (AutoML) and augmented BI platforms are blurring.

Then there is the question of data democratisation, which hasn’t resolved the language disconnect between business and IT. Business professionals speak about data, metrics and key performance indicators, while IT, data and analytics professionals talk about data models, schemas, dimensions and attributes.

Business professionals want customer insights and often don’t need to understand where in the data architecture the data originates. These different languages block what’s most important to driving value: working across the organisational chart to apply data and insights to actions that improve business outcomes. Although there is a determined desire to bridge this literacy gap through the proliferation of natural language query (NLQ) and low-code/no-code capabilities, the disconnect is still evident as global data and analytics decision-makers consider about half of their organisational data literacy to be intermediate level at best.

Speed to data insight

Forrester warns that impatience for underused and non-actionable data signals will grow. Data visualisation is not a panacea for underused data. Even seemingly obvious data visualisations won’t trigger insights or actions without proper context, effective data storytelling and advanced visualisation techniques.

A chart clearly showing a pattern like a trend does not indicate whether the trend is good or bad. Was the trend expected or unexpected? Does it exceed or fall short of goals and expectations? Is the red trend line meant to be an alert, or does the colour simply highlight a data point? Data and analytics business and tech leaders will increasingly be asked to answer these types of questions to better use data signals in a compelling and impactful way.

It’s no longer a question of whether data, analytics and insights are business or IT responsibilities. That decision/strategy is way too simplistic. Multiple shifts are happening in the C-suite, such as lines of business (LOBs) taking on more data and AI responsibilities, centralised enterprise functions moving towards governance and oversight roles, and advanced data-driven organisations increasingly separating data and analytics functions from IT and moving these teams outside of the CIO reporting lines.

In Forrester’s 2024 State of data, analytics, measurement and insights survey, 51% of organisations at the advanced level of data-driven business maturity with at least one specified senior data and analytics role have a CDO, compared with only 40% of beginner organisations. In 44% of advanced organisations, the CDO reports directly to the CEO, compared with 35% of beginner organisations. Below the C-suite, roles and responsibilities for delivering insights are also mixed among IT, business and data and analytics organisations.

The data-driven business

In this data-driven era, Forrester recommends that businesses evolve traditional roles to harness new opportunities. This dynamic transformation demands not only a shift in how we view and use data but also a profound reevaluation of the personas that lead and execute this vision within LOB teams.

Data belongs to the LOBs, where customer experience, marketing, finance and HR professionals generate the data and act on the signals it produces, while data teams ensure the ingested information is fit for use and is digestible by diverse users. Business leaders working with their data and analytics colleagues must explore pivotal new personas and enhanced skills that are mission-critical to thrive in this environment where data guides strategic decisions and actions.

No matter how many skills business teams are able to acquire, advanced data- and analytics-driven business capabilities will continue to depend heavily on technology-focused data and analytics professionals. Leaders are already introducing adaptive business processes such as BI governance, adaptive data architectures like data mesh, and adaptive technologies such as AutoML and augmented BI. Data and analytics leaders who are more aligned to the technology organisation must address the evolving and adaptive nature of tech-focused people skills.

Success in data-driven initiatives requires intricate coordination of multiple moving parts. Don’t be enamoured with emerging technologies claiming to be a panacea. Promised benefits won’t materialise unless your organisation has the right people and talent assigned to the right roles and with the right responsibilities and skills.

This article is based on an excerpt of Forrester’s Evolve data and analytics roles and skills for the adaptive enterprise report by Zeid Khater, with contributions from Aaron Katz, Boris Evelson, Kim Herrington, Karsten Monteverde and Jen Barton.