The transformative potential of ML and generative AI to drive hard-hitting business outcomes, such as improved productivity and innovation, is undeniable. However, foundational data issues are limiting these opportunities. Gartner predicts that at least 30% of generative AI projects will be abandoned due to issues such as poor data quality, among other factors.
With more data being produced than ever before and new sources of data emerging daily, organisations are finding it harder to derive value from critical information dispersed across the cloud, the edge, data centres, mainframes, and end-user devices.
IT leaders agree: 73% recognise that some of their data exists in silos across their organisation and is not connected, according to Cloudera’s 2024 State of the Enterprise AI and Modern Data Architecture survey of IT leaders. In fact, more than half say they would rather get a root canal than try to access all their company’s data. Without trusted data, accurate insights can’t be derived in a timely manner.
Business leaders across various functions, not just IT, consistently confirm this, with many saying that they are struggling to justify the ROI from AI or generative AI investments due to weak data estates.
The data conversation is business, not IT-oriented
Unused data that remains neglected is typically poor-quality data, often a result of legacy systems and outdated processes. Most archaic IT platforms and data management frameworks that businesses are saddled with were designed for a bygone analogue or pre-cloud era. These cannot keep up with the relentless pace of data production or the growing complexity of data sets generated today. Furthermore, they are unable to cater for real-time analysis or scalability, limiting speed and agility in critical decision-making.
The only way to ensure data reliability and AI readiness is through the adoption of a modern, hybrid data architecture, which makes it easy for businesses to simplify data access, structure the data, and extract actionable insights to drive business growth.
Deploying a robust platform is a business — not IT-oriented — discussion because it addresses several strategic enterprise imperatives: access to data, security, and cost efficiencies.
Achieving business imperatives
The multi-cloud, hybrid environments that most organisations operate today add complexity to identifying and accessing data sets. Additionally, the distribution of data and workloads between on-premises and public clouds is dynamically evolving, making it increasingly intricate to locate data at any given time. Over the past 12 months, more organisations have begun repatriating workloads back to the private cloud, driven by concerns over security or cost. A hybrid data platform simplifies this process by enabling seamless movement of data sets from any source to any destination without requiring applications to be rewritten for data management and analysis.
Another critical feature of a modern data architecture are the analytics tools to make sense of unstructured data. The Gartner 2022 Strategic Roadmap for Storage estimates that new formats of unstructured data are growing 30–60% annually, which can be converted into real business value. For example, the copious amount of user posts and reviews about a retailer’s products across its social channels, e-commerce site, and partner listings can yield excellent insights on what the company needs to do to increase traffic, boost ratings, and enhance the customer experience.
In terms of cost efficiencies, organisations benefit from scalable analytics infrastructure that supports business growth, whether through expanding operations or entering new markets. Scalability ensures that growth demands can be met without compromising performance or escalating costs unnecessarily.
Lastly, in an era where data security and compliance are core business concerns, adopting solutions with built-in security and governance capabilities is essential. These include features such as encryption, access controls, and auditing, which help safeguard sensitive information and reduce the risk of data breaches. Such measures are particularly important in industries with strict regulatory requirements, such as financial services.
A strong data foundation is a veritable gold mine
Case in point: PT Bank OCBC NISP, a publicly listed Indonesian bank, faced the challenge of positioning itself as a digital-first institution in an increasingly competitive landscape. Wanting to tap on the immense potential of Gen AI solutions, they adopted a hybrid data strategy designed to integrate seamlessly with the bank’s data lake and enable data scientists and business users to work efficiently with various integrated applications.
For example, a publicly listed Indonesian bank aimed to position itself as a digital-first institution in an increasingly competitive landscape. To harness the potential of generative AI solutions, it implemented a hybrid data strategy that integrates seamlessly with its data lake, enabling data scientists and business users to work efficiently with various applications.
With this strategy as a foundation, the bank developed a scalable infrastructure equipped with tools and frameworks to support generative AI projects. This infrastructure facilitates the deployment of transformer-based AI models, providing real-time, personalised recommendations to customers. By integrating AI capabilities at scale across the organisation, the bank has enhanced its ability to innovate for customers and streamline regulatory reporting processes.
Analytics, ML, AI, and generative AI hold immense potential for organisations looking to ramp up on innovation, drive productivity, uncover cost efficiencies, and stay relevant in a competitive market. But for this to happen, businesses need a strong data estate to bridge existing gaps in data management. Effective data governance is no longer just an IT responsibility — it’s conversation that needs to happen at the board and leadership level.
