A fast-changing landscape sees data architectures and data infrastructures adapting admirably to deliver to fast-changing requirements. Data skills and data expertise however? Not so much. We are still hopeful about potential synergies however, thanks to one of the most revolutionary tech of our time – Artificial Intelligence.

Companies like Cloudera have been at the forefront of innovation, adapting to changing customer needs and technological advancements. As the industry shifts from traditional data management to more sophisticated, real-time analytics and AI-driven solutions, organizations face both challenges and opportunities in harnessing the power of their data.

The Changing Face of Data Management

Remus Lim, Cloudera’s Senior VP in APJ, reflected on the company’s journey, “When Cloudera was born, big data was still a new concept to many organizations. The whole notion of machine learning, AI, was still not so mature. Back then, organizations were just trying to manage the data.”

Remus Lim

Since then, technologies have advanced, solutions and approaches have emerged and evolved. Organizations have gotten a better handle of their data, but the quantum of the challenge has increased as well.

The data landscape has transformed dramatically over the past decade, for example.

“Fast forward today, the data environment has gotten so much messier,” Remus explained. “You’ve got a lot more data.(Expectation of) the timeliness of the data has changed. Nowadays, for example, for fraud, even half an hour delay is not acceptable.”

Cloudera’s approach to serving customers involves a robust partnership ecosystem.

This shift has driven Cloudera to evolve its offerings. As Remus notes, “Cloudera has been driven by our clients for innovation and demand for more deeper and more comprehensive capabilities.”

The Technology Evolution

For some historical context of data management’s evolution, we could start by looking at the birth of relational data warehouses that was meant to address the limitations of operational databases. Fast forward to 2010, and data lakes emerged, offering scalable storage solutions. Here, the Apache Hadoop Distributed File System (HDFS) played a pivotal role in the development of data lakes, and Cloudera played a huge role in Hadoop’s and big data’s proliferation by being the first commercial vendor for Hadoop.

It was not long after when modern data warehouses followed, combining the best of both worlds and in recent years, we’ve witnessed the rise of data fabrics, lakehouses, and data mesh architectures. Each iteration has brought new capabilities, enhancing our ability to manage data and derive value from it.

Skills Gap and Industry Challenges

As data management architectures and infrastructure advances, so too must the skills of those managing the data on them. Remus acknowledged this challenge and said, “The skills don’t catch up. For example, the situation today where generative AI burst onto the scene and is being talked about so much – are there enough people who know about generative AI, already? It’s an issue that exists for the whole technology space.”

At a global level, Cloudera partners with technology giants Amazon Web Services (AWS), Microsoft Azure, Nvidia, and many more, to integrate solutions and serve joint customers.

Cloudera’s approach to serving customers involves a robust partnership ecosystem. As Regional VP in Southeast Asia, Lim Wee Tee described, “Cloudera doesn’t serve the customer alone. We serve our customers through some of these partnerships.”

Lim Wee Tee

These partnerships operate on two levels: global technology integrations and local territory-specific collaborations, whereby global partnerships require engineering to an extent to engage, educate, and bring awareness to customers about the offerings they have and can potentially offer.

“The other tier of partners are local and territory-level partners, who have deep understanding of our clients and deep credibility in the market. So for example, system integrators, data science specialists that are good at solving certain problems.

“We leverage the combination of the global partners who are tech partners, with the local partners (in our channel ecosystem) to solve problems,” Wee Tee explained.

The Future of Data Management

Looking ahead, the industry is poised for further transformation. What about telecommunications as an industry that could resell big data analytics or data management services? Remus noted, “Is it possible? Yes. Are there conversations? The answer is yes. Has there been significant success? I would say we are building towards it.

Image by freepik” width=”1000″ height=”668″>

“Telcos, at least in my region, are also transforming. If you look at the traditional telco business model, it was about selling SIM cards, it was about selling bandwidth, not data. For them to get into the data space, requires them to transform themselves, and we do see some telco operators doing so.”

Remus acknowledged that potential synergies exist, but beyond the data bandwidth to deliver data services, there are also skills, people, infrastructure, even regulation, and the business model, to consider.

The potential for collaboration is significant, especially at this time when AI, and data centers that can support AI processing, are in urgent demand. With the myriad of moving parts to take into account, it is almost impossible to pin down a definite formula for success.

The Cloudera executives unanimously agree however, that, “Everything is possible.”

The journey of big data management is far from over, and the next decade promises to bring even more exciting developments in how we collect, analyze, and derive value from data.