Explore challenges in end-to-end analytics orchestration and how data observability boosts developer productivity. Gain insights into trends and self-service analytics strategies with Rohit Choudhary, Founder and CEO of Acceldata.
From key trends driving the growth in data observability to how businesses can better plan and execute self-service analytics and data democratization, Rohit Choudhary, Founder, and CEO of Acceldata, says it is the time for data-driven organizations to develop a data strategy, provide data teams with the right capabilities to manage technologies to get ahead of the curve.
Excerpts from the interview;
How have data and analytics evolved recently? What have been some of Data Week’s most significant developments?
In the past decade, we have witnessed a significant shift in data processing, marked by the rise of successful open-source companies like Databricks, Cloudera, and Confluent. Cloud providers introduced purpose-built databases for specific use cases.
Data’s role has evolved from merely populating monthly reports to driving real-time operational use cases, intensifying the pressure on data teams to ensure data accuracy.
Democratizing data is crucial now, especially with universal access to technology through the cloud expected within five years. The leveling of this technological landscape underscores the importance for data teams to possess the capabilities necessary for effective technology management.
What should enterprises focus on to gain ground on data and analytics modernization initiatives?
Begin by defining your key business use cases. Are you utilizing data for operational purposes? What insights do analytical users require for their tasks? Do you aim to monetize your data? Anticipate the evolution of these use cases over the next 18 to 36 months.
After selecting a use case to invest in, prioritize technology with a robust community and a rapidly evolving ecosystem. Address unforeseen operational issues by incorporating data observability early in your planning process.
How can businesses better plan and execute self-service analytics and data democratization for users who need it?
Choose technology that suits your user community and has broad applicability. Opt for simplicity over overly complex solutions. Utilize a data lake or data warehouse architecture to centralize data, preventing silos and enabling diverse use cases with reduced complexity and administrative effort.
Why do data teams need data observability?
New data pipelines and databases are continually emerging, leading to a surge in data volumes and supported use cases. This compounding effect adds significant pressure on data teams to monitor everything effectively. Without the right approach, it can rapidly diminish data engineering productivity and hinder the success of your data teams.
The key to data success lies in developer productivity, and the strategy involves keeping them focused on business problems rather than grappling with operational issues related to computing, data quality, or data pipelines. Data observability encompasses the technology’s entire surface area, saving developers time and effort, ultimately boosting their productivity.
Furthermore, data observability establishes a common vocabulary, aligning data science, analytics, operations, and engineering groups. At Acceldata, we achieve this by providing a unified platform for data teams to manage their concerns, ensuring data is reliable, scalable, and optimized.
How do inefficiencies in the end-to-end orchestration of analytics pipelines affect organizations’ data scientists and engineers?
The data journey commences with generation in your ERP and other source systems. It progresses as it’s ingested into your data warehouse or lake and then transformed for self-service analytics, ML models, and recommendation engines.
Like any journey, unexpected detours and roadblocks can arise. Delays are especially disruptive for data scientists awaiting the application of value-added algorithms to fresh data sets. Such delays cause frustration for data engineers tasked with ensuring processes are complete as expected. They understand what went wrong and why is crucial in both cases.
Data observability provides end-to-end visibility across the entire data journey. Share an insightful use case for Acceldata.
Mobile payment app PhonePe, a division of Walmart, processes 400 million cash transactions monthly. With infrastructure challenges, PhonePe turned to Acceldata for monitoring HBase, Spark, and Kafka. This resulted in enhanced engineering productivity and an annual savings of $5 million in software licensing costs. Burzin Engineer, founder and chief reliability officer at PhonePe, attests, “Acceldata supports our hypergrowth, enabling us to manage one of the world’s largest instant payment systems. PhonePe’s monumental data infrastructure initiative was made possible with Acceldata.”
What are the key trends driving the growth in data observability?
Data volume stands out as the primary catalyst. Enterprises face an overwhelming influx of data and diverse use cases. Some organizations accumulate more data in a week than in a previous year. For on-premises setups, continual resource additions are necessary to prevent lagging. In the cloud, cost considerations play a crucial role.
Moreover, the need for more skilled engineers exacerbates the challenge. The demand for data scientists and engineers has surged, making it the top job in the United States.
How do C-suite executives leverage data to deliver business value to their organizations?
Effectively leveraging data transforms it into a competitive advantage. C-suite executives in organizations, regardless of size, are becoming increasingly aware of this reality, hastening the rapid growth of data utilization. Data-driven dashboards and reports enable C-suites and their teams to comprehend current performance, spot market opportunities, and expedite innovation. Harnessing real-time data for automating tasks, ranging from manufacturing processes to online shopping experiences, enhances overall performance at a reduced cost.
What are you excited about looking at the immediate future? What is your larger vision for big data and data observability?
I am enthusiastic about expanding the success of data initiatives to a broader range of companies, moving beyond just a select few large, internet-focused ones. At Acceldata, we strive to equalize opportunities through data observability. Our vision is to enhance data efficiency for 95% of the companies listed on Forbes’ Global 2000.
What advice would you give companies who are at the beginning of their digital transformation?
Consider the long term. Focus on systems and interconnectivity. Above all, prioritize data, as it is integral to your digital transformation strategy.
