Enterprise data management is a monumental task. It spans data architecture and systems, integration, quality, governance, and continuous improvement. While AI assistants can help specific persona, such as data engineers and stewards, to navigate and configure the data stack, they fall far short of full automation. However, as AI becomes increasingly capable of tackling tasks that have previously resisted automation due to inherent complexities, we believe there is an imminent opportunity to target fully autonomous data estates. Currently, AI is used in different parts of the data stack, but in this paper, we argue for a paradigm shift from the use of AI in independent data component operations towards a more holistic and autonomous handling of the entire data lifecycle. Towards that end, we explore how each stage of the modern data stack can be autonomously managed by intelligent agents to build self-sufficient systems that can be used not only by human end-users, but also by AI itself. We begin by describing the mounting forces and opportunities that demand this paradigm shift, examine how agents can streamline the data lifecycle, and highlight open questions and areas where additional research is needed. We hope this work will inspire lively debate, stimulate further research, motivate collaborative approaches, and facilitate a more autonomous future for data systems.
翻译:企业数据管理是一项艰巨的任务,涵盖数据架构与系统、集成、质量、治理及持续改进等多个方面。尽管人工智能助手能够帮助特定角色(如数据工程师与数据管理员)导航和配置数据栈,但距离实现全自动化仍相去甚远。然而,随着人工智能日益能够处理那些因内在复杂性而长期难以自动化的任务,我们认为实现完全自主的数据资产体系已近在眼前。当前,人工智能已在数据栈的不同环节得到应用,但本文主张推动范式转变:从人工智能在独立数据组件操作中的应用,转向更整体化、自主化地处理完整数据生命周期。为此,我们探讨了现代数据栈的每个阶段如何通过智能体实现自主管理,从而构建不仅能供人类终端用户使用、也能为人工智能自身服务的自足系统。我们首先阐述了推动这一范式转变的迫切需求与机遇,分析了智能体如何优化数据生命周期流程,并指出亟待解决的开放性问题及需要进一步研究的领域。我们希望这项工作能激发活跃的讨论,推动深入研究,促进协作式解决方案的发展,并为数据系统迈向更高自主性的未来铺平道路。