集群环境下基于内存的高性能数据管理与分析

项目名称： 集群环境下基于内存的高性能数据管理与分析

项目编号： No.61332006

项目类型： 重点项目

立项/批准年度： 2014

项目学科： 自动化技术、计算机技术

项目作者： 周傲英

作者单位： 华东师范大学

项目金额： 300万元

中文摘要： 随着市场竞争的加剧和企业信息化程度的提高，传统商务智能系统难以满足当前决策时效性的要求，实时商务智能已成为许多企业追求的目标，硬件和体系结构的发展为之提供了技术条件。本项目旨在研究集群环境下基于内存的高性能数据管理与分析技术，探索符合应用需求、充分发挥硬件效能的决策类大数据处理技术，为实现实时商务智能奠定基础。重点研究：1）非一致访问内存环境下的数据高效存储，包括列式密集存储、内存感知布局和压缩感知处理；2）大规模异构计算资源的充分利用，包括函数至核有向无环工作流式和处理器结合迭代式并行处理，以及会话调度策略和计算优先负载平衡；3）内存系统可靠性保障，包括基于世系的内存数据集容错、热备进程的任务快速恢复等；4）针对特定应用的基本算子和执行计划优化。本项目的研究符合现实应用需求和技术发展趋势，具有广阔的应用前景和学术价值。申请人在数据管理方面积累充分，研究方案可行，能保证本项目顺利完成。

中文关键词： 数据管理;内存数据容错;大规模并行处理;非一致内存访问;商务智能

英文摘要： With the increasing intension of market competition and the continuously development of enterprise informatization, it is hard for the conventional business intelligence systems to meet the requirements about the timely decision. The real-time business intelligence is then becoming a goal which more and more enterprises are pursuing. The great advance on computer hardware and architecture offers technical background for the real-time business intelligence. The project mainly will focus on the high performance data management and analytics based on in-memory cluster computing, which is expected to set a solid foundation for the efficient processing of the decision-making big data, taking full advantage of the progress on hardware, and taking the real life application into consideration. The major research topics are as followings. 1) High performance data storage with non-uniform access memory, including column-oriented dense packing storage, memory-sensitive data placement, and compression-aware data processing. 2) Fully using the heterogeneous computation resources, including DAG workflow parallel processing based on function-at-a-core strategy, interactive parallel processing based on processor-affined scheduling, session scheduling strategy, and load balance based on computation priority. 3) High availability of the building systems, such as fault-tolerant data set based on lineage, task recovery based on standby process. 4) The optimization on primary operators and execution plan for the specific applications, to achieve the ad hoc human real-time interactive analysis. The planned research conforms to the current applications and the development of the related technologies. It is of broad interests to the participants from academic and industries. The applicants have profound technical accumulation on the related areas, and have explored preliminarily on the proposed research plan, which ensure this project to be accomplished successfully.

英文关键词： In-memory computing;fault-tolerant for memory data;parallel processing;NUMA;business intellegence