硕士学位论文（工程硕士）作业管理系统中基于数据感知的调度子系统 的设计与实现THE DESIGN AND IMPLEMENTATION OFDATA-AWARE-BASED SCHEDULING SUBSYSTEM IN JOB MANAGEMENT SYSTEM李晓婕哈尔滨工业大学哈尔滨工业大学2012 年 6 月国内图书分类号：TP311国际图书分类号：621.3学校代码：10213密级：公开工程硕士学位论文作业管理系统中基于数据感知的调度子系统 的设计与实现硕 士 研 究 生：李晓婕导师：苏小红教授副导师：孙宏伟高级工程师申 请 学 位：工程硕士学科：软件工程所 在 单 位：软件学院答 辩 日 期： 2012 年 6 月授予学位单位：哈尔滨工业大学Classified Index: TP311 U.D.C.: 621.3Dissertation for the Masters Degree in EngineeringTHE DESIGN AND IMPLEMENTATION OFDATA-AWARE-BASED SCHEDULING SUBSYSTEM IN JOB MANAGEMENT SYSTEMCandidate:Supervisor:Associate Supervisor:Academic Degree Applied for:Speciality:Affiliation:Date of Defence:Degree-Conferring-Institution:Li XiaojieProf. Su XiaohongSenior Engineer Sun HongweiMaster of EngineeringSoftware EngineeringSchool of SoftwareJune, 2012Harbin Institute of Technology哈尔滨工业大学工程硕士学位论文摘要由于位于多个物理位置上的多集群环境的出现，和在这种环境下对数据访问 的限制和对数据管理的缺乏，导致了这种环境下缺乏对数据敏感型作业的有效管理。本文所研究的内容是开发能够与 LSF 作业管理系统协作的调度子系统，管理多集群中的数据资源并能够根据集群中的数据资源高效和准确进行作业调度。本论文主要是实现基于数据感知的调度，使作业在提交到系统后，系统能够根据作业需要的数据和系统所掌握的集群中的数据信息来将作业派遣到集群中合适的计算结点上执行。为实现基于数据感知调度，本论文还完成了对集群中数据资源的管理。本论文设计和实现了数据集管理模块、storage 管理模块、缓存管理模块、配置管理模块、数据传输模块、远端数据管理模块和调度模块。在数据集管理模块和 storage 管理模块中，分别实现了对集群中数据集和 storage 的管理功能；在缓存管理模块中，实现了对集群间传输的数据的缓存功能，并维护缓存数据的生命周期；在配置管理模块中，实现了根据用户配置的信息来进行系统信息初始化功能；在数据传输模块中，实现了在集群间传输数据的功能；在远端数据管理模块中，实现了集群与其他集群间共享数据信息的功能；在调度模块中，实现了根据数据信息来制定调度决策的功能。通过以上模块之间的交互，以及系统与作业管理系统的协作，实现了基于数据感知的调度，使数据敏感型作业能够完全靠系统内部的管理和协调而成功运行在计算结点上。通过本系统与 LSF 作业管理系统的协作，用户能够更专注于作业的逻辑而不需关心数据的分布，使得数据敏感型作业的整个周期都能够通过系统内部的协调和计算顺利的执行，而不再需要人工干预的过程，同时还能够为用户提供集群中的数据信息和存储信息。总体来说，实现了针对数据敏感型作业的调度，透明化了用户对数据的操作，提高了作业执行效率，提供了更完善的数据管理功能。关键字：多集群环境；数据感知；作业调度；缓存管理；数据敏感型作业-I-哈尔滨工业大学工程硕士学位论文AbstractAs the appearance of the multi-cluster environment which located in multiple physical locations, and the data access restrictions and lack of data management, Thiscondition leads to a lack of effective management of data sensitive jobs in thisenvironment. This research is to develop a scheduling subsystem which can workswith LSF job management system, to manage data resources in multiple clusterenvironment and support efficient and accurate job scheduling.This paper mainly implements data-aware scheduling, ensuring the system candispatch the job to the appropriate computing node according to the data job neededand the data information the system knows after job submitted to the cluster. Toachieve the data-aware-based scheduling, this paper has also implemented dataresources management in the cluster.This paper designs and implements the dataset management module, the storagemanagement module, the cache management module, the configuration managementmodule, the data transmission module, remote data management module andscheduling module. The dataset management module and storage management moduleimplement the management to the dataset and storage separately. The cachemanagement module implements the cache function for the data transferred betweenthe clusters and maintain the life cycle of the cached data. The configurationmanagement module implements information initialization function based on userprofile configuration. The data transfer module implements the data transfer betweenclusters. The remote data management module implements the data informationsharing between clusters. The schedule module implements making schedulingdecisions according to the data information. Through the interaction between the abovemodules, as well as the collaboration of the system and job management system,implements data aware scheduling, ensuring the data sensitive jobs can runsuccessfully on the compute host with the systems management and coordination.Through the collaboration of the system and LSF job management system, userscan more focus on the logic of the job without the concern of the distribution of thedata, making the entire lifecycle of data sensitive job is smooth through the computingand management inside the system without users operation, and provide cluster data- II -哈尔滨工业大学工程硕士学位论文and storage information for user. In general, the system implements the scheduling of the data sensitive job, and makes the operation of data transparency, and improves jobrunning efficiency, and provides more complete data management capabilities.Keywords: multi-cluster environment，data-aware，job scheduling，cache management，data sensitive job- III -哈尔滨工业大学工程硕士学位论文目录摘要IAbstract II第 1 章 绪论 11.1 课题背景11.1.1 课题来源 11.1.2 课题研究的目的与意义