[논문]An Efficient Approach to Consolidating Job Schedulers in Traditional Independent Scientific Workflows

Kong, Byungyun; Ryu, Geonmo; Bae, Sangwook; Noh, Seo-Young; Yoon, Heejun

doi:10.3390/app10041455

[해외논문] An Efficient Approach to Consolidating Job Schedulers in Traditional Independent Scientific Workflows 원문보기

Applied sciences, v.10 no.4, 2020년, pp.1455 -

Kong, Byungyun (Global Science Experimental Data Hub Center, Korea Institute of Science and Technology Information, 245 Daehak-ro, Yuseong-gu, Daejeon 34141, Korea) , Ryu, Geonmo (Global Science Experimental Data Hub Center, Korea Institute of Science and Technology Information, 245 Daehak-ro, Yuseong-gu, Daejeon 34141, Korea) , Bae, Sangwook (Global Science Experimental Data Hub Center, Korea Institute of Science and Technology Information, 245 Daehak-ro, Yuseong-gu, Daejeon 34141, Korea) , Noh, Seo-Young (Department of Computer Science, Chungbuk National University, Chungdae-ro 1, Seowon-Gu, Cheongju, Chungbuk 28644, Korea) , Yoon, Heejun (Global Science Experimental Data Hub Center, Korea Institute of Science and Technology Information, 245 Daehak-ro, Yuseong-gu, Daejeon 34141, Korea)

Abstract ▼ AI-Helper

The current research paradigm is one of data-driven research. Researchers are beginning to deploy computer facilities to produce and analyze large amounts of data. As requirements for computing power grow, data processing in traditional workstations is always under pressure for efficient resource management. In such an environment, a tremendous amount of data is being processed using parallel computing for efficient and effective research results. HTCondor, as an example, provides computing power for data analysis for researchers. Although such a system works well in a traditional computing cluster environment, we need an efficient methodology to meet the ever-increasing demands of computing using limited resources. In this paper, we propose an approach to integrating clusters that can share their computing power on the basis of a priority policy. Our approach makes it possible to share worker nodes while maintaining the resources allocated to each group. In addition, we have utilized the historical data of user usage in order to analyze problems that have occurred during job execution due to resource sharing and the actual operating results. Our findings can provide a reasonable guideline for limited computing powers shared by multiple scientific groups.

참고문헌 (27)

Hey, A., Tansley, S., and Tolle, K. (2009). The Fourth Paradigm: Data-Intensive Scientific Discovery, Microsoft Research.
Agrawal Perspective: Materials informatics and big data: Realization of the ‘fourth paradigm’ of science in materials science APL Mater. 2016 10.1063/1.4946894 4 053208

상세보기
Trajanov, D., Zdraveski, V., Stojanov, R., and Kocarev, L. (2018, January 12-14). Dark Data in Internet of Things (IoT): Challenges and Opportunities. Proceedings of the 7th Small Systems Simulation Symposium, Niš, Serbia.
10.1109/HPCC.2012.83 Oliveira, S.F., Fürlinger, K., and Kranzlmüller, D. (2012, January 25-27). Trends in computation, communication and storage and the consequences for data-intensive science. Proceedings of the 2012 IEEE 14th International Conference on High Performance Computing and Communication & 2012 IEEE 9th International Conference on Embedded Software and Systems, Liverpool, UK.
Cukier The rise of big data Foreign Aff. 2013 92 29
Hershey Data Analytics Implications IEEE Potentials 2018 10.1109/MPOT.2018.2824361 37 10

상세보기
Livny Mechanisms for High Throughput Computing SPEEDUP J. 1997 11 36
Thain Distributed computing in practice: The condor experience Concurr. Pract. Exp. 2005 10.1002/cpe.938 17 323

상세보기
10.7551/mitpress/1556.001.0001 Sterling, T. (2002). Condor-A Distributed Job Scheduler. Beowulf Cluster Computing with Linux, The MIT Press.
(2019, December 29). HTCondor, High Throughput Computing. Available online: https://research.cs.wisc.edu/htcondor.
Henderson, R., and Tweten, D. (1996). Portable Batch System: External Reference Specification, Ames Research Center. Technical Report.
(2019, December 29). PBS. Available online: http://pbspro.org.
Jette, M.A., Yoo, A.B., and Grondona, M. (2003, January 24). SLURM: Simple Linux Utility for Resource Management. Proceedings of the Job Scheduling Strategies for Parallel Processing (JSSPP), Seattle, WA, USA. Lecture Notes in Computer Science.
(2019, December 29). Slurm. Available online: https://slurm.schedmd.com.
Ruda The extension of TORQUE scheduler allowing the use of planning and optimization algorithms in Grids Comput. Sci. 2012 10.7494/csci.2012.13.2.5 13 5

상세보기
(2019, December 29). Torque. Available online: https://www.adaptivecomputing.com/products/open-source/torque.
10.22323/1.133.0013 Kim, B., Ahn, S., Khan, T., and Jnag, H. (2011, January 19-25). Introduction of GSDC project and activities. Proceedings of the International Symposium on Grids and Clouds and the Open Grid Forum, Taipei, Taiwan.
Cho, K. Personal communication.
Cho, K. (2019). 8th GSDC Resources Review Boards, KISTI.
Albrecht A Roadmap for HEP Software and Computing R&D for the 2020s Comput. Softw. Big Sci. 2019 10.1007/s41781-018-0018-8 3 7

상세보기
Kong Multi-experimental Support Through HTCondor Scheduling Policy Integrated Pool Configurationr Platf. Technol. Lett. 2019 6 17
Ahn Experience on HTCondor batch system for HEP and other research fields at KISTI-GSDC J. Phys. Conf. Ser. 2017 10.1088/1742-6596/898/8/082013 898 182938

상세보기
(2019, December 29). HTCondor Manual. Available online: https://htcondor.readthedocs.io/en/latest/classad-attributes/job-classad-attributes.html.
Raman, R., Livny, M., and Solomon, M. (1998, January 28-31). Matchmaking: Distributed Resource Management for High Throughput Computing. Proceedings of the Seventh IEEE International Symposium on High Performance Distributed Computing, Chicago, IL, USA.
Ryu A study on performance degradation when using Singularity container of HTCondor scheduler Platf. Technol. Lett. 2019 6 1
10.1371/journal.pone.0177459 Kurtzer, G.M., Sochat, V., and Bauer, M.W. (2017). Singularity: Scientific containers for mobility of compute. PLoS ONE, 12.

상세보기
(2019, December 29). Singularity, Available online: https://singularity.lbl.gov.

LOADING...

활용도 분석정보

상세보기

다운로드

내보내기

활용도 Top5 논문

해당 논문의 주제분야에서 활용도가 높은 상위 5개 콘텐츠를 보여줍니다.
더보기 버튼을 클릭하시면 더 많은 관련자료를 살펴볼 수 있습니다.

원문 보기

AccessON 원문보기

원문 URL 링크

DOI : 10.3390/app10041455 [무료]
Molecular Diversity Preservation International : 저널
AccessON : 저널

*원문 PDF 파일 및 링크정보가 존재하지 않을 경우 KISTI DDS 시스템에서 제공하는 원문복사서비스를 사용할 수 있습니다.

오픈액세스(OA) 유형

GOLD

오픈액세스 학술지에 출판된 논문

저작권 관리 안내

내보내기 메뉴

내보내기 구분

파일저장
인쇄
메일전송

구성항목

기본정보
상세정보

관리번호, 논문명, 저널/프로시딩명, 저자 , 발행년, 권, 호, 시작페이지, 끝페이지, 발행기관

저장형식

Text(ASCII format)
Excel format
RefWorks Direct Export
RIS format (for Reference Manager, ProCite, EndNote), Scholar's Aids, Mendeley

메일정보

받는사람 (필수): @
보내는사람 (선택): @
제목
내용: KISTI 검색결과 이메일 서비스

안내

총 건의 자료가 검색되었습니다.

다운받으실 자료의 인덱스를 입력하세요. (1-10,000)

검색결과의 순서대로 최대 10,000건 까지 다운로드가 가능합니다.

데이타가 많을 경우 속도가 느려질 수 있습니다.(최대 2~3분 소요)

다운로드 파일은 UTF-8 형태로 저장됩니다.
파일의 내용이 제대로 보이지 않을실 때는 웹브라우저 상단의 보기 -> 인코딩 -> 자동선택 여부를 확인하십시오.

Text(ASCII format)
Excel format

표제어: PCR

동의어: Packet Collision Rate

용어 설명 출처 목록 (6)

용어 설명: PCR은 세균 특이성이 있는 primer를 이용하여 적은 수의 세균이 있을지라도 쉽게 검출할 수 있는 유용한 방법이며, 이를 이용하여 구강 내 치면세균막이나 타액에서 직접 세균을 검출할 수 있게 되었다[8].

AI-Helper ※ AI-Helper는 을 사용합니다.

AI-Helper

안녕하세요, AI-Helper입니다. 좌측 "선택된 텍스트"에서 텍스트를 선택하여 요약, 번역, 용어설명을 실행하세요.
※ AI-Helper는 부적절한 답변을 할 수 있습니다.

연합인증