[논문]Transferable traffic signal control: Reinforcement learning with graph centric state representation

Yoon, Jinwon; Ahn, Kyuree; Park, Jinkyoo; Yeo, Hwasoo

doi:10.1016/j.trc.2021.103321

[해외논문] Transferable traffic signal control: Reinforcement learning with graph centric state representation

Transportation research. Part C, Emerging technologies, v.130, 2021년, pp.103321 -

Yoon, Jinwon (Department of Civil and Environmental Engineering, Korea Advanced Institute of Science and Technology) , Ahn, Kyuree (Department of Industrial and Systems Engineering, Korea Advanced Institute of Science and Technology) , Park, Jinkyoo (Department of Industrial and Systems Engineering, Korea Advanced Institute of Science and Technology) , Yeo, Hwasoo (Department of Civil and Environmental Engineering, Korea Advanced Institute of Science and Technology)

Abstract ▼ AI-Helper

Abstract Reinforcement learning (RL) has emerged as an alternative approach for optimizing the traffic signal control system. However, there is a restricted exploration problem encountered when a signal control model is trained with a predefined demand scenario in the traffic simulation. With the restricted exploration, the model learns a policy based only on partial experiences in the search space, which yields a partially-trained policy. Partially-trained policy fails to adapt to some unexperienced (‘unexplored’, ‘never-before-seen’) dataset that have different distributions from the training dataset. Although this issue has critical effects on training a signal control model, it has not been considered in the literature. Therefore, this research aims to obtain a transferable policy to enhance the model’s applicability on unexperienced traffic states. The key idea is to represent the state as graph-structured data, and train it using a graph neural network (GNN). Since this approach enables to learn the relationship between the features resulting from the spatial structure of the intersection, it is able to transfer the already-learned knowledge of the relationship to the unexperienced data. In order to investigate the transferability, an experiment is conducted on five unexperienced test demand scenarios. For the evaluation, the performance of the proposed GNN model is compared with the conventional DQN model that is based on vector-valued state. At first, the models are trained with only a single dataset (training demand scenario). Then, they are tested with different unexperienced dataset (test demand scenarios) without additional trainings. The results show that the proposed GNN model obtains a transferable policy so that it adapts better to the unexperienced traffic states, while the conventional DQN model fails. Highlights This study identifies the restricted exploration problem encountered when training a signal control model which causes a model to obtain a partially-trained policy. The goal of this research is to obtain a transferable policy that enhances the model’s applicability on traffic states of unexperienced demand pattern using already-experienced information. The key idea consists of two parts; 1) to represent the traffic state as graph-structured data by embedding it into a graph, 2) to process the information-condensed graph by employing message-passing graph neural network (GNN). The experiment conducted on five unexperienced test demand scenarios which have different levels of traffic volumes and traveling patterns shows that the proposed GNN-based model obtains a transferable policy so that it adapts better to the unexperienced traffic states, while the conventional RL model fails.

Keyword

참고문헌 (34)

Abdoos 1580 2011 2011 14th International IEEE conference on intelligent transportation systems (ITSC) Traffic light control in non-stationary environments based on multi agent Q-learning
J. Transp. Eng. Abdulhai 129 3 278 2003 10.1061/(ASCE)0733-947X(2003)129:3(278) Reinforcement Learning for True Adaptive Traffic Signal Control

상세보기
IET Intel. Transport Syst. Arel 4 2 128 2010 10.1049/iet-its.2009.0070 Reinforcement Learning-Based Multi-Agent System for Network Traffic Signal Control

상세보기
Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., & Zaremba, W., 2016. Openai gym. arXiv preprint arXiv:1606.01540.
Casas, N., 2017. Deep deterministic policy gradient for urban traffic light control. arXiv preprint arXiv:1703.09035.
Transp. Res. Board Chang 2010 New signal control optimization policy for oversaturated arterial systems
IEEE Trans. Intell. Transp. Syst. Chu 2019 Multi-agent deep reinforcement learning for large-scale traffic signal control
Control Eng. Pract. Diakaki 10 2 183 2002 10.1016/S0967-0661(01)00121-6 A multivariable regulator approach to traffic-responsive network- wide signal control

상세보기
IEEE Trans. Intell. Transp. Syst. El-Tantawy 14 3 1140 2013 10.1109/TITS.2013.2255286 Multi-Agent Reinforcement Learning for Integrated Network of Adaptive Traffic Signal Controllers (MARLIN-ATSC): methodology and large-scale application on downtown Toronto

상세보기
J. Intell. Transp. Syst. Technol. Plann. Oper. El-Tantawy 18 3 227 2014 10.1080/15472450.2013.810991 Design of reinforcement learning parameters for seamless application of adaptive traffic signal control

상세보기
IEEE Trans. Syst. Man Cybern. B Cybern. Ernst 39 2 517 2008 10.1109/TSMCB.2008.2007630 Reinforcement Learning versus model predictive control: a comparison on a power system problem

상세보기
Gao, J., Shen, Y., Liu, J., Ito, M., & Shiratori, N., 2017. Adaptive traffic signal control: Deep reinforcement learning algorithm with experience replay and target network. arXiv preprint arXiv:1705.02755.
Transp. Res. Board Gartner 1983 OPAC: A demand-responsive strategy for traffic signal control
Genders, W., & Razavi, S., 2016. Using a deep reinforcement learning agent for traffic signal control. arXiv preprint arXiv:1611.01142.
Traff. Eng. Control Hunt 23 4 1982 The SCOOT on-line traffic signal optimisation technique
Control Transp. Sys. Henry 305 1984 10.1016/B978-0-08-029365-3.50048-1 The PRODYN real time traffic algorithm
J. Intell. Transp. Syst. Technol. Plann. Oper. Jeon 22 5 433 2018 10.1080/15472450.2017.1394192 Artificial intelligence for traffic signal control based solely on video images

상세보기
Krajzewicz 269 2010 Fundamentals of Traffic Simulation Traffic simulation with SUMO-simulation of urban mobility
IEEE/CAA J. Autom. Sin. Li 3 3 247 2016 10.1109/JAS.2016.7508798 Traffic signal timing via deep reinforcement learning

상세보기
IEEE Trans. Veh. Technol. Liang 68 2 1243 2019 10.1109/TVT.2018.2890726 A deep reinforcement learning network for traffic light cycle control

상세보기
Lin, Y., Dai, X., Li, L., & Wang, F. Y., 2018. An efficient deep reinforcement learning model for urban traffic control. arXiv preprint arXiv:1808.01876.
Medina 596 2012 2012 15th International IEEE Conference on Intelligent Transportation Systems Traffic signal control using reinforcement learning and the max-plus algorithm as a coordinating strategy
Transp. Res. Board Miller 1963 A computer control system for traffic networks
Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., & Riedmiller, M., 2013. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602.
IEEE Trans. Intell. Transp. Syst. Prashanth 12 2 412 2011 10.1109/TITS.2010.2091408 Reinforcement Learning with Function Approximation for Traffic Signal Control

상세보기
Transport. Sci. Sen 31 1 5 1997 10.1287/trsc.31.1.5 Controlled Optimization of Phases at an Intersection

상세보기
IEEE Trans. Veh. Technol. Sims 29 2 130 1980 10.1109/T-VT.1980.23833 The sydney coordinated adaptive traffic (SCAT) system philosophy and benefits

상세보기
Online Thorpe 1 1997 Vehicle Traffic Light Control Using Sarsa
Traffic Eng. Control Vincent 27 7-8 385 1986 Self-Optimizing Traffic Signal Control Using Microprocessors: The TRRL “MOVA” Strategy for Isolated Intersections
J. Adv. Transport. Wang 2018 10.1155/2018/3631489 Evaluation and application of urban traffic signal optimizing control strategy based on reinforcement learning

상세보기
Watkins, C.J.C.H., 1989. Learning from delayed rewards. Ph. D. thesis, Cambridge Univ.
Webster, F. V., 1958. Traffic signal settings, Road Research Technical Paper.
Wiering 1151 2000 Machine Learning: Proceedings of the Seventeenth International Conference (ICML'2000) Multi-agent reinforcement learning for traffic light control
AI Open Zhou 1 57 2020 10.1016/j.aiopen.2021.01.001 Graph neural networks: a review of methods and applications

상세보기

LOADING...

활용도 분석정보

상세보기

다운로드

내보내기

활용도 Top5 논문

해당 논문의 주제분야에서 활용도가 높은 상위 5개 콘텐츠를 보여줍니다.
더보기 버튼을 클릭하시면 더 많은 관련자료를 살펴볼 수 있습니다.

내보내기 구분	파일저장 인쇄 메일전송
구성항목	기본정보 상세정보 관리번호, 논문명, 저널/프로시딩명, 저자 , 발행년, 권, 호, 시작페이지, 끝페이지, 발행기관 관리번호, 논문명, 대등논문명, 저자 , 저널/프로시딩명, 발행기관, 발행년, 발행언어, 권, 호, 시작페이지, 끝페이지, ISBN, ISSN, 주제분야, 키워드, 초록(한글), 초록(영문), 저자(소속기관)
저장형식	Text(ASCII format) Excel format RefWorks Direct Export RIS format (for Reference Manager, ProCite, EndNote), Scholar's Aids, Mendeley
메일정보	받는사람 (필수) @ 보내는사람 (선택) @ 제목 내용 KISTI 검색결과 이메일 서비스
안내	총 건의 자료가 검색되었습니다. 다운받으실 자료의 인덱스를 입력하세요. (1-10,000) 검색결과의 순서대로 최대 10,000건 까지 다운로드가 가능합니다. 데이타가 많을 경우 속도가 느려질 수 있습니다.(최대 2~3분 소요) 다운로드 파일은 UTF-8 형태로 저장됩니다. 파일의 내용이 제대로 보이지 않을실 때는 웹브라우저 상단의 보기 -> 인코딩 -> 자동선택 여부를 확인하십시오. ~ Text(ASCII format) Excel format

연합인증

[해외논문] Transferable traffic signal control: Reinforcement learning with graph centric state representation

Abstract ▼ AI-Helper

Keyword

참고문헌 (34)

이 논문을 인용한 문헌

연구과제 타임라인

활용도 분석정보

활용도 Top5 논문

관련 콘텐츠

원문 URL 링크

이 논문과 함께 이용한 콘텐츠

AI-Helper ※ AI-Helper는 오픈소스 모델을 사용합니다.

선택된 텍스트

연합인증

[해외논문] Transferable traffic signal control: Reinforcement learning with graph centric state representation

Abstract ▼ AI-Helper

Keyword

참고문헌 (34)

이 논문을 인용한 문헌

연구과제 타임라인

전체(0) 논문(0) 특허(0) 보고서(0)

전체(0) 논문(0) 특허(0) 보고서(0)

활용도 분석정보

활용도 Top5 논문 더보기

관련 콘텐츠

원문 URL 링크

이 논문과 함께 이용한 콘텐츠

AI-Helper ※ AI-Helper는 오픈소스 모델을 사용합니다.

선택된 텍스트

활용도 Top5 논문