[논문]Rate Adaptation with Q-Learning in CSMA/CA Wireless Networks

Cho, Soohyun

doi:10.3745/jips.03.0148

Rate Adaptation with Q-Learning in CSMA/CA Wireless Networks 원문보기

Journal of information processing systems, v.16 no.5, 2020년, pp.1048 - 1063

Cho, Soohyun (Dept. of General Studies, Hongik University)

Abstract ▼ AI-Helper

In this study, we propose a reinforcement learning agent to control the data transmission rates of nodes in carrier sensing multiple access with collision avoidance (CSMA/CA)-based wireless networks. We design a reinforcement learning (RL) agent, based on Q-learning. The agent learns the environment using the timeout events of packets, which are locally available in data sending nodes. The agent selects actions to control the data transmission rates of nodes that adjust the modulation and coding scheme (MCS) levels of the data packets to utilize the available bandwidth in dynamically changing channel conditions effectively. We use the ns3-gym framework to simulate RL and investigate the effects of the parameters of Q-learning on the performance of the RL agent. The simulation results indicate that the proposed RL agent adequately adjusts the MCS levels according to the changes in the network, and achieves a high throughput comparable to those of the existing data transmission rate adaptation schemes such as Minstrel.

주제어

표/그림 (18)

그림 Fig. 1. The ns3-gym framework with ns-3 network simulator.
표 Table 1. Modulation and coding schemes of IEEE 802.11a
그림 Fig. 2. Changes in application throughputs achieved by the receiver.
표 Table 2. Parameters used for simulations
표 Table 2. (Continued)
그림 Fig. 3. Changes in the MCS levels used by the sender in the stationary case.
그림 Fig. 4. Changes in the achieved throughputs by the receiver in the stationary case.
그림 Fig. 5. Changes in actions, rewards, and states in the stationary case.
그림 Fig. 6. Changes in actions, rewards, and states in the non-stationary case.
그림 Fig. 7. Changes in the MCS levels used by the sender in the non-stationary case.
그림 Fig. 8. Changes in achieved throughputs by the receiver in the non-stationary case.
그림 Fig. 9. Changes in achieved throughputs by the receiver with different learning rates.
그림 Fig. 10. Changes in achieved throughputs by the receiver with different discount rates.
그림 Fig. 11. Changes in achieved throughputs by the receiver with different exploration decay rates.
그림 Fig. 12. Changes in throughputs achieved by the receiver for 10 episodes.
그림 Fig. 13. Changes in MCS levels versus the distance between the sender and receiver.
그림 Fig. 14. Changes in the achieved throughput versus the distance between the sender and receiver.
그림 Fig. 15. Box plots of the achieved throughputs by the receiver for 10 consecutive episodes.

AI 본문요약
AI-Helper

* AI 자동 식별 결과로 적합하지 않은 문장이 있을 수 있으니, 이용에 유의하시기 바랍니다.

제안 방법

In this study, we introduced an RL agent based on Q-learning to control the data transmission rates of CSMA/CA nodes. The RL agent learned the environment using locally available information in the nodes and controlled the MCS levels of the data packets.
In this study, we use the ns3-gym framework to develop and evaluate the proposed RL agent in CSMA/CA wireless network scenarios.
, MCS level) for the state from the Q-table, and the ns3-gym framework applies the selected action to the sender node as the MCS level, which will be used by the sender node during the next time step. The size of the time step is set to 1 ms by considering the simulated network topology and network parameters, such as the frequency (5.18 GHz) and bandwidth (20 MHz), used for the simulations in this study.
To investigate the effect of the exploration decay rate on the performance of the RL agent further, we perform the simulations with the non-stationary case scenario. We run 10 consecutive episodes with the three different exploration decay rates.

참고문헌 (21)

H. Kim and S. Lee, "Document summarization model based on general context in RNN," Journal of Information Processing Systems, vol. 15, no. 6, pp. 1378-1391, 2019.

원문보기 상세보기
M. J. J. Ghrabat, G. Ma, I. Y. Maolood, S. S. Alresheedi, and Z. A. Abduljabbar, "An effective image retrieval based on optimized genetic algorithm utilized a novel SVM-based convolutional neural network classifier," Human-centric Computing and Information Sciences, vol. 9, article no. 31, 2019.
R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction, 2nd ed. Cambridge, MA: MIT Press, 2018.
M. A. Rahman, Y. D. Lee, and I. Koo, "An efficient transmission mode selection based on reinforcement learning for cooperative cognitive radio networks," Human-centric Computing and Information Sciences, vol. 6, article no. 2, 2016.

상세보기
Y. Bengio, Learning Deep Architectures for AI. Hanover, MA: Now Publishers Inc., 2009.
C. Zhang, P. Patras, and H. Haddadi, "Deep learning in mobile and wireless networking: a survey," IEEE Communications Surveys & Tutorials, vol. 21, no. 3, pp. 2224-2287, 2019.

상세보기
V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, et al., "Human-level control through deep reinforcement learning," Nature, vol. 518, pp. 529-533, 2015.
M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling, "The arcade learning environment: an evaluation platform for general agents," Journal of Artificial Intelligence Research, vol. 47, pp. 253-279, 2013.

상세보기
C. J. Watkins and P. Dayan, "Q-learning," Machine Learning, vol. 8, pp. 279-292, 1992.

상세보기
Y. Sun and W. Tan, "A trust-aware task allocation method using deep q-learning for uncertain mobile crowdsourcing," Human-centric Computing and Information Sciences, vol. 9, article no. 25, 2019.

상세보기
G. A. Preethi and C. Chandrasekar, "Seamless mobility of heterogeneous networks based on Markov decision process," Journal of Information Processing Systems, vol. 11, no. 4, pp. 616-629, 2015.

원문보기 상세보기
OpenAI Gym [Online]. Available: https://gym.openai.com.
P. Gawlowicz and A. Zubow, "Ns-3 meets OpenAI Gym: the playground for machine learning in networking research," in Proceedings of the 22nd International ACM Conference on Modeling, Analysis and Simulation of Wireless and Mobile Systems, Miami Beach, FL, 2019, pp. 113-120).
ns-3 is a discrete-event network simulator [Online]. Available: https://www.nsnam.org/.
IEEE Standard for Telecommunications and Information Exchange Between Systems - LAN/MAN Specific Requirements - Part 11: Wireless Medium Access Control (MAC) and physical layer (PHY) specifications: High Speed Physical Layer in the 5 GHz band, IEEE Std 802.11a-1999(R2003), 2003.
"Rate Adaptation for 802.11 Wireless Networks: Minstrel," 2010 [Online]. Available: http://blog.cerowrt.org/papers/minstrel-sigcomm-final.pdf.
J. Kim, S. Kim, S. Choi, and D. Qiao, "CARA: collision-aware rate adaptation for IEEE 802.11 WLANs," in Proceedings of the 25th IEEE International Conference on Computer Communications (INFOCOM), Barcelona, Spain, 2006.
IEEE Standard for Information technology - Telecommunications and information exchange between systems Local and metropolitan area networks - Specific requirements - Part 11: Wireless LAN Medium Access Control (MAC) and Physical Layer (PHY) Specifications, IEEE Std 802.11-2016, 2016.
S. Cho, "SINR-based MCS level adaptation in CSMA/CA wireless networks to embrace IoT devices," Symmetry, vol. 9, article no. 236, 2017.
D. Xia, J. Hart, and Q. Fu, "Evaluation of the Minstrel rate adaptation algorithm in IEEE 802.11g WLANs," in Proceedings of 2013 IEEE International Conference on Communications (ICC), Budapest, Hungary, 2013, pp. 2223-2228.
A. Geron, Hands-on Machine Learning with Scikit-Learn, Keras, and TensorFlow: Concepts, Tools, and Techniques to Build Intelligent Systems. Sebastopol, CA: O'Reilly Media, 2017.

내보내기 구분	파일저장 인쇄 메일전송
구성항목	기본정보 상세정보 관리번호, 논문명, 저널/프로시딩명, 저자 , 발행년, 권, 호, 시작페이지, 끝페이지, 발행기관 관리번호, 논문명, 대등논문명, 저자 , 저널/프로시딩명, 발행기관, 발행년, 발행언어, 권, 호, 시작페이지, 끝페이지, ISBN, ISSN, 주제분야, 키워드, 초록(한글), 초록(영문), 저자(소속기관)
저장형식	Text(ASCII format) Excel format RefWorks Direct Export RIS format (for Reference Manager, ProCite, EndNote), Scholar's Aids, Mendeley
메일정보	받는사람 (필수) @ 보내는사람 (선택) @ 제목 내용 KISTI 검색결과 이메일 서비스
안내	총 건의 자료가 검색되었습니다. 다운받으실 자료의 인덱스를 입력하세요. (1-10,000) 검색결과의 순서대로 최대 10,000건 까지 다운로드가 가능합니다. 데이타가 많을 경우 속도가 느려질 수 있습니다.(최대 2~3분 소요) 다운로드 파일은 UTF-8 형태로 저장됩니다. 파일의 내용이 제대로 보이지 않을실 때는 웹브라우저 상단의 보기 -> 인코딩 -> 자동선택 여부를 확인하십시오. ~ Text(ASCII format) Excel format

연합인증