Experience-Driven Congestion Control: When Multi-Path TCP Meets Deep Reinforcement Learning

In this paper, we aim to study networking problems from a whole new perspective by leveraging emerging deep learning, to develop an experience-driven approach, which enables a network or a protocol to learn the best way to control itself from its own experience (e.g., runtime statistics data), just...

Full description

Saved in:
Bibliographic Details
Published in:IEEE journal on selected areas in communications 2019-06, Vol.37 (6), p.1325-1336
Main Authors: Xu, Zhiyuan, Tang, Jian, Yin, Chengxiang, Wang, Yanzhi, Xue, Guoliang
Format: Article
Language:English
Subjects:
Active control
Algorithms
Congestion
congestion control
Control algorithms
deep learning
experience-driven control
Heuristic algorithms
Machine learning
Mathematical model
multi-path TCP
Packet switched networks
Protocols
Recurrent neural networks
Reinforcement learning
Representations
Resource management
Runtime
TCP
TCP (protocol)
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:In this paper, we aim to study networking problems from a whole new perspective by leveraging emerging deep learning, to develop an experience-driven approach, which enables a network or a protocol to learn the best way to control itself from its own experience (e.g., runtime statistics data), just as a human learns a skill. We present design, implementation and evaluation of a deep reinforcement learning (DRL)-based control framework, DRL-CC (DRL for Congestion Control), which realizes our experience-driven design philosophy on multi-path TCP (MPTCP) congestion control. DRL-CC utilizes a single (instead of multiple independent) agent to dynamically and jointly perform congestion control for all active MPTCP flows on an end host with the objective of maximizing the overall utility. The novelty of our design is to utilize a flexible recurrent neural network, LSTM, under a DRL framework for learning a representation for all active flows and dealing with their dynamics. Moreover, we, for the first time, integrate the above LSTM-based representation network into an actor-critic framework for continuous (congestion) control, which leverages the emerging deterministic policy gradient to train critic, actor, and LSTM networks in an end-to-end manner. We implemented DRL-CC based on the MPTCP implementation in the Linux kernel. The experimental results show that 1) DRL-CC consistently and significantly outperforms a few well-known MPTCP congestion control algorithms in terms of goodput without sacrificing fairness, 2) it is flexible and robust to highly-dynamic network environments with time-varying flows, and 3) it is friendly to regular TCP.
ISSN:0733-8716
1558-0008
DOI:10.1109/JSAC.2019.2904358