Reinforcement learning (RL) can automatically learn a better policy through a trial-and-error paradigm and has been adopted to revolutionize and optimize traditional trafic signal control systems that are usually based on handcrafted methods. However, most existing RL-based models are either based on a single scenario or multiple independent scenarios, where each scenario has a separate simulation environment with predefined road network topology and trafic signal settings. These models implement training and testing in the same scenario, thus being strictly tied up with the specific setting and sacrificing model generalization heavily. While a few recent models could be trained by multiple scenarios, they require a huge amount of manual labor to label the intersection structure, hindering the model's generalization. In this work, we aim at a general framework that could eliminate heavy labeling and model a variety of scenarios simultaneously. To this end, we propose a general Scenario-Agnostic (GESA) reinforcement learning framework for trafic signal control with: (1) A general plug-in module to map all diferent intersections into a unified structure, freeing us from the heavy manual labor to specify the structure of intersections; (2) A unified state and action space design to keep the model input and output consistently structured; (3) A large-scale co-training with multiple scenarios, leading to a generic trafic signal control algorithm. GESA can automatically handle various structured intersections from various cities without human labeling, and it co-trains a generalist agent to control trafic signals for multiple cities together, which also demonstrates superior transferability in zero-shot settings. In experiments, we demonstrate our algorithm as the first one that can be co-trained with seven diferent scenarios without manual annotation and gets 13.27% higher rewards than baselines. When dealing with a new scenario, our model can still achieve 9.39% higher rewards. The code, scenarios, and demos are available here. The full paper is available at [1].
Reinforcement learning (RL) [
Moreover, current RL-based TSC methods are trained with several pre-defined and fixed scenarios, whereas they cannot gain generalization capability without labeling, which limits the application of RL-based methods in the real world. These methods can exploit the various trafic lfows generated by the simulator to make the model efective in training scenarios, but finding a low-cost universal method with promising transferability meanwhile is still a research gap. As a result, the existing methods still face tremendous challenges in jumping out from the simulation and implementing them in real cities. This is known as sim2real challenge. The challenges mainly come from the wide gap between the real complex cities and the simplified simulation systems. In the real world, the intersection structure could be rather versatile in terms of diferent settings of approaches (i.e., north, south, east, west), movements (i.e., left, right, through), and lanes (e.g., two through lanes, one right-through lane). As shown in Fig. 1(a), an intersection could have a diferent amount of approaches. Within an approach, there can be diferent combinations of movements; A lane could also combine diferent movements. However, most of the existing methods only consider a standard simulation intersection with four approaches and three lanes (right, through, and left) within each approach. This largely limits the model generalization.
To conclude, a qualified TSC approach needs high generalization and efectiveness: it should handle various intersections and be able to transfer to other unseen targets easily and with low cost. In this paper, we aim to answer three questions: (1) How do we co-train an RL with multiscenarios without labeling, given the diverse intersection structures? (2) Will multi-scenario co-training improve the TSC? If yes, why? And the more scenarios, the better? (3)
lean ltaen t h feL igR
Right lane Through lane Through lane Through lane Right-Through-Left
!!
Left lane Through lane Through lane Through lane South (a1) Intersection of three approaches, (a2) Intersection of four approaches, with 2-4 entering lanes, and 2 with 1 entering lane of right-throughmovements on each approach left movement on each approach ! ht-Tohuroguhgh RigThrLeft ✅
RL Model ✅
… (b) Single-scenario Training v.s. Multiscenario Training
Training scenario Adapting scenario
Does the co-trained RL model still perform well in the new scenario?
To narrow the sim2real gap significantly and get more
ready to be deployed in real cities, in this paper, we provide a
GEneral Scenario-Agnostic (GESA) reinforcement learning
framework for the TSC task. To our best knowledge, GESA
is the first work that pursues high generability and co-trains
multiple scenarios without labels: it automatically handles
various scenarios; the reinforcement learning is designed
accordingly to achieve generalization; it is co-trained with
multiple scenarios simultaneously and demonstrates high
transferability. Specifically, to co-train in multiple
scenarios with various intersections, the vectors with approach
spatial information are employed to map shape-odd and
complex intersections into the standard intersection. Then,
the mapped intersections are used to generate the
characteristic information of each trafic movement and the phase
of the trafic lights in a specific order. Finally, we extend
the original FRAP [
The contributions are summarized in three-fold: (1) We present a general plug-in module to map the intersections into a unified structure, freeing us from the heavy manual labeling work to specify the intersection structure and enabling large-scale co-training under multiple diferent scenarios. (2) Accordingly, we design a unified state and action space to keep the model input and output structure consistent for more general capabilities. Moreover, the GESA can adapt to various unseen scenarios and achieve promising performance without re-training. (3) We build two realworld scenarios using the real city road map and the real trafic dynamics, together with five public scenarios, where we co-train and validate the GESA with prudent experiments. All these lead us closer to the ultimate goal: to implement RL-based TSC in real cities.