Formal Scenario-Based Testing of Autonomous Vehicles: From Simulation to the Real World

We present a new approach to automated scenario-based testing of the safety of autonomous vehicles, especially those using advanced artificial intelligence-based components, spanning both simulation-based evaluation as well as testing in the real world. Our approach is based on formal methods, combining formal specification of scenarios and safety properties, algorithmic test case generation using formal simulation, test case selection for track testing, executing test cases on the track, and analyzing the resulting data. Experiments with a real autonomous vehicle at an industrial testing facility support our hypotheses that (i) formal simulation can be effective at identifying test cases to run on the track, and (ii) the gap between simulated and real worlds can be systematically evaluated and bridged.

[1]  Sanjit A. Seshia,et al.  Formal Analysis and Redesign of a Neural Network-Based Aircraft Taxiing System with VerifAI , 2020, CAV.

[2]  Krzysztof Czarnecki,et al.  Modes of Automated Driving System Scenario Testing: Experience Report and Recommendations , 2020 .

[3]  Krzysztof Czarnecki,et al.  GeoScenario: An Open DSL for Autonomous Driving Scenario Representation , 2019, 2019 IEEE Intelligent Vehicles Symposium (IV).

[4]  Toni Giorgino,et al.  Computing and Visualizing Dynamic Time Warping Alignments in R: The dtw Package , 2009 .

[5]  Yi Zhang,et al.  Testing Scenario Library Generation for Connected and Automated Vehicles, Part I: Methodology , 2019, IEEE Transactions on Intelligent Transportation Systems.

[6]  Q. Lu,et al.  LGSVL Simulator: A High Fidelity Simulator for Autonomous Driving , 2020, 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC).

[7]  Sanjit A. Seshia,et al.  VerifAI: A Toolkit for the Formal Design and Analysis of Artificial Intelligence-Based Systems , 2019, CAV.

[8]  Friedrich Kruber,et al.  An Unsupervised Random Forest Clustering Technique for Automatic Traffic Scenario Categorization , 2018, 2018 21st International Conference on Intelligent Transportation Systems (ITSC).

[9]  Sanjit A. Seshia,et al.  Towards Verified Artificial Intelligence , 2016, ArXiv.

[10]  Hermann Winner,et al.  PEGASUS—First Steps for the Safe Introduction of Automated Driving , 2018, Lecture Notes in Mobility.

[11]  Houssam Abbas,et al.  Safe At Any Speed: A Simulation-Based Test Harness for Autonomous Vehicles , 2017, CyPhy.

[12]  Alberto L. Sangiovanni-Vincentelli,et al.  Scenic: a language for scenario specification and scene generation , 2018, PLDI.

[13]  Lutz Eckstein,et al.  A scenario-based assessment approach for automated driving by using time series classification of human-driving behaviour , 2016, 2016 IEEE 19th International Conference on Intelligent Transportation Systems (ITSC).

[14]  Yiheng Feng,et al.  Testing Scenario Library Generation for Connected and Automated Vehicles, Part II: Case Studies , 2019, IEEE Transactions on Intelligent Transportation Systems.

[15]  Rupak Majumdar,et al.  Quantifying Conformance Using the Skorokhod Metric , 2015, CAV.

[16]  George J. Pappas,et al.  Robustness of Temporal Logic Specifications , 2006, FATES/RV.

[17]  Ron Koymans,et al.  Specifying real-time properties with metric temporal logic , 1990, Real-Time Systems.