Too Afraid to Drive: Systematic Discovery of Semantic DoS Vulnerability in Autonomous Driving Planning under Physical-World Attacks

In high-level Autonomous Driving (AD) systems, behavioral planning is in charge of making high-level driving decisions such as cruising and stopping, and thus highly securitycritical. In this work, we perform the first systematic study of semantic security vulnerabilities specific to overly-conservative AD behavioral planning behaviors, i.e., those that can cause failed or significantly-degraded mission performance, which can be critical for AD services such as robo-taxi/delivery. We call them semantic Denial-of-Service (DoS) vulnerabilities, which we envision to be most generally exposed in practical AD systems due to the tendency for conservativeness to avoid safety incidents. To achieve high practicality and realism, we assume that the attacker can only introduce seemingly-benign external physical objects to the driving environment, e.g., off-road dumped cardboard boxes. To systematically discover such vulnerabilities, we design PlanFuzz, a novel dynamic testing approach that addresses various problem-specific design challenges. Specifically, we propose and identify planning invariants as novel testing oracles, and design new input generation to systematically enforce problemspecific constraints for attacker-introduced physical objects. We also design a novel behavioral planning vulnerability distance metric to effectively guide the discovery. We evaluate PlanFuzz on 3 planning implementations from practical open-source AD systems, and find that it can effectively discover 9 previouslyunknown semantic DoS vulnerabilities without false positives. We find all our new designs necessary, as without each design, statistically significant performance drops are generally observed. We further perform exploitation case studies using simulation and real-vehicle traces. We discuss root causes and potential fixes.

[1]  Joe D. Warren,et al.  The program dependence graph and its use in optimization , 1987, TOPL.

[2]  Darrell Whitley A genetic algorithm tutorial , 1994 .

[3]  Ryan Cunningham,et al.  Automated Vulnerability Analysis: Leveraging Control Flow for Evolutionary Input Crafting , 2007, Twenty-Third Annual Computer Security Applications Conference (ACSAC 2007).

[4]  Sebastian Thrun,et al.  Apprenticeship learning for motion planning with application to parking lot navigation , 2008, 2008 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[5]  Dawson R. Engler,et al.  KLEE: Unassisted and Automatic Generation of High-Coverage Tests for Complex Systems Programs , 2008, OSDI.

[6]  Sanjiv Singh,et al.  The DARPA Urban Challenge: Autonomous Vehicles in City Traffic, George Air Force Base, Victorville, California, USA , 2009, The DARPA Urban Challenge.

[7]  William Whittaker,et al.  Autonomous Driving in Urban Environments: Boss and the Urban Challenge , 2009, The DARPA Urban Challenge.

[8]  Daniele Loiacono,et al.  Learning to overtake in TORCS using simple reinforcement learning , 2010, IEEE Congress on Evolutionary Computation.

[9]  Laurent Mounier,et al.  An Evolutionary Computing Approach for Hunting Buffer Overflow Vulnerabilities: A Case of Aiming in Dim Light , 2010, 2010 European Conference on Computer Network Defense.

[10]  Lionel C. Briand,et al.  A practical guide for using statistical tests to assess randomized algorithms in software engineering , 2011, 2011 33rd International Conference on Software Engineering (ICSE).

[11]  Steve Novick,et al.  Urban Street Design Guide , 2013 .

[12]  Jürgen Schmidhuber,et al.  Evolving large-scale neural networks for vision-based reinforcement learning , 2013, GECCO '13.

[13]  Rüdiger Dillmann,et al.  Solving Continuous POMDPs: Value Iteration with Incremental Learning of an Efficient Space Representation , 2013, ICML.

[14]  Vitaly Shmatikov,et al.  Using Frankencerts for Automated Adversarial Testing of Certificate Validation in SSL/TLS Implementations , 2014, 2014 IEEE Symposium on Security and Privacy.

[15]  David Hsu,et al.  Integrated perception and planning in the continuous space: A POMDP approach , 2014, Int. J. Robotics Res..

[16]  Dawson R. Engler,et al.  Under-Constrained Symbolic Execution: Correctness Checking for Real Code , 2016, USENIX Annual Technical Conference.

[17]  David González,et al.  A Review of Motion Planning Techniques for Automated Vehicles , 2016, IEEE Transactions on Intelligent Transportation Systems.

[18]  Emilio Frazzoli,et al.  A Survey of Motion Planning and Control Techniques for Self-Driving Urban Vehicles , 2016, IEEE Transactions on Intelligent Vehicles.

[19]  Chen Yan Can You Trust Autonomous Vehicles : Contactless Attacks against Sensors of Self-driving Vehicle , 2016 .

[20]  Christopher Krügel,et al.  Driller: Augmenting Fuzzing Through Selective Symbolic Execution , 2016, NDSS.

[21]  Salvatore J. Stolfo,et al.  NEZHA: Efficient Domain-Independent Differential Testing , 2017, 2017 IEEE Symposium on Security and Privacy (SP).

[22]  Shlomo Zilberstein,et al.  Online Decision-Making for Scalable Autonomous Systems , 2017, IJCAI.

[23]  Shinpei Kato,et al.  Open Source Integrated Planner for Autonomous Navigation in Highly Dynamic Environments , 2017, J. Robotics Mechatronics.

[24]  Yadong Mu,et al.  Deep Steering: Learning End-to-End Driving Model from Spatial and Temporal Visual Cues , 2017, ArXiv.

[25]  Junfeng Yang,et al.  DeepXplore: Automated Whitebox Testing of Deep Learning Systems , 2017, SOSP.

[26]  Yongdae Kim,et al.  Illusion and Dazzle: Adversarial Optical Channel Exploits Against Lidars for Automotive Applications , 2017, CHES.

[27]  Abhik Roychoudhury,et al.  Directed Greybox Fuzzing , 2017, CCS.

[28]  Amnon Shashua,et al.  On a Formal Model of Safe and Scalable Self-driving Cars , 2017, ArXiv.

[29]  Trent Jaeger,et al.  PtrSplit: Supporting General Pointers in Automatic Program Partitioning , 2017, CCS.

[30]  Etienne Perot,et al.  Deep Reinforcement Learning framework for Autonomous Driving , 2017, ArXiv.

[31]  Mykel J. Kochenderfer,et al.  Imitating driver behavior with generative adversarial networks , 2017, 2017 IEEE Intelligent Vehicles Symposium (IV).

[32]  Sanjit A. Seshia,et al.  Compositional Falsification of Cyber-Physical Systems with Machine Learning Components , 2017, NFM.

[33]  Herbert Bos,et al.  VUzzer: Application-aware Evolutionary Fuzzing , 2017, NDSS.

[34]  Marcelo H. Ang,et al.  Perception, Planning, Control, and Coordination for Autonomous Vehicles , 2017 .

[35]  Yue Zhao,et al.  Seeing isn't Believing: Practical Adversarial Attack Against Object Detectors , 2018 .

[36]  Sen Wang,et al.  Deep Reinforcement Learning for Autonomous Driving , 2018, ArXiv.

[37]  Yiheng Feng,et al.  Exposing Congestion Attack on Emerging Connected Vehicle based Traffic Signal Control , 2018, NDSS.

[38]  Shinpei Kato,et al.  Autoware on Board: Enabling Autonomous Vehicles with Embedded Systems , 2018, 2018 ACM/IEEE 9th International Conference on Cyber-Physical Systems (ICCPS).

[39]  Bihuan Chen,et al.  Hawkeye: Towards a Desired Directed Grey-box Fuzzer , 2018, CCS.

[40]  Andrew Ruef,et al.  Evaluating Fuzz Testing , 2018, CCS.

[41]  Meng Xu,et al.  QSYM : A Practical Concolic Execution Engine Tailored for Hybrid Fuzzing , 2018, USENIX Security Symposium.

[42]  Wei Zhan,et al.  A Fast Integrated Planning and Control Framework for Autonomous Driving via Imitation Learning , 2017, ArXiv.

[43]  Sarfraz Khurshid,et al.  DeepRoad: GAN-Based Metamorphic Testing and Input Validation Framework for Autonomous Driving Systems , 2018, 2018 33rd IEEE/ACM International Conference on Automated Software Engineering (ASE).

[44]  Georgios Fainekos,et al.  Simulation-based Adversarial Test Generation for Autonomous Vehicles with Machine Learning Components , 2018, 2018 IEEE Intelligent Vehicles Symposium (IV).

[45]  Wen-Chuan Lee,et al.  Detecting Attacks Against Robotic Vehicles: A Control Invariant Approach , 2018, CCS.

[46]  Dawn Song,et al.  Physical Adversarial Examples for Object Detectors , 2018, WOOT @ USENIX Security Symposium.

[47]  Hao Chen,et al.  Angora: Efficient Fuzzing by Principled Search , 2018, 2018 IEEE Symposium on Security and Privacy (SP).

[48]  Insup Lee,et al.  Injected and Delivered: Fabricating Implicit Control over Actuation Systems by Spoofing Inertial Sensors , 2018, USENIX Security Symposium.

[49]  Suman Jana,et al.  DeepTest: Automated Testing of Deep-Neural-Network-Driven Autonomous Cars , 2017, 2018 IEEE/ACM 40th International Conference on Software Engineering (ICSE).

[50]  Duen Horng Chau,et al.  Robust Physical Adversarial Attack on Faster R-CNN Object Detector , 2018, ECML/PKDD.

[51]  Xi Chen,et al.  Learning From Demonstration in the Wild , 2018, 2019 International Conference on Robotics and Automation (ICRA).

[52]  Xinyan Deng,et al.  RVFuzzer: Finding Input Validation Bugs in Robotic Vehicles through Control-Guided Testing , 2019, USENIX Security Symposium.

[53]  Sanjit A. Seshia,et al.  VerifAI: A Toolkit for the Formal Design and Analysis of Artificial Intelligence-Based Systems , 2019, CAV.

[54]  Abhik Roychoudhury,et al.  Coverage-Based Greybox Fuzzing as Markov Chain , 2019, IEEE Transactions on Software Engineering.

[55]  David Janz,et al.  Learning to Drive in a Day , 2018, 2019 International Conference on Robotics and Automation (ICRA).

[56]  Alberto L. Sangiovanni-Vincentelli,et al.  Scenic: a language for scenario specification and scene generation , 2019, PLDI.

[57]  Sean Sedwards,et al.  Design Space of Behaviour Planning for Autonomous Driving , 2019, ArXiv.

[58]  John M. Dolan,et al.  Learning On-Road Visual Control for Self-Driving Vehicles With Auxiliary Tasks , 2019, 2019 IEEE Winter Conference on Applications of Computer Vision (WACV).

[59]  Kevin Fu,et al.  Adversarial Sensor Attack on LiDAR-based Perception in Autonomous Driving , 2019, CCS.

[60]  Yuval Elovici,et al.  Phantom of the ADAS: Phantom Attacks on Driver-Assistance Systems , 2020, IACR Cryptol. ePrint Arch..

[61]  Junjie Shen,et al.  Drift with Devil: Security of Multi-Sensor Fusion based Localization in High-Level Autonomous Driving under GPS Spoofing (Extended Version) , 2020, USENIX Security Symposium.

[62]  Sanjit A. Seshia,et al.  Formal Analysis and Redesign of a Neural Network-Based Aircraft Taxiing System with VerifAI , 2020, CAV.

[63]  Tao Wei,et al.  Fooling Detection Alone is Not Enough: Adversarial Attack against Multiple Object Tracking , 2020, ICLR.

[64]  Saurabh Jha,et al.  ML-Driven Malware that Targets AV Safety , 2020, 2020 50th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN).

[65]  Piotr Milos,et al.  Simulation-Based Reinforcement Learning for Real-World Autonomous Driving , 2019, 2020 IEEE International Conference on Robotics and Automation (ICRA).

[66]  Emilio Coppa,et al.  WEIZZ: automatic grey-box fuzzing for structured binary formats , 2020, ISSTA.

[67]  Wei Li,et al.  DeepBillboard: Systematic Physical-World Testing of Autonomous Driving Systems , 2020, 2020 IEEE/ACM 42nd International Conference on Software Engineering (ICSE).

[68]  Georgios Fainekos,et al.  Search-based Test-CASe Generation by Monitoring Responsibility Safety Rules , 2020, 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC).

[69]  Xiangyu Zhang,et al.  Cyber-Physical Inconsistency Vulnerability Identification for Safety Checks in Robotic Vehicles , 2020, CCS.

[70]  Qi Alfred Chen,et al.  Towards Robust LiDAR-based Perception in Autonomous Driving: General Black-box Adversarial Sensor Attack and Countermeasures , 2020, USENIX Security Symposium.

[71]  Daniel J. Fremont,et al.  Formal Scenario-Based Testing of Autonomous Vehicles: From Simulation to the Real World , 2020, 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC).

[72]  Joshua Garcia,et al.  A Comprehensive Study of Autonomous Vehicle Bugs , 2020, 2020 IEEE/ACM 42nd International Conference on Software Engineering (ICSE).

[73]  Jairo Giraldo,et al.  SAVIOR: Securing Autonomous Vehicles with Robust Physical Invariants , 2020, USENIX Security Symposium.

[74]  Qi Alfred Chen,et al.  AVGuardian: Detecting and Mitigating Publish-Subscribe Overprivilege for Autonomous Vehicle Systems , 2020, 2020 IEEE European Symposium on Security and Privacy (EuroS&P).

[75]  Daniele Loiacono,et al.  Short-Term Trajectory Planning in TORCS using Deep Reinforcement Learning , 2020, 2020 IEEE Symposium Series on Computational Intelligence (SSCI).

[76]  Alexander Carballo,et al.  A Survey of Autonomous Driving: Common Practices and Emerging Technologies , 2020, IEEE Access.

[77]  Saurabh Jha,et al.  AV-FUZZER: Finding Safety Violations in Autonomous Driving Systems , 2020, 2020 IEEE 31st International Symposium on Software Reliability Engineering (ISSRE).

[78]  Bolei Zhou,et al.  Neuro-Symbolic Program Search for Autonomous Driving Decision Module Design , 2020, CoRL.

[79]  Xinyan Deng,et al.  From Control Model to Program: Investigating Robotic Aerial Vehicle Accidents with MAYDAY , 2020, USENIX Security Symposium.

[80]  Qi Alfred Chen,et al.  Dirty Road Can Attack: Security of Deep Learning based Automated Lane Centering under Physical-World Attack , 2020, USENIX Security Symposium.

[81]  Z. Berkay Celik,et al.  PGFUZZ: Policy-Guided Fuzzing for Robotic Vehicles , 2021, NDSS.

[82]  Qi Alfred Chen,et al.  WIP: Deployability Improvement, Stealthiness User Study, and Safety Impact Assessment on Real Vehicle for Dirty Road Patch Attack , 2021 .

[83]  Andrew E. Santosa,et al.  Smart Greybox Fuzzing , 2018, ArXiv.

[84]  Qi Alfred Chen,et al.  WIP: End-to-End Analysis of Adversarial Attacks to Automated Lane Centering Systems , 2021 .

[85]  Ruigang Yang,et al.  Invisible for both Camera and LiDAR: Security of Multi-Sensor Fusion based Perception in Autonomous Driving Under Physical-World Attacks , 2021, 2021 IEEE Symposium on Security and Privacy (SP).

[86]  Qi Alfred Chen,et al.  Fooling Perception via Location: A Case of Region-of-Interest Attacks on Traffic Light Detection in Autonomous Driving , 2021 .

[87]  Taxonomy and definitions for terms related to driving automation systems for on-road motor vehicles , 2022 .