OpenSBT provides a modular and extandable code base to facilitate search-based testing of automated driving systems. It provides interfaces to integrate search algorithms, fitness/criticality functions and simulation environments in a modular way. It allows to visualize testing results and analyse the critical behaviour of the tested system.
An introductory video of OpenSBT can be found here: https://www.youtube.com/watch?v=qi_CTTzrk5s.
OpenSBT builds upon Pymoo 0.6.0 and extends internal optimization related models, such as Problem, Result, to tailor heuristic search algorithms for testing ADS systems.
OpenSBT requires Python to be installed and its compatibality has been tested with Python 3.8. OpenSBT can be run as a standalone application or can be imported as a library.
Installation instructions are available in the following jupyter notebook.
You can find several tutorials as jupyter notebooks which explain step-by-step of how to use OpenSBT. In these tutorials, we have integrated:
- A simplified SUT simulated in very simplistic simulator (linear motion planning, no GPU required)
- A real-world FMI-based AEB agent developed in the fortiss Mobility lab which is simulated in CARLA using the simulator adapter CARLA Runner Extension.
Note: We have also implemented a simulator adapter for testing Simulink-based SUTs in Prescan.
OpenSBT produces several result artefacts. All artefacts are written into the results folder in a folder named as the problem name. OpenSBT generates the following outputs:
| Type | Description | Example |
|---|---|---|
| Design Space Plot | Visualization of all evaluated test cases in the input space + of predicted critical regions using the decision tree algorithm, pairwise. Constraints of derived regions are stored in CSV file bounds_regions.csv and the learned tree in tree.pdf |  |
| Scenario 2D Visualization | Visualization of traces of the ego vehicle and adversaries in a two-dimensional GIF animation |  |
| Objective Space Plot | Visualization of fitness values of evaluated test cases, pairwise. |  |
| All Testcases | CSV file of all test inputs of all evaluated testcases | all_testcases.csv |
| All Critical Testcases | CSV file of all critical test inputs of all evaluated testcases | all_critical_testcases.csv |
| Calculation Properties | CSV file of all experiment configuration parameters (e.g. algorithm parameters, such as population size, number iterations; search space, fitness function etc..). | calculation_properties.csv |
| Evaluation Results | CSV file containing performance values of the algorithm, e.g., number critical test cases found in relation to all evaluations, execution time. | summary_results.csv |
OpenSBT has been already applied:
- For a replication experiment to replicate the results of a surrogate assisted testing technique, s. here.
- In an industrial case study to validate a systems behaviour for different operating scenarios, s. here and here: https://doi.org/10.1007/978-3-031-46002-9_15
- For the development and benchmarking an ML-based search algorithm, s. here.
If you like to contribute please contact one of the developers listed below or create a pull request. If you face any issue with OpenSBT feel free to create an issue or contact the developers.
OpenSBT has been developed by Lev Sorokin, Tiziano Munaro and Damir Safin within the FOCETA Project. Special thanks go to Brian Hsuan-Cheng Liao and Adam Molin from DENSO AUTOMOTIVE Deutschland GmbH for their valuable feedback and evaluation of OpenSBT on the AVP Case Study in the Prescan simulator.
If you use or extend OpenSBT please cite our framework. Here is an example BibTeX entry:
@inproceedings{10.1145/3639478.3640027,
author = {Sorokin, Lev and Munaro, Tiziano and Safin, Damir and Liao, Brian Hsuan-Cheng and Molin, Adam},
title = {OpenSBT: A Modular Framework for Search-based Testing of Automated Driving Systems},
year = {2024},
isbn = {9798400705021},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/3639478.3640027},
doi = {10.1145/3639478.3640027},
booktitle = {Proceedings of the 2024 IEEE/ACM 46th International Conference on Software Engineering: Companion Proceedings},
pages = {94–98},
numpages = {5},
keywords = {search-based software testing, metaheuristics, scenario-based testing, autonomous driving, automated driving},
location = {, Lisbon, Portugal, },
series = {ICSE-Companion '24}
}OpenSBT is licensed under the Apache License, Version 2.0.
