Experiments
Cockpit Package • Docs • License
To showcase the capabilities of Cockpit we performed several experiments illustrating the usefulness of our debugging tool in our paper. The code for all experiments, as well as the generated data is presented in this repository. For a discussion of those experiments please refer to our paper.
We provide an overivew of all experiments here, splitting them up in experiments that are presented in the main body of our paper and ones that are shown in the appendix.
All necessary requirements (besides Cockpit) can be found in requirements.txt
If you decide to re-run the code, you may want to delete the ~/temp/data_deepobs directory, that will be created to share the data sets among experiments, afterwards.
All experiments include some run.py file to reproduce the results. Our original results are also stored in each experiment folder in a results.zip file. The plot.py script produces the Figures shown in the paper.
| Experiment | Description |
|---|---|
| 01_benchmark | Benchmark the run time of individual instruments and configurations of Cockpit. Reproduces Figure 6 and 13-17. |
| 02_LINE | Illustrative Example of why just monitoring the loss is not enough. Reproduces Figure 1. |
| 03_scalar_deep | Concrete example why just monitoring the loss is not enough. Reproduces Figure 11. |
| 04_benchmark_memory | Benchmark the memory consumption of the histograms. Reproduces Figure 12. |
| 06_preprocessing | Study how incorrectly scaled data is represented in Cockpit. Reproduces Figure 3 and 10. |
| 07_learning_rate_selection | Grid search the learning rate and compare the alpha values vs. final performance. Reproduces Figure 5. |
| 09_layerwise | Analyzing the layerwise histogram for two different architectures. Reproduces Figure 4. |
| 10_showcase | Showcase of the full Cockpit for DeepOBS problems. Reproduces Figure 2 and 19. |
| 11_histogram2d | Benchmark the performance of different implementations for computing the 2D Histogram. Reproduces Figure 18. |
| 12_alpha_explanation | Illustrative plot explaining the Alpha Quantity. Reproduces Figure 8. |
Figure 1: Loss is not Enough - 02_LINE
Figure 2: Showcase - 10_showcase
Figure 3: Misscaled Data - 06_preprocessing
Figure 4: Layerwise Histogram - 09_layerwise
Figure 5: Tuning the Learning Rate with Alpha - 07_learning_rate_selection
Figure 6: Run Time Benchmark - 01_benchmark
Figure 7 is not included in this list, as it is a code example. The code is an adapted version of the basic and advanced examples from the examples directory and the documentation.
Figure 8: Motivational Sketch for Alpha - 12_alpha_explanation
Figure 10: Misscaled Data ImageNet - 06_preprocessing
Figure 11: Regularization Detection - 03_scalar_deep
Figure 12: Memory Consumption Histogram - 04_benchmark_memory
Figure 13 - 17: Additional Run Time Benchmarks - 01_benchmark
Figure 18: Implementation Performance 2D Histogram - 11_histogram2d
Figure 19: Additional Showcases - 10_showcase
