A Benchmark Suite for Systematically Evaluating Reasoning Shortcuts

Web site for the "RSBench: A Benchmark Suite for Systematically Evaluating Reasoning Shortcuts" paper made with Jekyll and hosted with GitHub Pages, NeurIPS 2024.

Apart from MathJax for LaTeX formula rendering and the dark-light mode toggle, the website is completely static and does not include JavaScript.

Github Pages deployment

If you do not have it yet, install Ruby. To install the latest version you can call:

make install-ruby

Install the Ruby gems with Bundler:

bundle install

or alternatively:

make install

For other functionalities you can check out the Makefile rules by typing:

make help

Codebase

The codebase for generating the datasets, evaluating them, and counting the Reasoning Shortcuts is available at the following link: GitHub repository.