Skip to content

Load and model the brain data of the Algonauts Project 2023 Challenge.

License

Notifications You must be signed in to change notification settings

gifale95/algonauts_2023

Repository files navigation

The Algonauts Project 2023 Challenge - devkit tutorial

Algonauts 2023 logo

You can also run this tutorial interactively on Google Colab!

NOTE: Besides the Algonauts Project 2023 Challenge, you might still find this tutorial useful as an introduction to fMRI data and encoding models of the visual brain.

The quest to understand the nature of human intelligence and engineer more advanced forms of artificial intelligence are becoming increasingly intertwined. The Algonauts Project brings biological and machine intelligence researchers together on a common platform to exchange ideas and advance both fields. The 2023 challenge focuses on explaining neural visual responses to complex naturalistic scenes.

The Challenge is based on the Natural Scenes Dataset (NSD), a massive dataset of 7T fMRI responses to images of natural scenes coming from the COCO dataset. The goal of the Challenge is to promote the development of cutting-edge encoding models which accurately predict (i.e., encode) the fMRI responses to visual input. You can read more about the Challenge in the website and paper.

Here we provide the development kit tutorial of the Challenge (algonauts_2023_challenge_tutorial.ipynb) where you will learn how to:

  • Load and visualize the Challenge fMRI data, functionally different areas of the visual cortex called regions-of-interest (ROIs), and the stimulus images.
  • Build linearizing encoding models using a pretrained AlexNet architecture, evaluate them and visualize the resulting prediction accuracy (i.e., encoding accuracy).
  • Prepare the predicted brain responses to the test images in the right format for submission to the Challenge leaderboard.

Please watch this video for an introduction to the Algonauts Project 2023 Challenge and a walkthrough of this development kit tutorial.

In the code folder you will code used for the Challenge, split into two subfolders:

NOTE: Even if you're not interested in the Algonauts Project 2023 Challenge, you might still find this tutorial useful as an introduction to fMRI data and encoding models of the visual brain.

Should you experience problems with the code, please get in touch with Ale (alessandro.gifford@gmail.com).

Useful links

Algonauts session at CCN 2023

Challenge Website

Challenge Paper

Challenge Data

Challenge DevKit Tutorial (Colab)

Challenge DevKit Tutorial (GitHub)

Challenge Overview and Colab Tutorial Walkthrough Video

CodaLab Challenge Submission Page

CodaLab Submission Walkthrough Video

Natural Scenes Dataset (NSD)

COCO Dataset

Adapt COCO Metadata to Cropped NSD Images

Citations

If you use the data provided for the Algonauts Project 2023 Challenge please cite the following papers:

  1. Gifford AT, Lahner B, Saba-Sadiya S, Vilas MG, Lascelles A, Oliva A, Kay K, Roig G, Cichy RM. 2023. The Algonauts Project 2023 Challenge: How the Human Brain Makes Sense of Natural Scenes. arXiv preprint, arXiv:2301.03198. DOI: https://doi.org/10.48550/arXiv.2301.03198

  2. Allen EJ, St-Yves G, Wu Y, Breedlove JL, Prince JS, Dowdle LT, Nau M, Caron B, Pestilli F, Charest I, Hutchinson JB, Naselaris T, Kay K. 2022. A massive 7T fMRI dataset to bridge cognitive neuroscience and computational intelligence. Nature Neuroscience, 25(1):116–126. DOI: https://doi.org/10.1038/s41593-021-00962-x