Graph Learning for Graph Based Semantic Matching

This repository contains PyTorch implementation of the graph semantic matching as a project of the deep learning course at ETH Zuich. The report can be found here. This code is based off of the implementation of Github.

Get started

1. Install all the python packages in requirements.txt by doing pip install -r reqiurements.txt. You could optionally create a virtualenv.
2. Execute PATH=$PATH:<Path to root of this folder>
3. If you want to run experiment on Pascal VOC Keypoint dataset:
   1. Download VOC2011 dataset and make sure it looks like data/PascalVOC/VOC2011
   2. Download keypoint annotation for VOC2011 from Berkeley server or google drive and make sure it looks like data/PascalVOC/annotations
   3. Rename the original annotations folder inside data/PascalVOC/VOC2011 to Annotations_original. So it should finally look like ````data/PascalVOC/VOC2011/Annotations_original/``
   4. Download the training pairs from Link. Make sure the train/test split is available in data/PascalVOC/voc2011_pairs.npz
4. If you want to run experiment on CUB dataset, please download dataset from Link and extract all the contents to data/CUB folder.
   1. Inside the data folder, execute python CUB_processing.py

Training

Run training and evaluation

python train_eval.py --cfg path/to/your/yaml

and replace path/to/your/yaml by path to your configuration file. Configuration files for the experimets reported in the paper are stored inexperiments/.

Config Files

The config files are as follows:

1. PascalVoc experiments:

   1. gl_voc.yaml - With graph Learning
   2. fc_voc.yaml - With fully connected graphs
   3. delaunay_voc.yaml - With delaunay triangulation on the graph
   4. knn_1_voc.yaml - With knn connectivity with k=1 neighbour.
   5. knn_2_voc.yaml - With knn connectivity with k=2 neighbours.
   6. knn_3_voc.yaml - With knn connectivity with k=3 neighbours.

2. CUB experiments:

   1. gl_cub.yaml - With graph Learning
   2. fc_cub.yaml - With fully connected graphs
   3. delaunay_cub.yaml - With delaunay triangulation on the graph
   4. knn_1_cub.yaml - With knn connectivity with k=1 neighbour.
   5. knn_2_cub.yaml - With knn connectivity with k=2 neighbours.
   6. knn_3_cub.yaml - With knn connectivity with k=3 neighbours.

Pipeline

Flow diagram of the different stages of the graphmatching optimisation:

Results

Graphs learnt by the network given images andkeypoints on Pascal VOC dataset