RESTful API for Modify.sg
- Normalizes data from NUS and NTU (Huge thanks to NUSMods-API)
- Provides a RESTful API to get information
- Simple
http://api.modify.sg/modulesList/nus/2016/1/
[
"code": "ACC1002",
"title": "Financial Accounting"
},
{
"code": "ACC1006",
"title": "Accounting Information Systems"
},
...
]
https://api.modify.sg/modules/ntu/2016/1/AAA18C
{
"code": "AAA18C",
"department": "NIE",
"credit": 3,
"title": "Introduction to Contemporary Sculpture Practice",
"description": "A study of the skills and knowledge ...",
"exam_time": null,
"exam_venue": null,
"exam_duration": null,
"prerequisite": "Only opened to NBS students.",
"preclusion": "AB0301, AB0401, AB0402, AB0501, AB0502, AB0603",
"availability": "Not available to Programme: ACBS(GA) 1 ...",
"remarks": "Course is available as ...",
"timetable": [
{
"class_no":"S1",
"day_text":"MON",
"lesson_type":"SEM",
"week_text":"Every week",
"venue":"LHS-TR+53",
"start_time":"18:00:00",
"end_time":"21:00:00"
}, ...
]
}
Server is a nodejs application hosted on a DigitalOcean droplet.
- node.js - back-end server language
- Express - node.js network app framework
- Nginx - reverse proxy
- Scrapy - spider for scraping info
- PostgreSQL - database
Download and install Scrapy and PostgreSQL, then, clone this repository.
git clone https://github.com/li-kai/modify-api.git
cd modify-api/scrapy
Set up the postgreSQL user scrapy as well as initialize the database with the schema in db.sql. Fill up user details in queries.js and settings.py. Then, run the spiders with the following terminal commands:
# for nus
scrapy crawl nus_details
# for ntu (note the order)
scrapy crawl ntu_details
scrapy crawl ntu_timetables
python exams.py
Download and install Node.js v4+ to run, then run the following in the terminal:
git clone https://github.com/li-kai/modify-api.git
cd modify-api
# install dependencies
npm install
# serve at localhost:3000
npm start
The API is still unfinished. Especially with regards to the schema of the database. Any help is welcomed!
MIT