GitHub - diwamishra21/Python-URL-Scrapper: fetch content through URL and save it to a json file using python Scrapy scrapper

diwamishra21 / Python-URL-Scrapper Public

Notifications You must be signed in to change notification settings
Fork 0
Star 2

fetch content through URL and save it to a json file using python Scrapy scrapper

Notifications

Name		Name	Last commit message	Last commit date
Latest commit History 3 Commits
__pycache__		__pycache__
spiders		spiders
__init__.py		__init__.py
items.py		items.py
middlewares.py		middlewares.py
pipelines.py		pipelines.py
readme.txt		readme.txt
settings.py		settings.py

Repository files navigation

# Objective-
to fetch content of URL and save it to a json file

# Directory structure-

# spiders-
this folder contains main file of scrapper ie. clinicaltrial.py

# scrapped_data-
this folder contains files which has extracted content

# How to run-
1. create project using scrapy by command
scrapy startproject tutorial
2. copy ur main python scrapper file(pythonscrapper.py) inside spiders folder of created project

# reference for scrapy-
https://docs.scrapy.org/en/latest/intro/tutorial.html