Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

crawler: ninegag #119

Open
2 tasks
jkowalleck opened this issue Oct 10, 2019 · 2 comments
Open
2 tasks

crawler: ninegag #119

jkowalleck opened this issue Oct 10, 2019 · 2 comments
Labels
2.0-rewrite tasks related to the 2.0 rewrite

Comments

@jkowalleck
Copy link
Member

jkowalleck commented Oct 10, 2019 •
edited
Loading

this ImageCrawler war temporary removed due to the rewrite.
needs to be rewritten.

  • "stills" should be quite easy - but not may ...
  • "gif": this might depends on Rewrite the web UI #183 since githubs images are mostly videos, not gif.
@jkowalleck jkowalleck added the 2.0-rewrite tasks related to the 2.0 rewrite label Oct 10, 2019
@jkowalleck jkowalleck changed the title v2.0-dev crawler: ninegag.py 2.0-dev crawler: ninegag Oct 10, 2019
@jkowalleck jkowalleck changed the title 2.0-dev crawler: ninegag crawler: ninegag Mar 24, 2020
@jkowalleck
Copy link
Member Author

this depends on #183 since githubs images are mostly videos, not gif.

@jkowalleck
Copy link
Member Author

jkowalleck commented May 21, 2020 •
edited
Loading

writung that crawler does not seam too easy ...
thanks to CloudFlare protection ...

curl -v -L -H 'Referer: https://9gag.com/' -H 'Accept: application/json' -H 'User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:75.0) Gecko/20100101 Firefox/75.0' 'https://9gag.com/v1/group-posts/group/default/type/hot?c=10&after='

leads to a CF protection page

using a headless web browser maybe can make a crawling possible.

jkowalleck referenced this issue in jkowalleck/fork_nichtparasoup May 21, 2020
@jkowalleck
started work on 9gag crawler #119 ... and set it on hold
025cbcb
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
2.0-rewrite tasks related to the 2.0 rewrite
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@jkowalleck