Collect all reddit topics and comments for a specific subreddit

Completed Posted 7 years ago Paid on delivery
Completed Paid on delivery

Hi,

Looking for someone to write a piece of software that will fetch all reddit posts and comments for a subreddit from the year 2015. Please take the reddit API limits into account and find a workaround for that.

Return format for each topic should be a python dict. Comments are stored nested in the post dict:

posts =

{{'author': {'username': redditusername,

'timestamp':timestamp,

'url': topicurl

'topictext':text,

'comments': {{'comment': text, 'timestamp': timestamp, 'author': {'username': redditusername, 'profileurl': url}, {comment2 etc..}}

}, second post here ... }

Output stored to a text file.

Looking to get this done ASAP. Get in touch to discuss details.

The Reddit API limit is set to a maximum of 1000 items. I think it is possible to get around the API limit by using timestamps, but I'm not sure. Use the reddit search API function to get less than 1000 items within a specific timeframe (which you can specify in the search), then use the timestamp of the last post to create a new time window. Open to other approaches.

API Documentation can be found here: https://www.reddit.com/dev/api

Data Processing Python Software Architecture Web Scraping

Project ID: #10606322

About the project

11 proposals Remote project Active 7 years ago

Awarded to:

akprj

Hello Final year CS undergrad at IIT Bombay. Had done a similar crawler project for codechef and stackoverflow a couple of months ago using bs4 and python. Can do this in a few hours. Looking forward to work on this More

$250 USD in 1 day
(3 Reviews)
3.1

11 freelancers are bidding on average $342 for this job

e3d

Hi, I can scrape those from reddit with no problem but to properly estimate this project I need to know which subreddit you're talking about.

$263 USD in 3 days
(268 Reviews)
8.7
mananraja

Hi there, I have read the project & would like to discuss.. I can scrape data from website using custom made scripts in Python.. I have good web scraping reviews as well.. I have experience with APIs as well as like More

$250 USD in 1 day
(171 Reviews)
6.4
mwarrenschultz

Hello! I can get around Reddit's API limit by avoiding the API altogether, and interfacing instead directly through the browser using Selenium. I am a professional programmer with many years of web scraping experience More

$444 USD in 10 days
(69 Reviews)
6.5
DhvanitAkbari

Hello, I am a computer engineer, I have experience in web scrapping using python so we can chat further to discuss project. Thanks

$500 USD in 2 days
(19 Reviews)
3.9
Fonseca25T

hello, I can automate this tasks considering the Reddit api limitations. This will be up the amount of post the subreddit would have and all the comments too. For example, if the api limitations will reach the top in a More

$500 USD in 15 days
(1 Review)
1.6
tonygiorgi

Hello, I am currently building my own software and am in need of work to help supplement the costs of building my own business. I am a graduate of UC Berkeley with a breadth of professional experience as a product m More

$277 USD in 5 days
(0 Reviews)
0.0