jprdonnelly/congress
1
0
Fork 0
mirror of https://github.com/unitedstates/congress.git synced 2025-12-25 01:00:04 -05:00
Code Issues Packages Projects Releases Wiki Activity

update scrapelib to 0.10, remove non-bacwards-compatible follow_robots arg.

Browse Source
This commit is contained in:
Eric Mill
2014-07-31 23:05:23 -04:00
parent f0bd0f66a6
commit 5d1272506e
2 changed files with 2 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
tasks/utils.py
View File

@@ -37,7 +37,7 @@ else:
eastern_time_zone = timezone('US/Eastern')
# scraper should be instantiated at class-load time, so that it can rate limit appropriately
scraper = scrapelib.Scraper(requests_per_minute=120, follow_robots=False, retry_attempts=3)
scraper = scrapelib.Scraper(requests_per_minute=120, retry_attempts=3)
scraper.user_agent = "unitedstates/congress (https://github.com/unitedstates/congress)"