-
Notifications
You must be signed in to change notification settings - Fork 59
Scrapy outsourcing contract
We chose to ask the scrapinghub to help us out with implementing our intentions using scrapy. Here is the specifications of the work to be achieved.
design and prepare specification for HCI crawler implementation on Scrapy, and Scrapy changes/improvements required. This work will be lead by the questions gathered here : Core#scrapy_implementation
Nov 15th - 30th (1 week)
6 billable days
The results of this task can be found in Scrapy implementation proposal
discuss and agree over the interaction between Scrapy and HCI open source projects. The objectives is to think about the best community management organisation so that both project can benefit one from another from future developments (scrapy-developers list for discussion, github pull requests, etc)
Nov 15th - 30th (2 weeks)
2 billable days
The results of this task can be found in Scrapy community management
- Scrapy training to HCI team
- Development of a prototype based on the implementation proposal from specifications written at this page Core and from Task 1 results
- task 3 will include a visit to Paris at Sciences Po médialab : a week, from Dec 5th to 9th
December 2011
16 billable days
Task closed on December 15th.
- Code cleaning and refactoring if needed
- Bug corrections and minor features implementation
- Writing the documentation
- Unit testing and quick benchmark
December 2011
13 billable days
Task closed on 27th of December. See documentation about the Crawler architecture.