novelSiteCrawler

Usage

Environment

1. python 3.6+
2. scrapy
3. scrapy-fake-useragent

SetUp

1. Install requirements

   pip install -r requirements.txt
   

2. List crawlers

   scrapy list
   

3. Run

   scrapy crawl xbiquge_so -a start=1 -a end=1
   

   Result will be stored in /output and /data directory.