Merge branch 'master' of https://github.com/danthespal/webScraper

danthespal · Feb 20, 2020 · 3663550 · 3663550
2 parents 3c482c6 + f828dfa
commit 3663550
Showing 1 changed file with 4 additions and 1 deletion.
diff --git a/README.md b/README.md
@@ -32,7 +32,10 @@ foo@bar:~$ scrapy crawl crawler -o items.json
  - ~~base structure for downloading items~~
  - ~~user agent~~
  - ~~download_delay and auto_throttle for better scraping~~
- -  database to store the scraped data
  - ~~download from multiple pages~~
+ - ~~database to store the scraped data~~
+ - establish list of sites
+ - remove unicode from product names
+ - check for duplicates product_name
  - customization for product items
  - ... more will come