README.md

Web Crawler project

• This project is going to use Jsoup library to handle HTML extractions with the help of CSS selectors and HTML tags
• The library also has a function that will simplify how we will make HTTP requests

Research Info links

Web crawler :

• What Is a Web Crawler? | How Web Spiders Work
• HOW SEARCH ENGINES WORK: CRAWLING, INDEXING, AND RANKING
• Web Scraping Without Getting Blocked by Anti Scraping Tools

Web crawler videos:

• Crawling, Indexing, and Ranking
• Web Crawling vs. Web Scraping | Oxylabs
• Googlebot: SEO Mythbusting
• What is a search index? [Basic search indexing]
• How Google searches one document among Billions of documents quickly? [In depth search indexing]

Robots.txt file :

• The robots.txt file explained and illustrated
• Robots.txt File [2020 Examples]

Proxy :

• The Complete Guide to Proxies for Web Scraping
• Connecting through Proxy Servers in Core Java

Proxy videos:

• What is a Proxy?
• Proxy Types | Oxylabs [Basic explanation]

Libraries:

• jsoup Java HTML Parser, with the best of HTML5 DOM methods and CSS selectors.
• Jsoup api overview
• //----------------------------------------
• OpenCSV

//---------------------

• Page content is loaded with JavaScript and Jsoup doesn't see it (Stack overflow)
• Parsing JavaScript Generated Page with Jsoup and HtmUnit
• HtmlUnit library
• HtmlUnit vs JSoup

Other links

.Md [Markdown] extension:

• Getting Started
• Editor.md - Open source online Markdown editor. [This is an online editor for .md file codes, also includes some documentations ]
• Mastering Markdown

//----------------------------------------------

• UserAgent Jsoup

• WebClient Class Microsoft C#

• Spring WebClient – GET, PUT, POST, DELETE examples

• Spring 5 WebClient

//---------------------------------------------

• Setting up libraries - Intellij