Only index the latest versions of a component in Google, Bing, ...

[jeromecambon]

Currently the web search engines are indexing all the versions. For example, the top referenced pages are today (May 12) the 2022.2 doc, when the 2023.1 is out.
Since now the URL for the latest version is always the same (use 'latest' instead of the version), we should benefit from that.

We got recommendations (2 years ago) for SEO from an SEO agency : see the slides (pages 20 to 26) (restricted access)

We could only allow all the latest URLs (so disallow the others), to make sure only the latest documentations are explored by the robots.
We should allow the search of all release notes.

Google doc about robots.txt

Previous PR (closed) created on this subject: #246