Skip to content

Home

Jump to bottom Edit New page
Jan-Petter Gundersen edited this page Dec 30, 2015 · 7 revisions

Roadmap

Future features

  • Full specification support (wiki)
    • X-Robots-Tag HTTP header
    • <META> tags
    • <noindex> tags
    • Attribute rel="nofollow" of the <a> tag
    • Clean-param directive
  • Robots.txt fetcher
    • Provide the full URL to check
      • Follow any redirects in case of different hosts
    • Using cURL or GuzzleHttp
    • Enhanced encoding detector and converter
  • Caching system
    • Keep the robots.txt files for up to 24 hours
Add a custom footer
Add a custom sidebar
Clone this wiki locally