Release Notes Heritrix 3.4.0 20190207

For an overview of this release, see A New Release of Heritrix 3.

The following summarises the changes since late 2017.

Bugfixes

• ToeThread death when using HighestUriPrecedenceProvider
• Add checks to guard against server sending 304 in error
• Ensure scope logs have different names per job
• HBase fixes: #222, #243, #224
• Do not checkpoint if crawl job has not started
• Resolved significant BDBFrontier thread safety issues
• HTML extractor does not handle the base href correctly when it's relative
• Catch exceptions scoping outlinks to stop them from derailing process
• Fix for test failures in a workspace on NFS-mounted filesystem
• Enforce robots.txt character limit per char not per line
• Treat a failed fetch (e.g. socket timeout) of robots.txt the same way as http errors
• Allow JavaDNS to be disabled as part of resolving outstanding build and test issues
• WARCLimitEnforcer.java - Add support for multiple warc writers.

Additions:

• By default only execute decide rules if they might change the outcome NOTE that this is potentially a breaking change IF decide rules have side effects you rely on!
• Can now deploy to Maven Central
• Add a simple way to launch the latest checkpoint
• Add parameter to allow even distribution for parallel queues.