CHANGELOG.md

Changelog

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

Unreleased

boreal

Breaking changes:

• The ScanResult::statistics field is now boxed. This reduces the size of the object greatly.

0.9.0 - 2024-10-11

This release brings several memory optimizations and small API improvements.

Memory optimizations comes in two forms:

• Generic optimizations to reduce the memory footprint of compiled rules, useful in all cases when the Scanner object is kept for a long time.
• The introduction of a new profile that can be set in the compiler, which will compile rules to optimize for memory usage rather than scanning speed.

boreal

Breaking changes:

• A memory pool was introduced to greatly reduce the memory footprint of compiled rules, notably when the same meta strings are used in all rules. This introduces two breaking changes:

  • The Metadata and MetadataValue objects are no longer re-exported from boreal-parser but are new types.
  • To retrieve strings and byte-strings from those objects, the new Scanner::get_bytes_symbol and Scanner::get_string_symbol must be used.

• A new CompilerBuilder object is introduced, to be able to configure a Compiler before any rule is added.

• Added UnwindSafe and RefUnwindSafe trait bounds on module datas:

  • add UnwindSafe traits to module private datas 43502307
  • add UnwindSafe traits for module user datas 56111d77

• MSRV is bumped from 1.65 to 1.66 825aaab

Added

• Add CompilerBuilder object to add modules and configure compiler profile: 261b11c2
• Add compiler profile to pick between memory usage or scanning speed: #167.
• Add compiler param to disable includes: #170.
• Update compatibility with YARA 4.5.2: #172.

Changed

• Add bytes intern pool to reduce memory consumption: #165.
• Guarantee Scanner is UnwindSafe and RefUnwindSafe: #171.

• Update memory benchmarks 68a1e046

• Update windows-sys dependency to version 0.59 ff996f77
• Update tlsh2 dependency to version 0.4.0 29097dc8

Fixed

• Fix unused warning on statistics in default features config: #168.

boreal-cli

Added

• Added option --profile to select memory or speed profile: c3a89c29.

0.8.0 - 2024-06-09

This release consists of several changes to make the library easier to use in any context or target:

• The dependency on OpenSSL (through the authenticode feature) is removed and replaced by pure-Rust dependencies, through the use of two features:

  • The authenticode feature is retained but is now enabled by default. It uses two new dependencies to parse the authenticode signatures.
  • A new authenticode-verify feature is added to handle the pe.is_signed, pe.signatures[*].verified and pe.signatures[*].countersignatures[*].verified fields. See the dedicated documentation for details.

• The patched version of object has been removed, making the use of the library much easier.

Those changes make boreal depend only on Rust libraries (except for the magic feature), which means the library can be used with any targets and is much easier to integrate.

In addition, this release brings full compatibility with YARA 4.5.1.

boreal

⚠ Breaking changes

• The authenticode feature has been revamped. It is now split into two features:

  • The authenticode feature, which implements all the pe.signatures field except the ones related to signature verification. This feature is now enabled by default.
  • The authenticode-verify feature, which implements the pe.is_signed and *.verified fields. This feature is disabled by default. See the dedicated documentation for details.

• The Compiler API has been reworked to remove all the ugly workarounds that were needed due to the unsafety brought by the OpenSSL dependency. The Compiler::new_with_pe_signatures and Compiler::new_without_pe_module functions has been removed.

Added

• add authenticode-verify feature for signature verification 9ced02bf.

Changed

• Remove hex dependency bb46e49e
• Remove object patched version #159.
• Replace authenticode-parser dependency with a custom impl f9521c5c
• Remove authenticode-parser dependency and clean API 21c5cd74
• Enable hash dependencies when authenticode feature is enabled b88fedb6

YARA 4.5.1 compatibility:

• only consider valid ascii bytes for pe dll names c219245e.
• add some safety checks in pe module for corrupted values 00235005
• update rva resolution in pe module 66c2d5f4
• list dotnet resources that are not located in the file b2fa436d

Fixed

• limit size of version info key and value in pe module 4a20f5c4
• fix parsing issues in version_info of pe module 8c00218a

boreal-parser 0.6.0 - 2024-06-09

Added

• Allow spaces in regex repetitions, eg a{2 , 3} 449c5fc4. This is a new feature introduced in YARA 4.5.1

0.7.0 - 2024-05-05

This release adds the last missing modules from YARA: magic, dex and cuckoo. It also fixes some bugs related to the use of global rules.

boreal

Added:

• The magic module is now available behind the magic feature (not enabled by default). #139.
• The dex module is now available behind the object feature (enabled by default). #141.
• The cuckoo module is now available behind the cuckoo feature (not enabled by default). #143, #144.

Fixed:

• Fix evaluation bug when global rules were declared after non-global rules. #146. If the global rules had any strings, it would make the evaluation of the rules that followed it invalid.
• Fix application of global rules to namespaces. #147, #149. Global rules were applied to all namespaces instead of only their own namespaces.

Changed:

• The type of boreal::module::StaticValue::Function and of the callback declared in the console module has changed from Arc<Box<...>> to Arc<...>. #142.
• Error reporting has been improved on IO error on the rules file. #140.

0.6.0 - 2024-04-14

This release mainly adds the dotnet module and simplifies a few dependencies.

boreal

Added:

• The dotnet module is now available behind the object feature (enabled by default). #127, #131, #133, #135.

Fixed:

• Fixed compilation when using --no-default-features and other feature combinations. #129, #130.
• Fixed exposure of some optional dependencies as their own features. #128.
• Added CI jobs to ensure common combinations of features compile and run tests properly. #132.

Changed:

• The bitmap dependency has been removed and replaced by an custom implementation for our very limited usecase. #120.
• The windows dependency has been replaced by windows-sys. #137.
• All dependencies have been updated to their latest versions.

Thanks to @demoray for their contributions.

0.5.0 - 2024-02-16

This release mainly consists of Yara 4.5 compatibility features and fixes:

boreal

Added:

YARA 4.5 support:

• New Warning on unknown escape sequences in regexes. See PR #68. This warning is more broad than the YARA one from YARA 4.5.
• always expose pe.is_signed 97d1d11
• Do not report strings whose name starts with _ as unused 1a8a8cd
• Add pe.export_details[*].rva field 7597d3f
• math.count and math.percentage now returns an undefined value when given a value outside the [0; 255] range. 6a09ed2
• Imported dlls are ignored if the dll name is longer than 255 bytes 28f8626
• Fix endianness issue in macho.magic field, see the Yara fix 50d418d
• filter imported functions with invalid name in pe module 5a0cb4e
• bump limit on number of listed export symbols in pe module to 16384 98032b3

Changed:

• crc32-fast dependency updated to 1.4 f1ae01a
• authenticode-parser dependency updated e68dde7

Fixed:

• Exclude test assets in package 24ca838. This avoids having the package be flagged by antiviruses, as unfortunately, some of the binaries copied from the yara repository and used for testing seems trigger false positives.

0.4.0 - 2024-02-11

This release introduces process memory scanning, implemented on Windows, Linux and macOS. In addition, different modes of scanning are available, documenting the exact semantics of scanning a process memory. This allows picking a mode that is less surpresing and faster than the default mode which reproduces YARA's behavior. See FragmentedScanMode for more details, as well as the updated updated benchmarks.

In addition, an API to scan fragmented memory is now available. This is the API which is used during process scanning, and allows custom handling of which memory blocks to scan.

Finally, a few additional features have been added, including an API to mmap files to scan, and the ability to get partial results when the scanning fails, for example due to a timeout.

boreal

Added

• Process scanning API on linux, windows #88 and macOS #110.
• Different scanning modes for fragmented memory, including process memory #101
• New memmap feature exposing API to open files to scan using mmap/MapViewOfFile #76
• New process feature exposing API to scan process memory #97
• Implementation of console module fe89efb
• Add fragmented memory handling API #82
• Add ScanError and return Result in scanning API #83

Changed:

Public API:

• Update MSRV to 1.65 1d5b005

Internal API:

• Rework raw variables matching #77
• Compute match details on match #78
• Simplify module evaluation #80
• Rework internal Scanner/Evaluator API #81
• Handle access to memory split in multiple fragments in modules #103

CI:

• Add macos 12 x64 tests in CI #109
• Add tests related to process scanning #111

boreal-cli

Added

• Handling of many flags to mirror the yara CLI tool #102.

  • --scan-list to specify a file listing the files to scan 9982c15
  • -d to define external symbols c584d6a
  • -e to print the namespace of matching rules 4485352
  • -w to disable warnings f9077bf
  • -a to specify a timeout 183d430
  • -m to print metadatas of matching rules d44cfef
  • -i to filter matching rules by name 25a35f8
  • --tag to filter matching rules by tag in boreal-cli cecaa7f
  • -q to disable console logs ce64391
  • -g, -s and -L to print details of the strings of matching rules 277f89f

• Launching a process scan when argument is interpreted as a PID #100

• Flags to control process scanning behavior #101

  • --max-process-memory-chunk to control the size of the memory chunks to scan from the process memory.
  • --max-fetched-region-size to control the maximum size of scanned chunks.
  • --fragmented-scan-mode to control the mode of scanning, see doc on FragmentedScanMode.

Fixed

• Prevent prints to be interleaved when using threads 8ef0b57

0.3.1 - 2023-12-11

Boreal

• Add rule metadata and tags in results of scans. Only the rule name and namespace was listed, which was an oversight. In addition, the Metadata and MetadataValue structs from boreal-parser are re-exported, to avoid having to depend on it to inspect matched rules metadatas. See PR #85.

0.3.0 - 2023-09-12

This is a huge release containing several months of work, including:

• Full compatibility with Yara 4.3. All the new features from Yara 4.3 are available.

• A complete rewrite of the strings compilation algorithm. Performance has been improved dramatically when using a lot of rules or when using strings of lesser quality. See the updated benchmarks.

• New tools to debug and improve performances of rules scanning, which new flags to display several kind of statistics.

  • Strings statistics can now be computed: how are strings compiled, the quality of the extracted atoms, ...j
  • Evaluation duration statistics can now be computed, detailing how long each evaluation step takes. This is only available if the new profiling feature is enabled, to not impact evaluation performance if not set.

• Improved testing on modules and on the boreal-cli binary.

Here are some more details on the new YARA features:

Yara 4.3:

• Negation in hex strings, eg { ~C3 ~?F }.
• New to_string function in math module.
• New string module with to_int and length functions.
• rva field in imported functions in pe module.
• pe.import_rva and pe.delayed_import_rva functions.
• pe.rich_signature.version_data field.
• Iterator on bytes literal, eg for any s in ("foo", "bar"): (...).
• at for expression, eg any of them at 0.
• New functions import_md5 and telfhash in elf module.
• Use of the authenticode-parser lib to parse signatures in pe module. This adds a lot of fields in pe.signatures.

Here are the changes grouped by crate:

Boreal

Added

• Yara 4.3 compatibility. Too many features to list, see above for a short recap of the main new features.
• New profiling feature, needed to compute evaluation statistics.

Changed

• Rewrite of the strings compilation algorithm to significantly improve statistics.
• openssl feature removed, replaced with the authenticode feature.
• Using the pe module with the signatures parsing now requires calling the unsafe function Compiler::new_with_pe_signatures.
• All dependencies updated. regex has been removed in favor of regex-automata.

Fixed

• Improved handling on invalid ranges in '$a in (from..to)' expression.
• Fixed minor differences in edge cases in elf.dynamic_section_entries and ``elf.number_of_sections` (e639df643b05).
• Fixed == operator on boolean values (cec439eee19f).
• Fixed some bugs occuring when using the fullword keyword with both the wide and ascii modifiers, see PR #51.
• Fix compilation of rules following the failed compilation of a rule using a rule dependency. I doubt this actually impacted anyone, see PR #60.
• Change regex behavior to allow non ascii bytes in regexes. See PR #62. A warning has however been added to warn against this situation.
• Fixed string comparison in the pe.imports and pe.(delayed_)import_rva functions to be case-insensitive, See PR #69.

boreal-cli

Added

• New -M flag to a list of available modules.
• New --string-stats flag to display strings' compilation statistics.
• New --scan-stats flag to display evaluation duration statistics.

Changed

• Number of dependencies reduced by removing any use of proc macros.
• boreal updated to 0.3, see boreal changes.

boreal-parser

Added

• Parsing of negation in hex strings, eg { ~C3 ~?F } (9c21fd446).
• Parsing of at for expression, eg any of them at 0 (b26fbc3b6).
• parse_regex and parse_hex_string added to public API (d6a7afc98).

Changed

• Exports of the crate have been entirely reworked. Objects are now nested in relevant modules (3e8682bec).
• Removal of bitflags dependency, rework of VariableModifiers object (05877aae4).
• Regex now accepts non ascii bytes when not in a class. See PR #62.
• AST for bytes and characters in a regex has been updated to provide escaping information and span location. See PR #68.

Fixed

• Some public objects were not properly exposed publicly, this should now be fixed (3e8682bec).

0.2.0 - 2023-02-12

Main changes:

• Compilation and evaluation of rules have been hardened, and it is now possible to use boreal with untrusted rules and inputs.
  • Protections against stack overflows in parsing and evaluation
  • Optional timeout usable during scans.
  • Limits for all arrays in module values (elf, pe, macho).
• Several corner cases in parsing of PE files have been handled, fixing a few inconsistencies with YARA.
• A new warning system has been added, compiled rules can now generate warnings.

Added

• Stack overflow protection during parsing and evaluation.
• Add warning when implicitly converting bytes to boolean (same as in YARA).
• Add AddRuleStatus object returned after compiling rules. This contains warnings emitted during compilation.
• Added new CompilerParams with tweakable params during compilation.
  • max_condition_depth to modify the stack overflow protection limit.
  • fail_on_warnings to ensure all warnings behave as errors.
• Add timeout parameter in ScanParams: any scanning is aborted once the timeout is reached.
• Minimal Rust version is now 1.62 and checked in CI
• ModuleValue now has an Undefined value.
• openssl feature for boreal-cli.
• CI: Coverage computation
• CI: Build with openssl is now properly tested on Windows.

Changed

• Module values are now tested for exact match with module values produced by YARA. This caught a few bugs and improves compatibility.

Fixed

• Ordinal functions in PE module now always have a name, defaulting to ord{n}. (41554fc2bc).
• pe.IMPORT_STANDARD and pe.IMPORT_DELAYED now have the proper values (0fa2477d06).
• pe.number_of_version_infos is now always set (86c6366684).
• pe.export_details[*].offset is now set to -1 when the offset is invalid (122d8bc6a9).
• pe.version_info is now properly built when values are padded in the file (fe7c2356d).
• pe.imports is now properly built when OriginalFirstThunk is invalid (3369ab3ad).
• The pe module now properly parses sections with VirtualSize = 0 (5a3202718).
• Generate an error when an identifier bounded by a for expression is used in the iterator (#15).
• pe.entry_point is now -1 when its file offset is outside the file (#16).

Removed

• boreal::module::StaticValue::Regex has been removed (a7e543b1dee).
• Removed errors VariableCompilationError::AtomsExtractionError and VariableCompilationError::WidenError. Those were logic errors that shouldn't be exposed to users.

0.1.0 - 2022-12-04

Initial release.