0.0.40

Full Changelog: v0.0.39...v0.0.40

v0.0.18

• Fixed the way the DataIterator deals with documents ending with a sentence formed of excluded tokens only.
• Fixed a typo in an import pattern
• (Latin Model) Dealt with some weird Unicode numerals which unexpectedly broke our .isnumeric() usage (e.g. ↀ )

0.0.17

(2020/07/26)

• Added a way to tag texts where word are already tokenized: new lines are word separator,
  double new lines are sentence separator
• Reworked the way preprocessing of special chars is done prior to sentence tokenization and after it.
  Creation of the class Excluder (pie_extended.pipeline.tokenizers.utils.excluder)
  • Allows for more code sharing across models.
• Fixed a typo that would prevent to tag with FREEM (and nobody saw that ! ;) )
• (Unseen in CHANGES.md) New excluder : [REF:1.a.b] will be ignored with the LASLA model.

0.0.13b

Citation ready