v0.7.1

daulet released this 10 Apr 23:30

· 13 commits to main since this release

Update core tokenizers library to latest: v0.15.2;
Expose init time parameter to encode special tokens (or not);

Full Changelog: v0.7.0...v0.7.1

Assets 9