Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
v0.0.3 introduces these changes: 1. Introduces TokenizerInfo to uniformly handle various tokenizers, including huggingface tokenizer, tiktoken and mlc-llm tokenizer-cpp. 2. Introduces GrammarMatcherInitContext, an object that can be used to fast initialize multiple GrammarMatchers. A cache for GrammarMatcherInitContext called GrammarMatcherInitContextCache is introduces as well. 3. Introduces WebAssembly bindings for web uses. 4. Documents are refined for clearity.
- Loading branch information