Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

tokenizers v0.20 not supported #33528

Open
nimz opened this issue Sep 17, 2024 · 2 comments
Open

tokenizers v0.20 not supported #33528

nimz opened this issue Sep 17, 2024 · 2 comments
Labels
Core: Tokenization Internals of the library; Tokenization. Feature request Request for a new feature

Comments

@nimz
Copy link

nimz commented Sep 17, 2024

Feature request

https://github.com/huggingface/tokenizers/releases/tag/v0.20.0 has been out for over a month, but there still isn't a compatible tag of transformers. Is there any plan to add this compatibility soon?

Motivation

It would be great to be able to use the latest tags of the huggingface repos together.

Your contribution

N/A
@nimz nimz added the Feature request Request for a new feature label Sep 17, 2024
@LysandreJik
Copy link
Member

cc @ArthurZucker

@ArthurZucker
Copy link
Collaborator

Hey! Will be shipped in #32535 in a bit!

@LysandreJik LysandreJik added the Core: Tokenization Internals of the library; Tokenization. label Sep 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Core: Tokenization Internals of the library; Tokenization. Feature request Request for a new feature
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@nimz @LysandreJik @ArthurZucker