Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add XLNetTokenizer #1206

Open
wants to merge 6 commits into
base: master
Choose a base branch
from
Open

Add XLNetTokenizer #1206

wants to merge 6 commits into from

Commits on Aug 10, 2023

  1. pre-init only licenses added

    @susnato
    susnato committed Aug 10, 2023
    Configuration menu
    Copy the full SHA
    b55691e View commit details
    Browse the repository at this point in the history

  2. tokenizer encode and decode done

    @susnato
    susnato committed Aug 10, 2023
    Configuration menu
    Copy the full SHA
    9ef8875 View commit details
    Browse the repository at this point in the history

  3. tests all green!

    @susnato
    susnato committed Aug 10, 2023
    Configuration menu
    Copy the full SHA
    f1fe2d8 View commit details
    Browse the repository at this point in the history

  4. style fix

    @susnato
    susnato committed Aug 10, 2023
    Configuration menu
    Copy the full SHA
    983d93d View commit details
    Browse the repository at this point in the history

  5. tests fix

    @susnato
    susnato committed Aug 10, 2023
    Configuration menu
    Copy the full SHA
    e03e9bb View commit details
    Browse the repository at this point in the history

Commits on Aug 12, 2023

  1. tokenizer now working with ds.map(...)

    @susnato
    susnato committed Aug 12, 2023
    Configuration menu
    Copy the full SHA
    e431196 View commit details
    Browse the repository at this point in the history