Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

character level model support #121

Open
etgld opened this issue Feb 16, 2023 · 4 comments
Open

character level model support #121

etgld opened this issue Feb 16, 2023 · 4 comments
Assignees

Comments

@etgld
Copy link
Contributor

etgld commented Feb 16, 2023

Support for character based models, in particular Flair and canine

@etgld etgld self-assigned this Feb 16, 2023
@etgld
Copy link
Contributor Author

etgld commented Aug 28, 2023

Clinical genomics stuff on pause for now but when that starts again can try to fold in the CANINE support with all the recent updates

@etgld
Copy link
Contributor Author

etgld commented Sep 11, 2023

Hoping to do some work on this issue this week since Timelines is on pause for ITCR. Will need to rebase the branch in my fork off of the error analysis branch and then adapt the error analysis code to work with character level.

@etgld
Copy link
Contributor Author

etgld commented Sep 20, 2023

Still figuring out the most efficient way to do this since a couple of the approaches I tried also have some issues.

Have also been thinking about how to do adapt character level stuff for relation extraction. Tagging has also been more expensive than its wordpiece counterpart, which would mean even more of a blowup for relex. Maybe there's some way to leverage sparse matrices for both

@etgld
Copy link
Contributor Author

etgld commented Oct 5, 2023

Have something that's basically working modulo an issue with the batches being the wrong shape if one uses say layer 8 out of 12 on canine and some formatting tweaks for the error analysis

The former issue might have something to do with the relatively unique downsampling -> upsampling part of the canine architecture. Actually only found out about this by accident since the layer 8 setting was something I was trying with a Roberta based model. Will look into both of these more though.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
1 participant