Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[JS/WebGPU] Add GatherBlockQuantized op support #21734

Merged
merged 48 commits into from
Aug 26, 2024
Merged

[JS/WebGPU] Add GatherBlockQuantized op support #21734

merged 48 commits into from
Aug 26, 2024

Conversation

satyajandhyala
Copy link
Contributor

@satyajandhyala satyajandhyala commented Aug 14, 2024 •
edited
Loading

Description

Add GatherBlockQuantized operator to JSEP.

Motivation and Context

Gemma model requires this.

@satyajandhyala satyajandhyala added the ep:WebGPU ort-web webgpu provider label Aug 14, 2024
@satyajandhyala satyajandhyala marked this pull request as ready for review August 14, 2024 13:06
@satyajandhyala
Commented out indices validation code.
963814a
@satyajandhyala
Added (u)int4.
5fbb497
@satyajandhyala
test related changes
d065cea
@satyajandhyala
fixed unused variable.
6a57c34
@satyajandhyala
everted changes tensor_helper.cc
163866a
@satyajandhyala
Updated expected output to match that of wasm.
a519237
@satyajandhyala
Use indicesGet/indicesSet to access index out of indices
b93ca99
@satyajandhyala
typo
60d6ba8
@satyajandhyala
renamed dequantize-linear_int4.jsonc as dequantize-linear-int4.jsonc
0a7387d
@satyajandhyala
Indices should be normalized before indexing. Added a test case.
9b5eac4
@satyajandhyala
format JSONC
d62058a
@satyajandhyala
Merge branch 'main' of github.com:microsoft/onnxruntime into sajandhy…
2162f92 
…/webgpu_add_block_quantized_gather
@satyajandhyala
Avoid producing presentKey/presentValue outputs if pastKey/pastValue …
18a3906 
…don't exists.
@satyajandhyala
Don't treat empty inputs as undefined in MHA. Let Attention deal with…
9e22564 
… empty inputs so that present_key/present_value may be produced even if past_key/past_value is empty.
@satyajandhyala
Feed pastKey/pastValue inputs down to the functions that generate sha…
c01f721 
…der code without checking non-zero tensors.
@satyajandhyala
Added a test case with zero-size pastKey/pastValue inputs that requir…
c8b187f 
…es presentKey/presentValue outputs.
@satyajandhyala
Added back the assumption comment.
d0b0627
guschmue
guschmue approved these changes Aug 26, 2024
View reviewed changes
Copy link
Contributor

@guschmue guschmue left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

works for me. There is some wired issue when using this with gemma-2-2b but I don't think this is related to this PR.

@satyajandhyala satyajandhyala merged commit af18824 into main Aug 26, 2024
97 checks passed
@satyajandhyala satyajandhyala deleted the sajandhy/webgpu_add_block_quantized_gather branch August 26, 2024 21:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@guschmue guschmue guschmue approved these changes

@fs-eire fs-eire Awaiting requested review from fs-eire

Assignees
No one assigned
Labels
ep:WebGPU ort-web webgpu provider
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@satyajandhyala @guschmue