[Feature] ISS-60: Implement Self Extend #431
Conversation
|
Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). View this failed invocation of the CLA check for more information. For the most up to date status, view the checks section at the bottom of the pull request. |
|
@jan-wassenberg would need your CR here, its in alpha stage right now. Let's go back and forth on this. Thanks |
jonpsy
force-pushed
the
feature/ISS-60/implement-self-extend
branch
from
4bfe885
to
5a2a7ee
Compare
|
Ooh nice :) Please note that we can only take pull requests on the dev branch. That code has just changed to replace template arguments with a runtime argument. Would you mind updating/rebasing your code to that? |
jonpsy
force-pushed
the
feature/ISS-60/implement-self-extend
branch
from
7bb4e0b
to
8cf3966
Compare
|
My bad, let me do the needful! Thanks for the pointer though. |
|
Haha I took so long to understand how main branch worked now I have to re-do it with this new base branch |
|
Note to self: Was able to compile gemma dev branch by commenting out Had to do this because the compiler has strict check against not sending non-trivial args to variadic method. Maybe I could've turned off that feature in clang using ^^ This issue should be resolved now |
| float* HWY_RESTRICT kv = kv_cache.kv_cache.get() + kv_offset; | ||
| const float* HWY_RESTRICT mha_kv = | ||
| activations_.q.Batch(interleaved_idx) + head * q_stride_ + | ||
| layer_config_.qkv_dim; | ||
|
|
||
| // When embedding position, we will use grouped key position | ||
| if (self_extend && pos > ngb_size) { | ||
| pos /= grp_size; |
There was a problem hiding this comment.
This might be expensive. Would you like to try precomputing and passing around a const hwy::Divisor& like we do with div_seq_len, see if that helps speed?
| @@ -127,6 +127,10 @@ struct LayerConfig { | |||
| size_t conv1d_width = 0; | |||
| bool ff_biases = false; | |||
| bool softmax_attn_output_biases = false; | |||
| bool self_extend = false; | |||
| size_t ngb_size = 0; | |||
There was a problem hiding this comment.
Is this n-gram block? Maybe expand to block_size for more clarity? We can also move these three new fields into a section (just newline before them) with a // Self-extension comment.
#60