Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Mlas int4 int8 with avx2/512 #20687

Merged
merged 48 commits into from
Aug 2, 2024
Merged

Mlas int4 int8 with avx2/512 #20687

merged 48 commits into from
Aug 2, 2024

Commits on May 3, 2024

  1. quick adapt llama.cpp to experiment performance. Only works with blkl… 

    …en32, symmetric1 hasBias0 Int8

Signed-off-by: Liqun Fu <[email protected]>
    @liqunfu
    liqunfu committed May 3, 2024
    Configuration menu
    Copy the full SHA
    293f121 View commit details
    Browse the repository at this point in the history

Commits on May 6, 2024

  1. fire 

    Signed-off-by: Liqun Fu <[email protected]>
    @liqunfu
    liqunfu committed May 6, 2024
    Configuration menu
    Copy the full SHA
    04c2e56 View commit details
    Browse the repository at this point in the history

Commits on May 7, 2024

  1. tile 2x4 SQNBITGEMM<4>/BlkLen:32/M:2048/N:4096/K:4096/Threads:1/Symme… 

    …tric:1/ComputeType:4/real_time_mean 1542487160 ns 1539062500 ns

Signed-off-by: Liqun Fu <[email protected]>
    @liqunfu
    liqunfu committed May 7, 2024
    Configuration menu
    Copy the full SHA
    cdfda6f View commit details
    Browse the repository at this point in the history

Commits on May 8, 2024

  1. use one_16_epi16 and accumulate_2blk_dot: SQNBITGEMM<4>/BlkLen:32/M:2… 

    …048/N:4096/K:4096/Threads:1/Symmetric:1/ComputeType:4/real_time_mean 1434872720 ns

Signed-off-by: Liqun Fu <[email protected]>
    @liqunfu
    liqunfu committed May 8, 2024
    Configuration menu
    Copy the full SHA
    92dad97 View commit details
    Browse the repository at this point in the history

Commits on May 9, 2024

  1. apply to M1, BQuant layout pack block (subblk) larger than blklen: SQ… 

    …NBITGEMM<4>/BlkLen:32/M:2048/N:4096/K:4096/Threads:1/Symmetric:1/ComputeType:4/real_time_mean 1265060620 ns 1265625000 ns

Signed-off-by: Liqun Fu <[email protected]>
    @liqunfu
    liqunfu committed May 9, 2024
    Configuration menu
    Copy the full SHA
    5418e9c View commit details
    Browse the repository at this point in the history

Commits on May 10, 2024

  1. use new AQuant layout (not work if total M is not RangeCountM): SQNBI… 

    …TGEMM<4>/BlkLen:32/M:2048/N:4096/K:4096/Threads:1/Symmetric:1/ComputeType:4/real_time_mean 1214042220 ns

Signed-off-by: Liqun Fu <[email protected]>
    @liqunfu
    liqunfu committed May 10, 2024
    Configuration menu
    Copy the full SHA
    0401f72 View commit details
    Browse the repository at this point in the history

Commits on May 13, 2024

  1. apply blksum to blklen32 and 64: SQNBITGEMM<4>/BlkLen:32/M:2048/N:409… 

    …6/K:4096/Threads:1/Symmetric:1/ComputeType:4/real_time_mean 784668090 ns; SQNBITGEMM<4>/BlkLen:64/M:2048/N:4096/K:4096/Threads:1/Symmetric:1/ComputeType:4/real_time_mean 754939430 ns

Signed-off-by: Liqun Fu <[email protected]>
    @liqunfu
    liqunfu committed May 13, 2024
    Configuration menu
    Copy the full SHA
    a57eeba View commit details
    Browse the repository at this point in the history

Commits on May 15, 2024

  1. blklen16 

    Signed-off-by: Liqun Fu <[email protected]>
    @liqunfu
    liqunfu committed May 15, 2024
    Configuration menu
    Copy the full SHA
    f2c33af View commit details
    Browse the repository at this point in the history

Commits on May 26, 2024

  1. impl avx512: SQNBITGEMM<4>/BlkLen:32/M:2048/N:4096/K:4096/Threads:1/S… 

    …ymmetric:1/ComputeType:4/real_time_mean 664029830 ns

Signed-off-by: liqunfu <[email protected]>
    @liqunfu
    liqunfu committed May 26, 2024
    Configuration menu
    Copy the full SHA
    0ca24f4 View commit details
    Browse the repository at this point in the history

Commits on Jun 1, 2024

  1. matmul_nbit & fix alignment for sgemm 

    Signed-off-by: Liqun Fu <[email protected]>
    @liqunfu
    liqunfu committed Jun 1, 2024
    Configuration menu
    Copy the full SHA
    7f89d5f View commit details
    Browse the repository at this point in the history

Commits on Jun 4, 2024

  1. merge main 

    Signed-off-by: Liqun Fu <[email protected]>
    @liqunfu
    liqunfu committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    ed0e666 View commit details
    Browse the repository at this point in the history

Commits on Jun 10, 2024

  1. fix mlas benchmark not using multi threads 

    Signed-off-by: Liqun Fu <[email protected]>
    @liqunfu
    liqunfu committed Jun 10, 2024
    Configuration menu
    Copy the full SHA
    35d02a6 View commit details
    Browse the repository at this point in the history

  2. profiling 

    Signed-off-by: Liqun Fu <[email protected]>
    @liqunfu
    liqunfu committed Jun 10, 2024
    Configuration menu
    Copy the full SHA
    b9493ad View commit details
    Browse the repository at this point in the history

  3. Merge branch 'liqun/mlas-q4-tile-avx' of https://github.com/microsoft… 

    …/onnxruntime into liqun/mlas-q4-tile-avx
    @liqunfu
    liqunfu committed Jun 10, 2024
    Configuration menu
    Copy the full SHA
    c443eb5 View commit details
    Browse the repository at this point in the history

Commits on Jun 16, 2024

  1. sgemm after sq4bit for avx2 

    Signed-off-by: Liqun Fu <[email protected]>
    @liqunfu
    liqunfu committed Jun 16, 2024
    Configuration menu
    Copy the full SHA
    ac66951 View commit details
    Browse the repository at this point in the history

Commits on Jun 17, 2024

  1. avx512 

    Signed-off-by: liqunfu <[email protected]>
    @liqunfu
    liqunfu committed Jun 17, 2024
    Configuration menu
    Copy the full SHA
    42a1305 View commit details
    Browse the repository at this point in the history

Commits on Jun 27, 2024

  1. layout to follow compute, M1 separate with M > 1 

    Signed-off-by: Liqun Fu <[email protected]>
    @liqunfu
    liqunfu committed Jun 27, 2024
    Configuration menu
    Copy the full SHA
    740031a View commit details
    Browse the repository at this point in the history

Commits on Jun 28, 2024

  1. make avx512 run 

    Signed-off-by: liqunfu <[email protected]>
    @liqunfu
    liqunfu committed Jun 28, 2024
    Configuration menu
    Copy the full SHA
    1a6031e View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into liqun/mlas-q4-tile-avx

    @liqunfu
    liqunfu committed Jun 28, 2024
    Configuration menu
    Copy the full SHA
    283fd2d View commit details
    Browse the repository at this point in the history

Commits on Jul 4, 2024

  1. avx512 blklen64 pass 

    Signed-off-by: liqunfu <[email protected]>
    @liqunfu
    liqunfu committed Jul 4, 2024
    Configuration menu
    Copy the full SHA
    d035939 View commit details
    Browse the repository at this point in the history

Commits on Jul 5, 2024

  1. pass avx512 blklen32 

    Signed-off-by: liqunfu <[email protected]>
    @liqunfu
    liqunfu committed Jul 5, 2024
    Configuration menu
    Copy the full SHA
    f329d2d View commit details
    Browse the repository at this point in the history

  2. pass avx512 blklen 16, 128, 256 

    Signed-off-by: liqunfu <[email protected]>
    @liqunfu
    liqunfu committed Jul 5, 2024
    Configuration menu
    Copy the full SHA
    27cfd9c View commit details
    Browse the repository at this point in the history

Commits on Jul 11, 2024

  1. pass fp32, refactor sqnbitgemm 

    Signed-off-by: Liqun Fu <[email protected]>
    @liqunfu
    liqunfu committed Jul 11, 2024
    Configuration menu
    Copy the full SHA
    edee319 View commit details
    Browse the repository at this point in the history

Commits on Jul 12, 2024

  1. merge main 

    Signed-off-by: liqunfu <[email protected]>
    @liqunfu
    liqunfu committed Jul 12, 2024
    Configuration menu
    Copy the full SHA
    fb9221a View commit details
    Browse the repository at this point in the history

Commits on Jul 18, 2024

  1. avx512vnni 

    Signed-off-by: liqunfu <[email protected]>
    @liqunfu
    liqunfu committed Jul 18, 2024
    Configuration menu
    Copy the full SHA
    c109b4b View commit details
    Browse the repository at this point in the history

  2. merge main 

    Signed-off-by: Liqun Fu <[email protected]>
    @liqunfu
    liqunfu committed Jul 18, 2024
    Configuration menu
    Copy the full SHA
    6654d22 View commit details
    Browse the repository at this point in the history

Commits on Jul 20, 2024

  1. avxvnni 

    Signed-off-by: Liqun Fu <[email protected]>
    @liqunfu
    liqunfu committed Jul 20, 2024
    Configuration menu
    Copy the full SHA
    4b91bed View commit details
    Browse the repository at this point in the history

Commits on Jul 23, 2024

  1. rm unused ComputeParallelTasksSGemm 

    Signed-off-by: liqunfu <[email protected]>
    @liqunfu
    liqunfu committed Jul 23, 2024
    Configuration menu
    Copy the full SHA
    8674b9f View commit details
    Browse the repository at this point in the history

Commits on Jul 24, 2024

  1. avoid _mm256_dpbusds_avx_epi32 in avx512vnni 

    Signed-off-by: liqunfu <[email protected]>
    @liqunfu
    liqunfu committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    e26e29e View commit details
    Browse the repository at this point in the history

  2. fix linux build 

    Signed-off-by: liqunfu <[email protected]>
    @liqunfu
    liqunfu committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    2b0307e View commit details
    Browse the repository at this point in the history

Commits on Jul 26, 2024

  1. Merge branch 'main' into liqun/mlas-q4-tile-avx

    @liqunfu
    liqunfu committed Jul 26, 2024
    Configuration menu
    Copy the full SHA
    40df782 View commit details
    Browse the repository at this point in the history

  2. refactor for Arm64 

    Signed-off-by: Liqun Fu <[email protected]>
    @liqunfu
    liqunfu committed Jul 26, 2024
    Configuration menu
    Copy the full SHA
    51e97c8 View commit details
    Browse the repository at this point in the history

  3. more refactor for Arm64 

    Signed-off-by: Liqun Fu <[email protected]>
    @liqunfu
    liqunfu committed Jul 26, 2024
    Configuration menu
    Copy the full SHA
    48e8639 View commit details
    Browse the repository at this point in the history

Commits on Jul 29, 2024

  1. hsum_float_16 

    Signed-off-by: liqunfu <[email protected]>
    @liqunfu
    liqunfu committed Jul 29, 2024
    Configuration menu
    Copy the full SHA
    705aa1f View commit details
    Browse the repository at this point in the history

  2. hsum_float_16 

    Signed-off-by: liqunfu <[email protected]>
    @liqunfu
    liqunfu committed Jul 29, 2024
    Configuration menu
    Copy the full SHA
    012e9c4 View commit details
    Browse the repository at this point in the history

Commits on Jul 30, 2024

  1. condition for -mavxvnni 

    Signed-off-by: liqunfu <[email protected]>
    @liqunfu
    liqunfu committed Jul 30, 2024
    Configuration menu
    Copy the full SHA
    21b9138 View commit details
    Browse the repository at this point in the history

  2. CMAKE_CXX_COMPILER_VERSION VERSION_GREATER 10 

    Signed-off-by: liqunfu <[email protected]>
    @liqunfu
    liqunfu committed Jul 30, 2024
    Configuration menu
    Copy the full SHA
    1fb1c83 View commit details
    Browse the repository at this point in the history

  3. missed 2 files from (__GNUC__ > 10) 

    Signed-off-by: liqunfu <[email protected]>
    @liqunfu
    liqunfu committed Jul 30, 2024
    Configuration menu
    Copy the full SHA
    85918e9 View commit details
    Browse the repository at this point in the history

  4. missed _mm256_dpbusds_avx_epi32 and print out cmake msgs 

    Signed-off-by: liqunfu <[email protected]>
    @liqunfu
    liqunfu committed Jul 30, 2024
    Configuration menu
    Copy the full SHA
    9530ac5 View commit details
    Browse the repository at this point in the history

  5. unused zp, etc. 

    Signed-off-by: liqunfu <[email protected]>
    @liqunfu
    liqunfu committed Jul 30, 2024
    Configuration menu
    Copy the full SHA
    f77cffd View commit details
    Browse the repository at this point in the history

  6. unused zp, etc. 

    Signed-off-by: liqunfu <[email protected]>
    @liqunfu
    liqunfu committed Jul 30, 2024
    Configuration menu
    Copy the full SHA
    a6fd378 View commit details
    Browse the repository at this point in the history

  7. remove test code changes 

    Signed-off-by: liqunfu <[email protected]>
    @liqunfu
    liqunfu committed Jul 30, 2024
    Configuration menu
    Copy the full SHA
    c875e5c View commit details
    Browse the repository at this point in the history

  8. remove test code changes 

    Signed-off-by: liqunfu <[email protected]>
    @liqunfu
    liqunfu committed Jul 30, 2024
    Configuration menu
    Copy the full SHA
    3b56710 View commit details
    Browse the repository at this point in the history

  9. lint 

    Signed-off-by: liqunfu <[email protected]>
    @liqunfu
    liqunfu committed Jul 30, 2024
    Configuration menu
    Copy the full SHA
    746562f View commit details
    Browse the repository at this point in the history

  10. lint 

    Signed-off-by: liqunfu <[email protected]>
    @liqunfu
    liqunfu committed Jul 30, 2024
    Configuration menu
    Copy the full SHA
    52fc7fa View commit details
    Browse the repository at this point in the history

  11. code name 

    Signed-off-by: liqunfu <[email protected]>
    @liqunfu
    liqunfu committed Jul 30, 2024
    Configuration menu
    Copy the full SHA
    0933a6b View commit details
    Browse the repository at this point in the history

Commits on Jul 31, 2024

  1. update reviewers' comments 

    Signed-off-by: liqunfu <[email protected]>
    @liqunfu
    liqunfu committed Jul 31, 2024
    Configuration menu
    Copy the full SHA
    2b35c82 View commit details
    Browse the repository at this point in the history

Commits on Aug 1, 2024

  1. Merge branch 'main' into liqun/mlas-q4-tile-avx

    @liqunfu
    liqunfu committed Aug 1, 2024
    Configuration menu
    Copy the full SHA
    caeb35e View commit details
    Browse the repository at this point in the history