Whole Matrix Multiply certain dims produces program memory overflow

[gregoryJ-amd]

M?=576
K?=768
N?=768
m?=48
k?=64
n?=64
n_aie_cols?=4

For the above dimensions produces the following error. However, larger dimensions i.e. N=1024 does not, irrespective of datatype.

mkdir -p build
python3 /home/greg/Desktop/mlir-aie/programming_examples/basic/matrix_multiplication/whole_array/aie2.py -m 48 -k 64 -n 64 --n-aie-cols 4 -M 576 -K 768 -N 768 --dtype_in i16 --dtype_out i32 > build/aie_576x768x768_48x64x64_4c.mlir
mkdir -p build
cd build && xchesscc_wrapper aie2 -I /tools/Xilinx/Vitis/2023.2/aietools/include  -DBIT_WIDTH=8 -DDIM_M=48 -DDIM_K=64 -DDIM_N=64 -c /home/greg/Desktop/mlir-aie/programming_examples/basic/matrix_multiplication/whole_array/../../../../aie_kernels/aie2/mm.cc -o mm_48x64x64.o
/home/greg/Desktop/mlir-aie/programming_examples/basic/matrix_multiplication/whole_array/../../../../aie_kernels/aie2/mm.cc:11:9: warning: '__AIENGINE__' macro redefined [-Wmacro-redefined]
#define __AIENGINE__ 2
        ^
<command line>:3:9: note: previous definition is here
#define __AIENGINE__ 1
        ^
In file included from /home/greg/Desktop/mlir-aie/programming_examples/basic/matrix_multiplication/whole_array/../../../../aie_kernels/aie2/mm.cc:23:
In file included from /tools/Xilinx/Vitis/2023.2/aietools/include/aie_api/aie.hpp:10185:
In file included from /tools/Xilinx/Vitis/2023.2/aietools/include/aie_api/aie_adf.hpp:75:
In file included from /tools/Xilinx/Vitis/2023.2/aietools/include/aie_api/adf/stream.hpp:54:
In file included from /tools/Xilinx/Vitis/2023.2/aietools/include/adf.h:7:
/tools/Xilinx/Vitis/2023.2/aietools/include/adf/intrinsics.h:28:9: warning: 'REL_WRITE' macro redefined [-Wmacro-redefined]
#define REL_WRITE -1
        ^
/home/greg/Desktop/mlir-aie/programming_examples/basic/matrix_multiplication/whole_array/../../../../aie_kernels/aie2/mm.cc:20:9: note: previous definition is here
#define REL_WRITE 0
        ^
2 warnings generated.
Warning in "": (imprecise line-number, the error occurred somewhere in this function): cannot move keep_with_operand operation `v16acc64 ups_w2c(v16int16, uint6_t, uint1_t, uint1_t, uint2_t, bool &)' <474>, would violate structural limitations [-Wkeep-with-operand]
Warning in "": (imprecise line-number, the error occurred somewhere in this function): ... intended destination [-Wkeep-with-operand]
Warning in "": (imprecise line-number, the error occurred somewhere in this function): cannot move keep_with_operand operation `v16acc64 ups_w2c(v16int16, uint6_t, uint1_t, uint1_t, uint2_t, bool &)' <476>, would violate structural limitations [-Wkeep-with-operand]
Warning in "": (imprecise line-number, the error occurred somewhere in this function): ... intended destination [-Wkeep-with-operand]
Warning in "": (imprecise line-number, the error occurred somewhere in this function): cannot move keep_with_operand operation `v16acc64 ups_w2c(v16int16, uint6_t, uint1_t, uint1_t, uint2_t, bool &)' <478>, would violate structural limitations [-Wkeep-with-operand]
Warning in "": (imprecise line-number, the error occurred somewhere in this function): ... intended destination [-Wkeep-with-operand]
Warning in "": (imprecise line-number, the error occurred somewhere in this function): cannot move keep_with_operand operation `v16acc64 ups_w2c(v16int16, uint6_t, uint1_t, uint1_t, uint2_t, bool &)' <480>, would violate structural limitations [-Wkeep-with-operand]
Warning in "": (imprecise line-number, the error occurred somewhere in this function): ... intended destination [-Wkeep-with-operand]
Warning in "": (imprecise line-number, the error occurred somewhere in this function): cannot move keep_with_operand operation `v16acc64 ups_w2c(v16int16, uint6_t, uint1_t, uint1_t, uint2_t, bool &)' <482>, would violate structural limitations [-Wkeep-with-operand]
Warning in "": (imprecise line-number, the error occurred somewhere in this function): ... intended destination [-Wkeep-with-operand]
Warning in "": (imprecise line-number, the error occurred somewhere in this function): cannot move keep_with_operand operation `v16acc64 ups_w2c(v16int16, uint6_t, uint1_t, uint1_t, uint2_t, bool &)' <484>, would violate structural limitations [-Wkeep-with-operand]
Warning in "": (imprecise line-number, the error occurred somewhere in this function): ... intended destination [-Wkeep-with-operand]
Warning in "": (imprecise line-number, the error occurred somewhere in this function): cannot move keep_with_operand operation `v16acc64 ups_w2c(v16int16, uint6_t, uint1_t, uint1_t, uint2_t, bool &)' <486>, would violate structural limitations [-Wkeep-with-operand]
Warning in "": (imprecise line-number, the error occurred somewhere in this function): ... intended destination [-Wkeep-with-operand]
Warning in "": (imprecise line-number, the error occurred somewhere in this function): cannot move keep_with_operand operation `v16acc64 ups_w2c(v16int16, uint6_t, uint1_t, uint1_t, uint2_t, bool &)' <488>, would violate structural limitations [-Wkeep-with-operand]
Warning in "": (imprecise line-number, the error occurred somewhere in this function): ... intended destination [-Wkeep-with-operand]
Warning in "": (imprecise line-number, the error occurred somewhere in this function): cannot move keep_with_operand operation `v16acc64 ups_w2c(v16int16, uint6_t, uint1_t, uint1_t, uint2_t, bool &)' <474>, would violate structural limitations [-Wkeep-with-operand]
Warning in "": (imprecise line-number, the error occurred somewhere in this function): ... intended destination [-Wkeep-with-operand]
Warning in "": (imprecise line-number, the error occurred somewhere in this function): cannot move keep_with_operand operation `v16acc64 ups_w2c(v16int16, uint6_t, uint1_t, uint1_t, uint2_t, bool &)' <476>, would violate structural limitations [-Wkeep-with-operand]
Warning in "": (imprecise line-number, the error occurred somewhere in this function): ... intended destination [-Wkeep-with-operand]
Warning in "": (imprecise line-number, the error occurred somewhere in this function): cannot move keep_with_operand operation `v16acc64 ups_w2c(v16int16, uint6_t, uint1_t, uint1_t, uint2_t, bool &)' <478>, would violate structural limitations [-Wkeep-with-operand]
Warning in "": (imprecise line-number, the error occurred somewhere in this function): ... intended destination [-Wkeep-with-operand]
Warning in "": (imprecise line-number, the error occurred somewhere in this function): cannot move keep_with_operand operation `v16acc64 ups_w2c(v16int16, uint6_t, uint1_t, uint1_t, uint2_t, bool &)' <480>, would violate structural limitations [-Wkeep-with-operand]
Warning in "": (imprecise line-number, the error occurred somewhere in this function): ... intended destination [-Wkeep-with-operand]
Warning in "": (imprecise line-number, the error occurred somewhere in this function): cannot move keep_with_operand operation `v16acc64 ups_w2c(v16int16, uint6_t, uint1_t, uint1_t, uint2_t, bool &)' <482>, would violate structural limitations [-Wkeep-with-operand]
Warning in "": (imprecise line-number, the error occurred somewhere in this function): ... intended destination [-Wkeep-with-operand]
Warning in "": (imprecise line-number, the error occurred somewhere in this function): cannot move keep_with_operand operation `v16acc64 ups_w2c(v16int16, uint6_t, uint1_t, uint1_t, uint2_t, bool &)' <484>, would violate structural limitations [-Wkeep-with-operand]
Warning in "": (imprecise line-number, the error occurred somewhere in this function): ... intended destination [-Wkeep-with-operand]
Warning in "": (imprecise line-number, the error occurred somewhere in this function): cannot move keep_with_operand operation `v16acc64 ups_w2c(v16int16, uint6_t, uint1_t, uint1_t, uint2_t, bool &)' <486>, would violate structural limitations [-Wkeep-with-operand]
Warning in "": (imprecise line-number, the error occurred somewhere in this function): ... intended destination [-Wkeep-with-operand]
Warning in "": (imprecise line-number, the error occurred somewhere in this function): cannot move keep_with_operand operation `v16acc64 ups_w2c(v16int16, uint6_t, uint1_t, uint1_t, uint2_t, bool &)' <488>, would violate structural limitations [-Wkeep-with-operand]
Warning in "": (imprecise line-number, the error occurred somewhere in this function): ... intended destination [-Wkeep-with-operand]
Warning in "": (imprecise line-number, the error occurred somewhere in this function): cannot move keep_with_operand operation `v16acc64 ups_x2c(v16int32, uint6_t, uint1_t, uint1_t, uint2_t, bool &)' <357>, would violate structural limitations [-Wkeep-with-operand]
Warning in "": (imprecise line-number, the error occurred somewhere in this function): ... intended destination [-Wkeep-with-operand]
Warning in "": (imprecise line-number, the error occurred somewhere in this function): cannot move keep_with_operand operation `v16acc64 ups_x2c(v16int32, uint6_t, uint1_t, uint1_t, uint2_t, bool &)' <359>, would violate structural limitations [-Wkeep-with-operand]
Warning in "": (imprecise line-number, the error occurred somewhere in this function): ... intended destination [-Wkeep-with-operand]
Warning in "": (imprecise line-number, the error occurred somewhere in this function): cannot move keep_with_operand operation `v16acc64 ups_x2c(v16int32, uint6_t, uint1_t, uint1_t, uint2_t, bool &)' <361>, would violate structural limitations [-Wkeep-with-operand]
Warning in "": (imprecise line-number, the error occurred somewhere in this function): ... intended destination [-Wkeep-with-operand]
Warning in "": (imprecise line-number, the error occurred somewhere in this function): cannot move keep_with_operand operation `v16acc64 ups_x2c(v16int32, uint6_t, uint1_t, uint1_t, uint2_t, bool &)' <363>, would violate structural limitations [-Wkeep-with-operand]
Warning in "": (imprecise line-number, the error occurred somewhere in this function): ... intended destination [-Wkeep-with-operand]
Warning in "": (imprecise line-number, the error occurred somewhere in this function): cannot move keep_with_operand operation `v16acc64 ups_x2c(v16int32, uint6_t, uint1_t, uint1_t, uint2_t, bool &)' <357>, would violate structural limitations [-Wkeep-with-operand]
Warning in "": (imprecise line-number, the error occurred somewhere in this function): ... intended destination [-Wkeep-with-operand]
Warning in "": (imprecise line-number, the error occurred somewhere in this function): cannot move keep_with_operand operation `v16acc64 ups_x2c(v16int32, uint6_t, uint1_t, uint1_t, uint2_t, bool &)' <359>, would violate structural limitations [-Wkeep-with-operand]
Warning in "": (imprecise line-number, the error occurred somewhere in this function): ... intended destination [-Wkeep-with-operand]
Warning in "": (imprecise line-number, the error occurred somewhere in this function): cannot move keep_with_operand operation `v16acc64 ups_x2c(v16int32, uint6_t, uint1_t, uint1_t, uint2_t, bool &)' <361>, would violate structural limitations [-Wkeep-with-operand]
Warning in "": (imprecise line-number, the error occurred somewhere in this function): ... intended destination [-Wkeep-with-operand]
Warning in "": (imprecise line-number, the error occurred somewhere in this function): cannot move keep_with_operand operation `v16acc64 ups_x2c(v16int32, uint6_t, uint1_t, uint1_t, uint2_t, bool &)' <363>, would violate structural limitations [-Wkeep-with-operand]
Warning in "": (imprecise line-number, the error occurred somewhere in this function): ... intended destination [-Wkeep-with-operand]
Warning in "": (imprecise line-number, the error occurred somewhere in this function): loop found to have 4 iterations, fewer than the explicitly annotated minimum 8 [-Wincorrect-annotation]
Warning: : (loop #8)
        Non leaf loop was prepared for pipelining. But the pipelined solutions have not been selected.
        Consider removing the chess_prepare_for_pipelining directive as it may improve results
Warning: : (loop #8)
        Non leaf loop was prepared for pipelining. But the pipelined solutions have not been selected.
        Consider removing the chess_prepare_for_pipelining directive as it may improve results
mkdir -p build
cd build && aiecc.py --aie-generate-cdo --no-compile-host --xclbin-name=final_576x768x768_48x64x64_4c.xclbin \
                        --aie-generate-npu --npu-insts-name=insts_576x768x768_48x64x64_4c.txt ../build/aie_576x768x768_48x64x64_4c.mlir
Found Vitis at /tools/Xilinx/Vitis/2023.2
 MLIR compilation: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:-- 0:00:01 0/1 1 Workerwarning: overriding the module target triple with pdarch-unknown-unknown-elf [-Woverride-module]
 MLIR compilation: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:-- 0:00:01 0/1 1 Worker1 warning generated.
 AIE Compilation: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━  82% 0:00:02 0:00:08 14/17 4 WorkersGenerating: /home/greg/Desktop/mlir-aie/programming_examples/basic/matrix_multiplication/whole_array/build/aie_576x768x768_48x64x64_4c.mlir.prj/aie_cdo_elfs.bin
[AIE ERROR] _XAie_LoadProgMemSection():230: Overflow of program memory
 AIE Compilation: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00 0:00:08 17/17 4 Workers
Traceback (most recent call last):
  File "/home/greg/Desktop/mlir-aie/install/bin/aiecc.py", line 12, in <module>
    main()
  File "/home/greg/Desktop/mlir-aie/install/python/aie/compiler/aiecc/main.py", line 1222, in main
    run(module_str)
  File "/home/greg/Desktop/mlir-aie/install/python/aie/compiler/aiecc/main.py", line 1196, in run
    asyncio.run(runner.run_flow())
  File "/usr/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/home/greg/Desktop/mlir-aie/install/python/aie/compiler/aiecc/main.py", line 1116, in run_flow
    await self.process_cdo()
  File "/home/greg/Desktop/mlir-aie/install/python/aie/compiler/aiecc/main.py", line 561, in process_cdo
    generate_cdo(input_physical.operation, self.tmpdirname)
ValueError: Failed to generate cdo because: 
make: *** [/home/greg/Desktop/mlir-aie/programming_examples/basic/matrix_multiplication/whole_array/../makefile-common:92: build/final_576x768x768_48x64x64_4c.xclbin] Error 1