Why do some checkpoints have a count of 0?

[zzh1010016195]

I am using RISCV-ISAC to generate the add instruction coverage. I found that some checkpoints in CGF are 0, but it is obvious from the assembly file that this constraint is satisfied. Is this normal?

The riscv_isac was installed by source code. The spike version is 1.1.1-dev. The gcc version is "riscv32-unknown-elf-gcc (g2ee5e430018) 12.2.0".
I used the following command to generate the instruction coverage.
I first cloned arch-test using RISCOF and then created the default configuration.

riscof --verbose info arch-test --clone
riscof setup --refname=sail_cSim --dutname=spike

Compile:

riscv32-unknown-elf-gcc -march=rv32i -static -mcmodel=medany -fvisibility=hidden -nostdlib -nostartfiles -T ./sail_cSim/env/link.ld         -I ./sail_cSim/env/ -I ./riscv-arch-test/riscv-test-suite/env -mabi=ilp32 ./riscv-arch-test/riscv-test-suite/rv32i_m/I/src/add-01.S -o ref.elf -DTEST_CASE_1=True -DXLEN=32

Generate trace:

spike --log-commits --isa=rv32imc +signature=Reference-spike.signature +signature-granularity=4 ref.elf > add-01.log 2>&1

Generate coverage:

riscv_isac --verbose info coverage -d -t add-01.log --parser-name spike -o coverage.rpt  -e ref.elf -c ./riscv-arch-test/coverage/dataset.cgf -c ./riscv-arch-test/coverage/rvi.cgf -x32 -l add

From the Coverage report, I see that the constraint, "rs1_val == -2147483648", has a count of 0. However, the string can be searched directly from the assembly file, which indicates there is at least one test case that satisfies this constraint.

I have packed all the necessary files. If I make a mistake, please let me know. Thanks.
Test.tar.gz

[allenjbaum]

That is very odd. Could you try it again with parser-name sail? I wonder if there is a parser problem somewhere

…

On Mon, Apr 22, 2024 at 3:48 AM zzhai ***@***.***> wrote: I am using RISCV-ISAC to generate the add instruction coverage. I found that some checkpoints in CGF are 0, but it is obvious from the assembly file that this constraint is satisfied. Is this normal? The riscv_isac was installed by source code. The spike version is 1.1.1-dev. The gcc version is "riscv32-unknown-elf-gcc (g2ee5e430018) 12.2.0". I used the following command to generate the instruction coverage. I first cloned arch-test using RISCOF and then created the default configuration. riscof --verbose info arch-test --clone riscof setup --refname=sail_cSim --dutname=spike Compile: riscv32-unknown-elf-gcc -march=rv32i -static -mcmodel=medany -fvisibility=hidden -nostdlib -nostartfiles -T ./sail_cSim/env/link.ld -I ./sail_cSim/env/ -I ./riscv-arch-test/riscv-test-suite/env -mabi=ilp32 ./riscv-arch-test/riscv-test-suite/rv32i_m/I/src/add-01.S -o ref.elf -DTEST_CASE_1=True -DXLEN=32 Generate trace: spike --log-commits --isa=rv32imc +signature=Reference-spike.signature +signature-granularity=4 ref.elf > add-01.log 2>&1 Generate coverage: riscv_isac --verbose info coverage -d -t add-01.log --parser-name spike -o coverage.rpt -e ref.elf -c ./riscv-arch-test/coverage/dataset.cgf -c ./riscv-arch-test/coverage/rvi.cgf -x32 -l add From the Coverage report, I see that the constraint, "rs1_val == -2147483648", has a count of 0. However, the string can be searched directly from the assembly file, which indicates there is at least one test case that satisfies this constraint. I have packed all the necessary files. If I make a mistake, please let me know. Thanks. Test.tar.gz <https://github.com/riscv-software-src/riscv-isac/files/15061548/Test.tar.gz> — Reply to this email directly, view it on GitHub <#88>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHPXVJV3CNOGPJVMWICLLLDY6TTHJAVCNFSM6AAAAABGSQ3FFKVHI2DSMVQWIX3LMV43ASLTON2WKOZSGI2TMMRRGMZDSOI> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[allenjbaum]

I've been informed that we only use the Sail parser, so the spike parser likely has problems. If it still fails, we will look into it further, but we don't have missing coverage when the tests are run before merging them. Please close the issue if that is the case. On Tue, Apr 23, 2024 at 9:21 PM Allen Baum ***@***.***> wrote:

…

That is very odd. Could you try it again with parser-name sail? I wonder if there is a parser problem somewhere On Mon, Apr 22, 2024 at 3:48 AM zzhai ***@***.***> wrote: > I am using RISCV-ISAC to generate the add instruction coverage. I found > that some checkpoints in CGF are 0, but it is obvious from the assembly > file that this constraint is satisfied. Is this normal? > > The riscv_isac was installed by source code. The spike version is > 1.1.1-dev. The gcc version is "riscv32-unknown-elf-gcc (g2ee5e430018) > 12.2.0". > I used the following command to generate the instruction coverage. > I first cloned arch-test using RISCOF and then created the default > configuration. > > riscof --verbose info arch-test --clone > riscof setup --refname=sail_cSim --dutname=spike > > Compile: > > riscv32-unknown-elf-gcc -march=rv32i -static -mcmodel=medany -fvisibility=hidden -nostdlib -nostartfiles -T ./sail_cSim/env/link.ld -I ./sail_cSim/env/ -I ./riscv-arch-test/riscv-test-suite/env -mabi=ilp32 ./riscv-arch-test/riscv-test-suite/rv32i_m/I/src/add-01.S -o ref.elf -DTEST_CASE_1=True -DXLEN=32 > > Generate trace: > > spike --log-commits --isa=rv32imc +signature=Reference-spike.signature +signature-granularity=4 ref.elf > add-01.log 2>&1 > > Generate coverage: > > riscv_isac --verbose info coverage -d -t add-01.log --parser-name spike -o coverage.rpt -e ref.elf -c ./riscv-arch-test/coverage/dataset.cgf -c ./riscv-arch-test/coverage/rvi.cgf -x32 -l add > > From the Coverage report, I see that the constraint, "rs1_val == > -2147483648", has a count of 0. However, the string can be searched > directly from the assembly file, which indicates there is at least one test > case that satisfies this constraint. > > I have packed all the necessary files. If I make a mistake, please let me > know. Thanks. > Test.tar.gz > <https://github.com/riscv-software-src/riscv-isac/files/15061548/Test.tar.gz> > > — > Reply to this email directly, view it on GitHub > <#88>, or > unsubscribe > <https://github.com/notifications/unsubscribe-auth/AHPXVJV3CNOGPJVMWICLLLDY6TTHJAVCNFSM6AAAAABGSQ3FFKVHI2DSMVQWIX3LMV43ASLTON2WKOZSGI2TMMRRGMZDSOI> > . > You are receiving this because you are subscribed to this thread.Message > ID: ***@***.***> >

[zzh1010016195]

Initially, I used the default parameter of RISCOF, which is Sail, but the count of all checkpoints was 0.
I suspect that c_sail.py did not properly separate the instructions. To fix this issue, I change the separator from "\n\n" to "\n".
Then I run the command:

riscv_isac --verbose info coverage -d -t add-01.log.sail --parser-name c_sail -o coverage.rpt --sig-label begin_signature end_signature --test-label rvtest_code_begin rvtest_code_end -e ref.elf -c ./riscv-arch-test/coverage/dataset.cgf -c ./riscv-arch-test/coverage/rvi.cgf -x32 -l add

As a result, the count for each checkpoint in val_comb group is 0, while all checkpoints in other groups are normal.

I suspect that instructionObject evaluates an incorrect register value.

My RISCV-ISAC installation is from source code. Specifically commit: 1bdc0a7.

[zzh1010016195]

This is the generated Coverage file, I changed the suffix.
coverage.log

[UmerShahidengr]

@zzh1010016195 I would recommend to use riscof coverage command to evaluate the full coverage against the arch_test.

[zzh1010016195]

@zzh1010016195 I would recommend to use riscof coverage command to evaluate the full coverage against the arch_test.

Yes, that's what I initially did, but I ended up with incorrect results.
I rerun all the riscof commands with the above compressed file.

riscof setup
riscof validateyaml --config=config.ini
riscof testlist --config=config.ini --suite=riscv-arch-test/riscv-test-suite/ --env=riscv-arch-test/riscv-test-suite/env
riscof run --config=config.ini --suite=riscv-arch-test/riscv-test-suite/ --env=riscv-arch-test/riscv-test-suite/env
riscof coverage --config=config.ini --suite=riscv-arch-test/riscv-test-suite/ --env=riscv-arch-test/riscv-test-suite/env -c riscv-arch-test/coverage/dataset.cgf -c riscv-arch-test/coverage/rvi.cgf

I only test the add instruction. Here are the latest results.

Maybe there is some problem with my environment. If anyone runs successfully, please let me know and I will close the issue.

[UmerShahidengr]

This command is running successfully at my side. We can have a zoom call to resolve this query. There can be multiple reasons for this, e.g, riscof-plugins are not doing what they intend to do, so probably ISAC wasn't installed correctly.
Ping me [email protected], we will setup a zoom meeting to discuss this issue.

[allenjbaum]

Thank you so much for dealing with this!

…

On Wed, Apr 24, 2024 at 10:19 PM Umer Shahid ***@***.***> wrote: This command is running successfully at my side. We can have a zoom call to resolve this query. There can be multiple reasons for this, e.g, riscof-plugins are not doing what they intend to do, so probably ISAC wasn't installed correctly. Ping me ***@***.***, we will setup a zoom meeting to discuss this issue. — Reply to this email directly, view it on GitHub <#88 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHPXVJSRSYJUW7XNPZW22FLY7CG6JAVCNFSM6AAAAABGSQ3FFKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANZWGM4DANRUGE> . You are receiving this because you commented.Message ID: ***@***.***>

[zzh1010016195]

This command is running successfully at my side. We can have a zoom call to resolve this query. There can be multiple reasons for this, e.g, riscof-plugins are not doing what they intend to do, so probably ISAC wasn't installed correctly. Ping me [email protected], we will setup a zoom meeting to discuss this issue.

Thank you for your assistance. However, it may not be convenient for me to attend the online discussion. I am attempting to recreate the issue in docker container. Once I can successfully replicate it, I will upload the Dockerfile here so others can reproduce the issue. If all goes well, I'll explain it here and close this issue. Thank you once again for your help.

[zzh1010016195]

This command is running successfully at my side. We can have a zoom call to resolve this query. There can be multiple reasons for this, e.g, riscof-plugins are not doing what they intend to do, so probably ISAC wasn't installed correctly. Ping me [email protected], we will setup a zoom meeting to discuss this issue.

I have created a Dockerfile and replicated the issue on Ubuntu.
Create a Dockerfile and write the following:

FROM ubuntu:22.04
ENV DEBIAN_FRONTEND=noninteractive
WORKDIR /home
RUN apt update
RUN apt install --yes python3 git opam device-tree-compiler build-essential libgmp-dev z3 pkg-config zlib1g-dev wget curl
RUN curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py && python3 get-pip.py && pip3 install ruamel.yaml==0.17.26
RUN opam init -y --disable-sandboxing && opam switch create 5.1.0 && opam install -y sail

RUN git clone https://github.com/riscv/sail-riscv.git && eval $(opam config env) && cd sail-riscv && ARCH=RV32 make -j `nproc`; ARCH=RV32 make -j `nproc`; ln -sf ${PWD}/c_emulator/riscv_sim_RV32 /usr/bin/riscv_sim_RV32 && cd /home
RUN git clone https://github.com/riscv-software-src/riscv-isa-sim.git && cd riscv-isa-sim && mkdir build install && cd build && ../configure --prefix=${PWD}/../install && make -j `nproc` && make install && ln -sf ${PWD}/../install/bin/spike /usr/bin/spike && cd /home
RUN wget https://github.com/riscv-collab/riscv-gnu-toolchain/releases/download/2024.04.12/riscv32-elf-ubuntu-22.04-gcc-nightly-2024.04.12-nightly.tar.gz -O riscv32-toolchains.tar.gz && tar -zxvf riscv32-toolchains.tar.gz && echo "export PATH=${PWD}/riscv/bin:$PATH" >> $HOME/.bashrc && cd /home
RUN git clone https://github.com/riscv/riscof.git && cd riscof && pip3 install --editable . && cd /home
RUN git clone https://github.com/riscv/riscv-isac.git && cd riscv-isac && pip3 install --editable . && cd /home

RUN export PATH=/home/riscv/bin:$PATH && wget https://github.com/riscv-software-src/riscv-isac/files/15140919/Test.tar.gz -O Test.tar.gz && tar -zxvf Test.tar.gz && cd /home/Test && \
    riscof setup && \
    riscof validateyaml --config=config.ini && \
    riscof testlist --config=config.ini --suite=riscv-arch-test/riscv-test-suite/ --env=riscv-arch-test/riscv-test-suite/env && \
    riscof run --config=config.ini --suite=riscv-arch-test/riscv-test-suite/ --env=riscv-arch-test/riscv-test-suite/env && \
    riscof coverage --config=config.ini --suite=riscv-arch-test/riscv-test-suite/ --env=riscv-arch-test/riscv-test-suite/env -c riscv-arch-test/coverage/dataset.cgf -c riscv-arch-test/coverage/rvi.cgf

CMD ["bash"]

Then run the following command after installing docker. sudo may be required.

docker build -t ubuntu22.04:test . 
docker run -it --rm ubuntu22.04:test bash

The coverage file should be here: /home/Test/riscof_work/rv32i_m/I/src/add-01.S/coverage.rpt