Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Modern testing #2384

Closed
wants to merge 317 commits into from
Closed

Modern testing #2384

wants to merge 317 commits into from

Conversation

Rot127
Copy link
Collaborator

@Rot127 Rot127 commented Jun 8, 2024

Superseded by #2456

Your checklist for this pull request

  • I've documented or updated the documentation of every API function and struct this PR changes.
  • I've added tests that prove my fix is effective or that my feature works (if possible)

Detailed description

Modernize testing.

The new unit testing is build around yaml files. They can be consumed by the C test engine (cstest) or any bindings easily.

Statistics

The new cstest logs technically two statistics. One by cmocka and one by cstest.
The reason is:

  1. We might want to group tests in the future (by arch or file for example) so we need a global statistics.
  2. Errors of incorrect yaml parsing are not tracked by cmocka.
  3. Also, cmocka doesn't provide the detailed statistics to us. So we cannot print a nice summary at the end.

Changes

  • New cstest C-implementation
  • New py_cstest Python-implementation
  • Moves almost all tests to yaml files.
  • Old tests which can't be moved (because they are effectively unit tests) are kept.
  • Add numerous test jobs in the workflow
  • Adds several CMake test targets
  • Add LoongArch Python bindings
  • Several bug fixes discovered during testing.

Current test coverage of Auto-Sync Mapping.c files by running cstest test/:

file coverage covered
arch/LoongArch/LoongArchMapping.c 67.29% 144 / 214
arch/PowerPC/PPCMapping.c 72.78% 254 / 349
arch/AArch64/AArch64Mapping.c 60.47% 918 / 1518
arch/TriCore/TriCoreMapping.c 45.00% 45 / 100
arch/Alpha/AlphaMapping.c 86.08% 68 / 79
arch/ARM/ARMMapping.c 61.68% 792 / 1284
arch/HPPA/HPPAMapping.c 29.85% 20 / 67

Todo

  • No _set testing in Python code.

After merge

  • Add windows build to CI

Document

  • Multi-mode MCUpdater and incomplete MC test files.

Test plan

Added and all green

Closing issues

closes #1984
closes #2405

@Rot127 Rot127 added the Testing Test related issue label Jun 8, 2024
@Rot127 Rot127 added this to the v6 milestone Jun 8, 2024
@github-actions github-actions bot added Auto-Sync-files Auto-Sync and removed HPPA Arch labels Jun 26, 2024
@Rot127 Rot127 mentioned this pull request Jul 4, 2024
2 tasks
@Rot127 Rot127 force-pushed the modern-testing branch 2 times, most recently from 1673b1f to c4e1ab2 Compare July 8, 2024 07:33
@github-actions github-actions bot removed X86 Arch ARM Arch Mips Arch AArch64 Arch Alpha Arch PowerPC Arch labels Jul 8, 2024
@Rot127 Rot127 closed this Aug 24, 2024
@Rot127
Copy link
Collaborator Author

Rot127 commented Aug 25, 2024

Superseded by #2456

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

test_riscv.py failing Modernize Capstone testing
1 participant