Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Automate manual tests #847

Merged
merged 29 commits into from
Sep 26, 2024
Merged

Automate manual tests #847

merged 29 commits into from
Sep 26, 2024

Conversation

sim642
Copy link
Member

@sim642 sim642 commented Oct 6, 2022

Now that we have cram from #787, it's a good time to use it to automate previously manual tests.

Changes

  1. Add tests for YAML witness generation. The yamlWitnessStrip executable is used to remove non-deterministic metadata parts from the output. (Add cram tests for YAML witnesses #1357)
  2. Fix ana.apron.invariant.one-var not filtering enough. (Cherry-picked before)
  3. Make update_suite.rb print test names that have no automatic checks. Add NOCRASH, FIXPOINT, NOTIMEOUT and CRAM annotations to suppress that by indicating which implicit aspect is being tested.
  4. Add such suppressing annotations to tests where the reason for no explicit checks is apparent from comments or git history.

TODO

  • Fix EvalAssert transformation to work on coreutil programs #855

  • Add missing annotations: (existing ones suppressed with NOCHECK)

    No automatic checks in 01/51 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?)
    No automatic checks in 02/57 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?)
    No automatic checks in 03/03 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?)
    No automatic checks in 03/22 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?)
    No automatic checks in 13/62 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?)
    No automatic checks in 13/65 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?)
    No automatic checks in 18/22 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?)
    No automatic checks in 18/23 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?)
    No automatic checks in 18/24 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?)
    No automatic checks in 19/01 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?)
    No automatic checks in 20/01 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?)
    No automatic checks in 20/02 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?)
    No automatic checks in 20/03 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?)
    No automatic checks in 20/05 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?)
    No automatic checks in 22/08 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?)
    No automatic checks in 22/12 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?)
    No automatic checks in 22/13 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?)
    No automatic checks in 22/16 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?)
    No automatic checks in 23/08 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?)
    No automatic checks in 23/12 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?)
    No automatic checks in 25/03 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?)
    No automatic checks in 27/02 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?)
    No automatic checks in 27/07 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?)
    No automatic checks in 29/25 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?)
    No automatic checks in 29/26 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?)
    No automatic checks in 33/01 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?)
    No automatic checks in 33/02 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?)
    No automatic checks in 33/03 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?)
    No automatic checks in 33/04 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?)
    No automatic checks in 34/05 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?)
    No automatic checks in 41/03 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?)
    No automatic checks in 43/12 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?)
    No automatic checks in 43/25 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?)
    No automatic checks in 44/00 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?)
    No automatic checks in 44/01 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?)
    No automatic checks in 44/02 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?)
    No automatic checks in 44/03 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?)
    No automatic checks in 44/13 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?)
    No automatic checks in 44/15 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?)
    No automatic checks in 44/16 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?)
    No automatic checks in 44/17 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?)
    No automatic checks in 44/23 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?)
    No automatic checks in 44/33 (maybe NOCRASH/FIXPOINT/NOTIMEOUT/CRAM?)
    
  • Fix cram test failures on MacOS.

src/analyses/base.ml Outdated Show resolved Hide resolved
@michael-schwarz
Copy link
Member

michael-schwarz commented Nov 24, 2023

What are our plans vis-à-vis this PR? If we want to have it, I'll be happy to review once it's brought up-to-date with master. Otherwise, I suggest we close it.

@sim642 sim642 self-assigned this Jul 16, 2024
@jerhard
Copy link
Member

jerhard commented Jul 16, 2024

To merge this PR, it would be good to add a white list of tests that already exist.
After this PR is merged, old tests can gradually be converted.

@sim642 sim642 marked this pull request as ready for review July 23, 2024 12:08
@sim642 sim642 added this to the v2.5.0 milestone Jul 31, 2024
docs/developer-guide/testing.md Show resolved Hide resolved
scripts/update_suite.rb Show resolved Hide resolved
src/framework/analysisResult.ml Show resolved Hide resolved
@michael-schwarz
Copy link
Member

Alright, LGTM!

@sim642 sim642 merged commit e77b41c into master Sep 26, 2024
20 of 21 checks passed
@sim642 sim642 deleted the cram-manual branch September 26, 2024 13:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants