Skip to content

Commit

Permalink
feat: add missing golden references
Browse files Browse the repository at this point in the history
  • Loading branch information
@guibou
guibou committed Sep 21, 2024
1 parent 23baea5 commit 64d34d8
Show file tree
Hide file tree
Showing 4 changed files with 1,222 additions and 0 deletions.
19 changes: 19 additions & 0 deletions test_resources/defaultSettings-show.golden
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
Settings
{ settingSeed = FixedSeed 42
, settingRandomiseExecutionOrder = True
, settingThreads = ByCapabilities
, settingMaxSuccess = 100
, settingMaxSize = 100
, settingMaxDiscard = 10
, settingMaxShrinks = 100
, settingGoldenStart = False
, settingGoldenReset = False
, settingColour = Nothing
, settingFilters = []
, settingFailFast = False
, settingIterations = OneIteration
, settingRetries = 3
, settingFailOnFlaky = False
, settingReportProgress = ReportNoProgress
, settingProfile = False
}
286 changes: 286 additions & 0 deletions test_resources/documentation.txt
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,286 @@
Usage: sydtest [--config-file FILE_PATH] [--random-seed | --seed INT] [--[no-]randomise-execution-order|--[no-]randomize-execution-order] [--jobs|--threads INT | --synchronous] [--max-size Int] [--max-success Int] [--max-discard Int] [--max-shrinks Int] [--[no-]golden-start] [--[no-]golden-reset] [--[no-]colour|--[no-]color] [FILTER [FILTER] | -f|--filter|-m|--match FILTER] [--[no-]fail-fast] [--continuous | --iterations INT] [--retries INTEGER] [--[no-]fail-on-flaky] [--progress | --no-progress] [--[no-]debug] [--[no-]profile]

All settings:
Show this help text
switch: -h|--help

Output version information
switch: --version

Path to the configuration file
option: --config-file FILE_PATH
env: SYDTEST_CONFIG_FILE FILE_PATH

Use a random seed for pseudo-randomness
switch: --random-seed
env: SYDTEST_RANDOM_SEED ANY

Seed for pseudo-randomness
option: --seed INT
env: SYDTEST_SEED INT
config:
seed: # any of
[ null
, random
, <integer> # 64 bit signed integer
]

Run test suite in a random order
switch: --[no-]randomise-execution-order|--[no-]randomize-execution-order
env: SYDTEST_RANDOMISE_EXECUTION_ORDER|SYDTEST_RANDOMIZE_EXECUTION_ORDER BOOL
config:
randomise-execution-order: # or null
<boolean>
config:
randomize-execution-order: # or null
<boolean>

How many threads to use to execute tests in asynchrnously
option: --jobs|--threads INT
env: SYDTEST_JOBS|SYDTEST_THREADS INT

Use only one thread, to execute tests synchronously
switch: --synchronous
env: SYDTEST_SYNCHRONOUS ANY

How parallel to run the test suite
config:
threads: # or null
<integer> # 64 bit unsigned integer

Maximum size parameter to pass to generators
option: --max-size Int
env: SYDTEST_MAX_SIZE Int
config:
max-size: # or null
<integer> # 64 bit signed integer
default: 100

Number of property test examples to run
option: --max-success Int
env: SYDTEST_MAX_SUCCESS Int
config:
max-success: # or null
<integer> # 64 bit signed integer
default: 100

Maximum number of property test inputs to discard before considering the test failed
option: --max-discard Int
env: SYDTEST_MAX_DISCARD Int
config:
max-discard: # or null
<integer> # 64 bit signed integer
default: 10

Maximum shrinks to try to apply to a failing property test input
option: --max-shrinks Int
env: SYDTEST_MAX_SHRINKS Int
config:
max-shrinks: # or null
<integer> # 64 bit signed integer
default: 100

Produce initial golden output if it does not exist yet
switch: --[no-]golden-start
env: SYDTEST_GOLDEN_START BOOL
config:
golden-start: # or null
<boolean>

Overwrite golden output
switch: --[no-]golden-reset
env: SYDTEST_GOLDEN_RESET BOOL
config:
golden-reset: # or null
<boolean>

Use colour in output
switch: --[no-]colour|--[no-]color
env: SYDTEST_COLOUR|SYDTEST_COLOR BOOL
config:
colour: # or null
<boolean>
config:
color: # or null
<boolean>

Filter to select parts of the test suite
argument: FILTER

Filter to select parts of the test suite
argument: FILTER

Filter to select parts of the test suite
option: -f|--filter|-m|--match FILTER

Stop testing when a test failure occurs
switch: --[no-]fail-fast
env: SYDTEST_FAIL_FAST BOOL
config:
fail-fast: # or null
<boolean>

Run the test suite over and over again until it fails, for example to diagnose flakiness
switch: --continuous

How many iterations of the suite to run, for example to diagnose flakiness
option: --iterations INT

The number of retries to use for flakiness diagnostics. 0 means 'no retries'
option: --retries INTEGER
env: SYDTEST_RETRIES INTEGER
config:
retries: # or null
<integer> # 64 bit unsigned integer

Fail when any flakiness is detected, even when flakiness is allowed
switch: --[no-]fail-on-flaky
env: SYDTEST_FAIL_ON_FLAKY BOOL
config:
fail-on-flaky: # or null
<boolean>

Report per-example progress
switch: --progress

Don't report per-example progress
switch: --no-progress

Turn on debug mode
switch: --[no-]debug
env: SYDTEST_DEBUG BOOL
config:
debug: # or null
<boolean>

Turn on profiling mode
switch: --[no-]profile
env: SYDTEST_PROFILE BOOL
config:
profile: # or null
<boolean>

Options:
-h|--help Show this help text
--version Output version information
--config-file Path to the configuration file
--random-seed Use a random seed for pseudo-randomness
--seed Seed for pseudo-randomness
--[no-]randomise-execution-order|--[no-]randomize-execution-order Run test suite in a random order
--jobs|--threads How many threads to use to execute tests in asynchrnously
--synchronous Use only one thread, to execute tests synchronously
--max-size Maximum size parameter to pass to generators default: 100
--max-success Number of property test examples to run default: 100
--max-discard Maximum number of property test inputs to discard before considering the test failed default: 10
--max-shrinks Maximum shrinks to try to apply to a failing property test input default: 100
--[no-]golden-start Produce initial golden output if it does not exist yet
--[no-]golden-reset Overwrite golden output
--[no-]colour|--[no-]color Use colour in output
FILTER Filter to select parts of the test suite
FILTER Filter to select parts of the test suite
-f|--filter|-m|--match Filter to select parts of the test suite
--[no-]fail-fast Stop testing when a test failure occurs
--continuous Run the test suite over and over again until it fails, for example to diagnose flakiness
--iterations How many iterations of the suite to run, for example to diagnose flakiness
--retries The number of retries to use for flakiness diagnostics. 0 means 'no retries'
--[no-]fail-on-flaky Fail when any flakiness is detected, even when flakiness is allowed
--progress Report per-example progress
--no-progress Don't report per-example progress
--[no-]debug Turn on debug mode
--[no-]profile Turn on profiling mode

Environment Variables:
SYDTEST_CONFIG_FILE FILE_PATH Path to the configuration file
SYDTEST_RANDOM_SEED ANY Use a random seed for pseudo-randomness
SYDTEST_SEED INT Seed for pseudo-randomness
SYDTEST_RANDOMISE_EXECUTION_ORDER|SYDTEST_RANDOMIZE_EXECUTION_ORDER BOOL Run test suite in a random order
SYDTEST_JOBS|SYDTEST_THREADS INT How many threads to use to execute tests in asynchrnously
SYDTEST_SYNCHRONOUS ANY Use only one thread, to execute tests synchronously
SYDTEST_MAX_SIZE Int Maximum size parameter to pass to generators default: 100
SYDTEST_MAX_SUCCESS Int Number of property test examples to run default: 100
SYDTEST_MAX_DISCARD Int Maximum number of property test inputs to discard before considering the test failed default: 10
SYDTEST_MAX_SHRINKS Int Maximum shrinks to try to apply to a failing property test input default: 100
SYDTEST_GOLDEN_START BOOL Produce initial golden output if it does not exist yet
SYDTEST_GOLDEN_RESET BOOL Overwrite golden output
SYDTEST_COLOUR|SYDTEST_COLOR BOOL Use colour in output
SYDTEST_FAIL_FAST BOOL Stop testing when a test failure occurs
SYDTEST_RETRIES INTEGER The number of retries to use for flakiness diagnostics. 0 means 'no retries'
SYDTEST_FAIL_ON_FLAKY BOOL Fail when any flakiness is detected, even when flakiness is allowed
SYDTEST_DEBUG BOOL Turn on debug mode
SYDTEST_PROFILE BOOL Turn on profiling mode

Configuration Values:
Seed for pseudo-randomness
seed:
# any of
[ null
, random
, <integer> # 64 bit signed integer
]
Run test suite in a random order
randomise-execution-order:
# or null
<boolean>
randomize-execution-order:
# or null
<boolean>
How parallel to run the test suite
threads:
# or null
<integer> # 64 bit unsigned integer
Maximum size parameter to pass to generators
default: 100
max-size:
# or null
<integer> # 64 bit signed integer
Number of property test examples to run
default: 100
max-success:
# or null
<integer> # 64 bit signed integer
Maximum number of property test inputs to discard before considering the test failed
default: 10
max-discard:
# or null
<integer> # 64 bit signed integer
Maximum shrinks to try to apply to a failing property test input
default: 100
max-shrinks:
# or null
<integer> # 64 bit signed integer
Produce initial golden output if it does not exist yet
golden-start:
# or null
<boolean>
Overwrite golden output
golden-reset:
# or null
<boolean>
Use colour in output
colour:
# or null
<boolean>
color:
# or null
<boolean>
Stop testing when a test failure occurs
fail-fast:
# or null
<boolean>
The number of retries to use for flakiness diagnostics. 0 means 'no retries'
retries:
# or null
<integer> # 64 bit unsigned integer
Fail when any flakiness is detected, even when flakiness is allowed
fail-on-flaky:
# or null
<boolean>
Turn on debug mode
debug:
# or null
<boolean>
Turn on profiling mode
profile:
# or null
<boolean>

Loading

0 comments on commit 64d34d8

Please sign in to comment.