Automated Replay Testing (Chicken Dance) - MVP

[stephenpdeos]

Short Description

As an Antelope Core developers, I want to be able to run an automated replay test from genesis within 1 day, to confirm that consensus is maintained and uptime reliability continues.

What

To verify that the consensus mechanism is intact, we can run a full replay from genesis and confirm that every block can be applied without error, and that we achieve the expected state after replay. It may be valuable to run multiple replay tests with different options (for example: supported CPU features, using various WASM runtimes). This would allow us to gain confidence that consensus is robust under many different environmental situations.

Why

Running a full replay from genesis manually would be a months-long endeavor. We need to be able to run a replay test within 1 day in an automated and repeatable fashion.

Considerations

• These tests may be expensive to run
• If developed correctly, it should be trivial to add other Antelope chains

Requirements

• A developer can "hit a button" to launch a replay test for a particular commit, and see if completed within a day.
• Ability to specify command line options for test (including WASM runtime)

Non-requirements (Potential Future Version of Chicken Dance)

• Ability to easily run with history from other Antelope chains (Jungle, WAX, TELOS, etc)
• Ability to run on different hardware to simulate a variety of environmental factors (CPU features)

Tasks

Beta Give feedback

1. Chicken Dance Solution Design AntelopeIO/leap#1353
   👍 lgtm +1
2. Create AWS S3 Bucket for Snapshots and Blocklogs AntelopeIO/leap#1419
   👍 lgtm +1
   
3. Chicken Dance AWS Account AntelopeIO/leap#1420
   👍 lgtm +1
   
4. Gather snapshots into 120 roughly equal chunks AntelopeIO/leap#1421
   triage +1
   
5. Gather blockslog from Genesis to Current AntelopeIO/leap#1441
   triage +1
   
6. Create manifest for snapshots and block positions AntelopeIO/leap#1422
   triage +1
   
7. Create two AWS hosts for master syncing AntelopeIO/leap#1423
   triage +1
   
8. Create two AWS hosts for master syncing AntelopeIO/leap#1427
   triage +1
   
9. Test spin up snapshot node #10 and test sync AntelopeIO/leap#1429
   triage +1
   
10. Write validate scripts AntelopeIO/leap#1430
    triage +1
    
11. Run validation script AntelopeIO/leap#1431
    triage +1
    
12. Test Run a replay on Leap v4.04 (run on 10 slices to speed up replay) AntelopeIO/leap#1426
    triage +1
    
13. Create two AWS hosts for master syncing AntelopeIO/leap#1424
    triage +1
14. Expand IAM for Zero-Trust S3-EC2 Integration replay-test#14
    DevOps Infrastructure Security +3
    

[bhazzard]

@BenjaminGormanPMP Next step is task breakdown.

COULD depend on Pinned Builds (but doesn't have to, it would just be ideal).

[bhazzard]

@bhazzard Create issue for Solution Design

[bhazzard]

@bhazzard Create issue for Solution Design

Done: AntelopeIO/leap#1353

[bhazzard]

Relevant past work: https://hackmd.io/DFtefcHjRoGxU03N_86g0g