Create AWS S3 Bucket for Snapshots and Blocklogs

[bhazzard]

Need a place to store and upload both snapshots and blockslogs. S3 bucket to be several TB in size.
Once the files are uploaded, we need to copy over files into our ephemeral hosts for replay tests.
We'll need permissions for read and write access to S3.

[ericpassmore]

We need a place to store and upload both snapshots and blockslogs. This S3 bucket will be several TB in size.

Once the files are uploaded we will need to copy over files into our ephemeral hosts for replay tests. We'll need permissions for read and write access to S3.

[BenjaminGormanPMP]

Per @ericpassmore target completion for @kj4ezj is 08/01/23.

[kj4ezj]

An S3 bucket called chicken-dance was created in the chickens-prod AWS account from issue 1420. This bucket is encrypted with an AWS-managed key, all IAM users (as of this writing) have full read/write access, and public access to the bucket is blocked. Enabling public read-only access will make it easier to build out this system. The bucket of snapshot data was public-read at B1, but this requires special consideration and may incur additional costs for community access so we should discuss internally if that approach is appropriate here.

I have also created a skeleton folder hierarchy in the bucket to show how we have organized this bucket in the past, to great success. This may or may not be the best way to move forward for this project.

You will find three files at /mainnet/2000000/ - two placeholders for a blocks.log and snapshot.bin as an example, with a third metadata.json file containing an example of information about a hypothetical snapshot and the software used to collect it.

{
    "argv": ["nodeos", "-e", "-d", "/mnt/data", "--config-dir", "/mnt/data/config", "--disable-replay-opts", "--snapshot", "/mnt/data/snapshots/snapshot.bin", "--plugin", "eosio::state_history_plugin", "--p2p-peer-address", "develop.mainnet.chickens.example.com"],
    "blocklog": {
        "begin": 1000012,
        "end": 2000008,
        "version": 3
    },
    "chain": {
        "id": "aca376f206b8fc25a6ed44dbdc66547c36c6c33e3a119ffbeaef943642f0e90",
        "name": "main"
    },
    "git": {
        "branch": "develop",
        "commit": "5b6226b97806d8fd2a1d7c3c17bf2f629855ee02",
        "tag": null
    },
    "height": 2000000,
    "lib": {
        "id": "e70aaab8997e1dfce58fbfac80cbbb8fecec7b99cf982a9444273cbc64c41473",
        "number": 2000008,
        "time": 1691102734236
    },
    "plugin": [
        "eosio::chain_plugin",
        "eosio::chain_api_plugin",
        "eosio::producer_api_plugin",
        "eosio::state_history_plugin"
    ],
    "repo": "https://github.com/AntelopeIO/chickens",
    "version": "v2.2.0-rc1-62b8c9964cdf9a3b6afc5ae7cd562468920ef700"
}

This information was recreated from memory, but it is an extremely close approximation of the types of useful data we have collected about snapshots in the past. In particular, the information about what the snapshot contains, what software version collected it, and what plugins were running on nodeos at the time are extremely useful for decision-making with respect to the direction of the larger test system and, crucially, cannot currently be recovered after-the-fact. I am happy to discuss why we collected each or all of these pieces of information in the past, and why I believe we should do it again.