Ecosystem test environment

[bkchr]

When doing runtime upgrades of the relay chains, we want to ensure that nothing breaks unintentionally. There are situations where things need to change, because of security considerations etc. However, even in these cases it would be nice that teams are informed before the relay chain is upgraded. Recently we have seen that changes to XCM broke messages between parachains in the ecosystem. This was detected on Kusama, so Kusama actually has done its job, but these things could also be detected before they are on Kusama to make everyones lifes easier. Thus, I (and others) think that we need some kind of test environment where we things can be tested automatically.

I think the best would be that we start with chopsticks for creating this test environment. With chopsticks it should be possible to only run the runtimes and also to fork off from the latest on chain state. Forking off from the latest on chain state brings us closer to the real world. We will probably need some kind of way to configure chopsticks to be able to talk to all these parachains. Parachains teams on their own should add their parachains to this repo to have them run as part of the tests. Each parachain should also be able to declare test cases. These test cases will probably be mainly XCM or other cross chain functionality they are using.

The environment should automate as much as possible. It could be for example running every night, pulling in the latest fellowship runtimes or even the runtimes from all the parachains. At the end of each run the relevant people should be notified about their test cases failing. We may also want to include it as part of this repo CI at some point. There are a lot of possibilities on what we could do.

I think the most complicated part is to start setting up this stuff. Delivering the first test cases and after this it should be fairly simple to add new test cases/chains. In the best case we really achieve some network wide test environment to ensure that things do not break that often anymore. I would also like to transform this later into a bounty, so that we can find a team to implement this.

CC @xlc @joepetrowski

[xlc]

you are basically describing https://github.com/AcalaNetwork/e2e-tests

some features:
always run with latest onchain state
can test xcm interactions from as many chains as needed
already support many chains
wasm override so it can be used to verify new runtime compatibility

unfortunately it is not very well maintained atm due to resource limitations on our side and not enough tests so it didn’t caught the last KSM issue. but it only takes 30s to add such test

[bkchr]

Yeah nice, I was already assuming that you have something similar already.

unfortunately it is not very well maintained atm due to resource limitations on our side and not enough tests so it didn’t caught the last KSM issue.

Yeah these are the things we can improve. If you say that this is already a good basis, we can probably create a bounty for improving this and making this more "well known".

[joepetrowski]

We started a project many months ago to do exactly this in https://github.com/paritytech/polkadot-integration-tests. This was set up so that anyone could add their parachain runtimes and add tests for them. We also have a lot of tests in Typescript/YAML here for the system parachains. No, it's not perfect, it doesn't fork off chain state, but it allows a lot of test cases. There could have been a case for para-to-para reserve transfer in there. Have also written out a lot of requirements for the future of this in paritytech/roadmap#56

Unfortunately there are always heated discussions about Rust vs. Typescript vs. YAML, XCM Emulator vs. Chopsticks, etc. and we never make progress on these.

We know that XCM Emulator is not perfect, and Chopsticks is really ideal in being able to fork off the actual chain state, especially when it comes to testing migrations. But when it comes to these things like reserve transfers breaking, we know that Emulator can test for them.

I really don't care if people use Chopsticks or Emulator. But this is just a continuation of people complaining about not having tests instead of writing tests. As @xlc said, it takes 30s to add such a test, OK so somebody do it.

[bkchr]

I really don't care if people use Chopsticks or Emulator. But this is just a continuation of people complaining about not having tests instead of writing tests. As @xlc said, it takes 30s to add such a test, OK so somebody do it.

I think the problem being that we don't have a central place for this. I knew about your issue, but I have never seen this repo before for example. I mean @xlc and co apparently have done a lot of the groundwork, now we just need to take this and make it more easier usable and write more tests. Let's go and give tips per test. If it only takes 30 seconds, it should be quite simple to add new tests. However, parachain teams are probably also interested in their stuff not breaking and they could provide some tests as well.

[acatangiu]

For this kind of separate/central test repo, xcm-emulator or any other test framework that needs to build code cannot work because of code deps incompatibilities. We'll never realistically have all chains in the ecosystem all anchored with compatible versions of polkadot-sdk (or other upstream repos) crates.

We need some framework modeled like https://github.com/AcalaNetwork/e2e-tests where tests run using on-chain state, released binaries/wasm-blobs, custom provided binaries/wasm-blobs. Each binary/wasm-blob needs to be updated (kept up-to-date) by the owning team/project.

As long as we can fork from some customisable on-chain state we can build any custom test scenario. E.g. we can test a relay chain (or parachain) upgrade on a spawned test fork before doing it on live chain (do the upgrade then run battery of existing tests).