diff --git a/applications/sarp-2-swdesign.md b/applications/sarp-2-swdesign.md index a82d1d3ffe5..42ff74f83c2 100644 --- a/applications/sarp-2-swdesign.md +++ b/applications/sarp-2-swdesign.md @@ -6,7 +6,7 @@ ## Project Overview :page_facing_up: -This is the follow up to our initial [research proposal](https://github.com/w3f/Grants-Program/blob/master/applications/sarp-basic-functionality.md), that we delivered [here](https://github.com/w3f/Grant-Milestone-Delivery/pull/880). The goal of this work package is to evaluate different software designs to implement a static code analysis on substrate pallets with MIRAI. Furthermore we want to investigate issues and bugs, we found in MIRAI in the previous work package. +This is the follow up to our initial [research proposal](https://github.com/w3f/Grants-Program/blob/master/applications/sarp-basic-functionality.md), that we delivered [here](https://github.com/w3f/Grant-Milestone-Delivery/pull/880). The goal of this work package is to implement a first version of SARP (Static Analysis Tool for Runtime Pallets). ### Overview @@ -18,13 +18,13 @@ Our team has a good understanding of substrate and Rust. We are still getting st ### Project Details -We will base our work on [MIRAI](https://github.com/facebookexperimental/MIRAI/) and extend it with checks on substrate pallets. For details see the [Development Roadmap](#development-roadmap-nut_and_bolt) +Our work is based on [MIRAI](https://github.com/facebookexperimental/MIRAI/), it extends it with checks on substrate pallets. For details see the [Development Roadmap](#development-roadmap-nut_and_bolt) ### Ecosystem Fit The tool will help any team developing substrate pallets. It can further be integrated in the CI pipelines of the teams, providing a continuous quality check on the pallet code. -In the long term it could be interesting to connect the work done here with the new emerging auditing DAOs (like [QRUCIAL DAO](https://github.com/w3f/Grants-Program/blob/master/applications/QRUCIAL_DAO.md)). +In the long term it could be interesting to connect the work done here with new emerging auditing DAOs (like [QRUCIAL DAO](https://github.com/w3f/Grants-Program/blob/master/applications/QRUCIAL_DAO.md)). ## Team :busts_in_silhouette: @@ -83,10 +83,7 @@ In a first research project we investigated, if MIRAI can be used for static cod - Check of [incorrect origin](https://github.com/scs/MIRAI/blob/Milestone1_Research/substrate-examples/pallet_template/README.md) in the [substrate node template](https://github.com/substrate-developer-hub/substrate-node-template/tree/e0c480c0f322d0b0d1b310c93fa646fc0cfdd2df/pallets/template) - Validation of [unsigned transactions](https://github.com/scs/MIRAI/blob/Milestone1_Research/substrate-examples/offchain-worker/README.md) for substrate's [offchain worker example](https://github.com/paritytech/substrate/tree/ea9ce4c0af36310c0b0db264ab12cf4766b83750/frame/examples/offchain-worker) -The overall conclusion was, that it is best to run the analysis only on the newly written pallet code, but not on the code generated by substrate's macros. To facilitate this a detailed analysis of different software designs has to be evaluated. - - -The full documentation of our findings can be found [here](https://github.com/scs/MIRAI/tree/Milestone1_Research/substrate-examples). +The overall conclusion was, that it is best to run the analysis only on the newly written pallet code, but not on the code generated by substrate's macros. The full documentation of our findings can be found [here](https://github.com/scs/MIRAI/tree/Milestone1_Research/substrate-examples). ## Development Roadmap :nut_and_bolt: @@ -95,58 +92,107 @@ The full documentation of our findings can be found [here](https://github.com/sc ### Overview -- **Total Estimated Duration:** 1 month -- **Full-Time Equivalent (FTE):** 1 FTE -- **Total Costs:** 30,000 USD +- **Total Estimated Duration:** 4 months +- **Full-Time Equivalent (FTE):** 0.8 FTE +- **Total Costs:** 100'000 USD +In our previous work, we found the following problems: -### Milestone 1 Software Design & Bug Fixes +1. **Crashes and timeouts of MIRAI** Certain pieces of substrate code lead to crashes of MIRAI. In other cases, parts of the code are not analyzed/do not produce warnings, because MIRAI runs into a timeout before reaching this code. -- **Estimated duration:** 1 month -- **FTE:** 1 -- **Costs:** 30,000 USD +2. **Complexity due to substrate macros** The main reason for crashes and timeouts, was caused by substrate macros, adding a lot of complexity to the code in the background. Ideally SARP only analyzes the newly written code of a pallet. -In our previous work, we found the following problems: +3. **Invasiveness of tag analysis** The code we wrote in our PoC is very invasive and changes the code of the pallet. This is not practical for end-users. Ideally the user doesn't need to change anything on their code, or at least the changes should be very simple. -1. **Crashes and timeouts of MIRAI** Certain pieces of substrate code lead to crashes of MIRAI. In other cases, parts of the code are not analyzed/do not produce warnings, because MIRAI runs into a timeout before reaching this code. Because of this, our examples are rather simple and we couldn't add and check tags at the locations we originally wanted to. +To approach these issues, we plan to implement two core features in SARP: -2. **Complexity due to substrate macros** The main reason for crashes and timeouts in our examples, was caused by substrate macros, adding a lot of complexity to the code in the background. Ideally SARP only analyzes the newly written code of a pallet. +1. **Mock substrate functions**, that increase the complexity, to improve 1. and 2. +2. **Automatically generate annotations and checks**, that are needed for the control flow analysis, to improve 3. -3. **Invasiveness of tag analysis** The code we wrote in our PoC is very invasive and changes the code of the pallet. This is not practical for end-users. Ideally the user doesn't need to change anything on their code, or at least the changes should be very simple. +The planned workflow of the tool is: +1. Create a copy of the analyzed pallet. +2. For the copy: + 1. Adjust dependencies, so that the mocked substrate libraries are used + 2. Add annotations and checks, needed for the analysis +3. Run MIRAI on the adjusted copy. + +#### Vulnerabilities +The tool will cover the following cases of vulnerabilities: +- [incorrect origin](https://github.com/bhargavbh/MIRAI/blob/main/substrate_examples/incorrect-origin/description.md) of dispatchable functions +
We will provide documentation, example and tests for the described case of enforcing a custom origin type. +- [unsigned transaction](https://github.com/bhargavbh/MIRAI/blob/main/substrate_examples/unsigned-transaction/description.md) validation +
We will provide documentation, example and tests for enforcing checks on all input parameters into a function. Specifically for the case of validating unsigned transactions, but not limited to this. + +We plan to do a detailed analysis on interesting vulnerabilities in the [future](#future-plans). -To address 2. and 3. we plan to evaluate different software designs. These will be part of our deliverables and we plan to discuss these with Parity and/or Web3 Foundation. The challenge here is the non-invasiveness of the solution. Specifically we plan to look into the following questions: -1. Can we implement a preprocessing, that automatically annotates the pallet code for analysis in MIRAI? -2. Can MIRAI be tweaked to abstract out low-level details in the flow of function calls? -3. Is it possible to separate the newly written pallet logic from the macro logic? If so, is it possible, without changing the logic or adding artificial code (as the macros also connect the different functions of a pallet)? -4. Are there good combinations of the above approaches? -To address 1. we want to further analyze timeouts and crashes in MIRAI. Possibly they can be resolved by bugfixes in MIRAI. If not, we need to find workarounds. -Apart from the documentation of our analysis, we will deliver a first prototype version of our tool. +### Milestone 1 - Basic Setup & Mocking of Substrate Libraries +- **Estimated duration:** 2 months +- **FTE:** 0.7 +- **Costs:** 47'000 USD + +With the first milestone we want to do the basic setup of the project, integrate a first set of substrate mocks and deliver a first runnable tool. + +| Number | Deliverable | Specification | +|--------|-----------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| 0a. | License | MIT | +| 0b. | User Documentation | We will provide a basic **tutorial** that explains how the tool can be used on a substrate pallet. | +| 0c. | Testing and Testing Guide | Core functions will be fully covered by comprehensive unit tests to ensure functionality and robustness. In the guide, we will describe how to run these tests. | +| 0d. | Docker | We will provide a Dockerfile(s) that can be used to test all the functionality delivered with this milestone. | +| 1. | Tool | A first version of SARP, that runs the above described workflow, without the annotations generator. The tool will cover the described [vulneraribilities](#Vulnerabilities). Annotations need to be added manually. | +| 2. | Engagement | Engage with teams at Web3 Foundation and Parity for prioritization. | + +### Milestone 2 - Annotations Generator +- **Estimated duration:** 1.5 month +- **FTE:** 0.8 +- **Costs:** 37'000 USD + +In the second milestone we implement the annotations generator (needed for step 2.ii. of the workflow process). + +| Number | Deliverable | Specification | +|--------|-----------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| 0a. | License | MIT | +| 0b. | User Documentation | We will provide a basic **tutorial** that explains how the tool can be used on a substrate pallet. | +| 0c. | Testing and Testing Guide | Core functions will be fully covered by comprehensive unit tests to ensure functionality and robustness. In the guide, we will describe how to run these tests. | +| 0d. | Docker | We will provide a Dockerfile(s) that can be used to test all the functionality delivered with this milestone. | +| 1. | Tool | We will implement an annotations generator and integrate it in SARP. The annotations generator will cover annotations needed for the described [vulneraribilities](#Vulnerabilities). | +| 2. | Engagement | Engage with teams at Web3 Foundation and Parity for prioritization. | + +### Milestone 3 - Complete Tool with Documentation +- **Estimated duration:** 0.5 month +- **FTE:** 1 +- **Costs:** 16'000 USD + +In the final milestone we finetune the tool, making sure it runs on productive pallet code and write the final documentation. -#### Deliverables +| Number | Deliverable | Specification | +|--------|-----------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| 0a. | License | MIT | +| 0b. | User Documentation | We will provide a basic **tutorial** that explains how the tool can be used on substrate pallets. | +| 0c. | Testing and Testing Guide | Each check, that the tool can perform, will be covered by comprehensive unit tests to ensure functionality and robustness. In the testing guide, we will describe how to run these tests.
The tests will further include running SARP on 5 pallets from the [frame pallets](https://github.com/paritytech/substrate/tree/master/frame) | +| 0d. | Docker | We will provide a Dockerfile(s) that can be used to test all the functionality delivered with this milestone. | +| 1. | Tool | We will finetune SARP, with the help of real world examples. The real world examples we will be taken from [substrate's frame pallets](https://github.com/paritytech/substrate/tree/master/frame). | +| 2. | Documentation | Technical documentation of the tool, incl. reasoning on the design decisions. | +| 3. | Engagement | Engage with teams at Web3 Foundation and Parity for prioritization. | -| Number | Deliverable | Specification | -|--------|-----------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| 0a. | License | MIT | -| 0b. | User Documentation | We will provide a basic **tutorial** that explains how the tool can be used on the examples provided. | -| 0c. | Testing and Testing Guide | We will *not* provide a test suite with this milestone, but documentation on how to run our examples as in 0b. | -| 1. | Tool | We will provide a prototype version of the tool. Following the approach, decided in 3. We will provide examples for applying the tool to 3 of the following 5 vulnerability classes: | -| 2. | Documentation | Technical documentation of the tool, incl. reasoning on the design decisions. | -| 3. | Engagement | We will discuss different solutions and their implications with Web3 Foundation and/or Parity. For this we will document each approach with | -| 4. | Analysis of errors in MIRAI | We will document each error we encounter in MIRAI (specifically crashes and faulty analyses) with information on: This analysis will include the problems we discovered in the previous work package, but haven't analyzed yet (see [1](https://github.com/scs/MIRAI/tree/Milestone1_Research/substrate-examples/pallet_template#open-issues) and [2](https://github.com/scs/MIRAI/tree/Milestone1_Research/substrate-examples/offchain-worker#open-issues)).

We will report each issue in the [MIRAI repository](https://github.com/facebookexperimental/MIRAI), resp. provide a PR there. | +### Limitations +#### Mocking +Static code analysis, in its nature, is a problem of exponential time complexity. As such, it needs good heuristics to deliver useful results. Our strategy is to mock the library code, that adds the most complexity and is used the most. For this delivery we plan to take an educated guess, on what to mock. Naturally this will not cover all cases. Especially with this first version, we expect that there will be many cases, where a vulnerability is not found or a false positive is returned. +Due to this limitation, we commit ourselves to 5 working real world examples, taken from the [frame pallets](https://github.com/paritytech/substrate/tree/master/frame). That means, the analysis will not crash and provide only few false positives. Note, that this is currently not possible, due to the induced complexity by the substrate macros. +#### Annotations Generator +The annotations generator will generate the needed annotations for the examples we provide for the described [vulnerabilities](#vulnerabilities). We expect, that there will be situations, where the user has to add annotations themselves. ## Future Plans -1. Decide on vulnerabilities for an MVP. +1. Specify and prioritize vulnerabilities for roadmap.
For this we plan to engage with Web3 Foundation / Parity and auditing companies such as [OtterSec](https://osec.io/) or [FYEO](https://www.fyeo.io/). -2. Implement a first simple version of the tool, together with tests and documentation. 3. Improve the usability, by providing * means to surpress warnings * a comprehensive user tutorial, incl. documentation on the risks of each vulnerability