WeTEE milestone 3 Delivery

[BurnWW]

Milestone Delivery Checklist

• The milestone-delivery-template.md has been copied and updated.
• This pull request is being made by the same account as the accepted application.
• I have disclosed any and all sources of reused code in the submitted repositories and have done my due diligence to meet its license requirements.
• In case of acceptance, invoices must be submitted and payments will be transferred to the Polkadot AssetHub and/or fiat account provided in the application.
• The delivery is according to the Guidelines for Milestone Deliverables.

Link to the application pull request: w3f/Grants-Program#2065

[PieWol]

Hey @BurnWW,
thanks for the delivery.

Do you have any advice on how to go through your testing guide if I don't have a physical machine with a required intel chip that features sgx? I tried to check numerous cloud providers but couldn't find anyone advertised with sgx enabled. Looking forward to hearing from you.

[BurnWW]

hello @PieWol, Thank you very much for your reminder.
I’ve conducted a comprehensive research on the Azure operating environment, and I found that Azure fully meets the requirements for software debugging and usage.
This is the official tutorial and documentation https://learn.microsoft.com/en-us/azure/confidential-computing/quick-create-portal

Free trial accounts do not have access to the VMs in this tutorial. Please upgrade to a Pay-As-You-Go subscription.

This is the configuration for the virtual machine I created.

Select DC2s_v2 size

This is the result of my test in azure DC2s_v2 virtual machine.

[PieWol]

Thank you @BurnWW for the advice. I'll get me a cloud provider with the same features as you showed in the example. 👍

[PieWol]

Hey @BurnWW ,

Just as a quick follow up before I proceed with the setup. In azure it explicitly states that TEE is only available in the most strict security setting (confidential virtual machines) but in your setup you are using the standard security settings.

It even seems to be impossible to boot up an ubuntu instance with TEE capabilities in azure.

I wonder why you think that the TEE is actually in place with the setup tutorial you provided. Is there an integrity check of the machine in any place of your code that somewhat proves that the TEE is active?

Looking forward to your feedback on this :)

[BurnWW]

@PieWol，hello， I fully understand your doubts, and I have also had similar questions before.
confidential virtual machines refers to starting a virtual machine running on either SEV or TDX technology, which is a form of virtualization at the operating system level. In our upcoming plans, we will also support SEV and TDX technologies and be able to provide containerized services as well as unified TEE attestation. Currently, we do not support SEV and TDX.

In our current version, we provide a thread-level Trusted Execution Environment (TEE base on SGX) solution. All TEE initialization and authentication are based on SGX technology, hence we opt for regular CPUs in the DC2s_v2 series that support SGX rather than choosing an operating system-level TEE virtual machine（base on SEV/TDX）. What SGX requires is CPU support for SGX, rather than the entire host operating system being dedicated to confidential computing. So here we do not choose confidential virtual machines but instead opt for standard

Below is the documentation for DC2s_v2
https://learn.microsoft.com/en-us/azure/virtual-machines/dcv2-series?WT.mc_id=itopstalk-ch9-pierrer

This is the underlying technology we use, ego and gramine, which are two very popular SGX technologies, By obtaining the report through them and using the code from this milestone, users can actually verify that the program is indeed running under the protection of SGX.

and here are the relevant documents.

https://gramine.readthedocs.io/en/stable/attestation.html?highlight=report
https://docs.edgeless.systems/ego/reference/attest

[PieWol]

Hey @BurnWW,
thanks for clarifying.
I'm trying to run the local setup but run into an issue with the pods. The device plugin pod doesn't work and I can't seem to figure out the reason for this. Do you have any idea on how to fix this? I listed the output below. I appreciate your input.

worker$ kubectl get pod -A
NAMESPACE       NAME                                         READY   STATUS             RESTARTS      AGE
worker-system   worker-controller-manager-7577d5b96c-98srf   0/2     Pending            0             67s
kube-system     local-path-provisioner-6c86858495-klkk2      1/1     Running            0             67s
worker-addon    sgx-pccs-api-67c56bc9d5-lfmq9                1/1     Running            0             67s
kube-system     coredns-6799fbcd5-v8rq4                      1/1     Running            0             67s
worker-addon    wetee-node-5d4bf8bc57-9slpn                  1/1     Running            0             67s
worker-addon    wetee-dapp-58b79f7c8f-lv889                  1/1     Running            0             67s
kube-system     svclb-traefik-89b1bccc-tvm5t                 2/2     Running            0             64s
kube-system     traefik-f4564c4f4-rgmwp                      1/1     Running            0             67s
kube-system     metrics-server-67c658944b-m79tr              1/1     Running            0             66s
kube-system     sgx-device-plugin-ds-5pk29                   0/1     CrashLoopBackOff   3 (20s ago)   65s

[PieWol]

The error output I have gotten is the following err: go command required, not found: exec: "go": executable file not found in $PATH: stderr:

Given I have modified the PATH as given in the documentation I'm wondering what could have gone wrong.

[BurnWW]

@PieWol, Thank you very much for your reply.

kube-system sgx-device-plugin-ds-5pk29 0/1 CrashLoopBackOff 3 (20s ago) 65s

We haven’t come across this kind of issue before. Would it be possible for you to follow the steps below to give us more details? We are in urgent need of the results of the following command.

# step1 check sgx support
$ cpuid -1 -l 0x7 | grep SGX
SGX: Software Guard Extensions supported = true
SGX_LC: SGX launch config supported      = true

# step2 check host sgx device
$ ls /dev/sgx
enclave  provision

# step3 check sgx-device start error
> The pod name might have been changed，You can retrieve the information again using `kubectl get pods -A`
$ kubectl describe pod -n kube-system sgx-device-plugin-ds-5pk29

# step4 check sgx-device run error
> The pod name might have been changed，You can retrieve the information again using `kubectl get pods -A`
$ kubectl logs -n kube-system sgx-device-plugin-ds-5pk29

The error output I have gotten is the following err: go command required, not found: exec: "go": executable file not found in $PATH: stderr:

We are in urgent need of the results of the following command.

$ ls /usr/lib/go-1.21/
api  bin  doc  go.env  misc  pkg  src  test  VERSION

$ echo $PATH
/home/wetee/Soft/jdk/bin:/home/wetee/.vscode-server/bin/863d2581ecda6849923a2118d93a088b0745d9d6/bin/remote-cli:/home/wetee/Soft/jdk/bin:/home/wetee/.cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/wetee/Soft/flutter/bin:/home/wetee/Soft/go//bin:/home/wetee/go/bin:/home/wetee/bin:/home/wetee/go/bin:/home/wetee/.local/bin:/home/wetee/.cargo/bin:/opt/occlum/build/bin:/opt/occlum/sgxsdk-tools/bin:/opt/occlum/toolchains/gcc/bin:/opt/edgelessrt/bin:/home/wetee/Android/Sdk/tools:/home/wetee/Android/Sdk/tools/bin:/home/wetee/Android/Sdk/platform-tools:/home/wetee/Soft/flutter/bin:/home/wetee/Soft/go//bin:/home/wetee/go/bin:/home/wetee/bin:/home/wetee/go/bin:/home/wetee/.local/bin:/home/wetee/.cargo/bin:/opt/occlum/build/bin:/opt/occlum/sgxsdk-tools/bin:/opt/occlum/toolchains/gcc/bin:/opt/edgelessrt/bin:/home/wetee/Android/Sdk/tools:/home/wetee/Android/Sdk/tools/bin:/home/wetee/Android/Sdk/platform-tools:/usr/lib/go-1.21//bin

[BurnWW]

@PieWol hello
I purchased a virtual machine on Azure and recorded a video of the entire testing process, hoping that it will be useful for verification.
https://youtu.be/sg-qzH9HPMo
https://youtu.be/NHbiF-y9WZY

[PieWol]

Thank you @BurnWW,
thats very helpful. I will give it a new try with a fresh machine and follow your steps in the video. In case I run into the same issue I will give you the info of the commands you requested. Thanks a lot for your quick responses so far.

[PieWol]

Hi @BurnWW ,
well then. My mistake for not having this checked out earlier. I get the following output

cpuid -1 -l 0x7 | grep SGX
      SGX: Software Guard Extensions supported = false
      SGX_LC: SGX launch config supported      = false

I'm sure this is the reason for the crashing container issue. I will boot up a variety of azure vms in standard security mode to figure out if I can get any that features SGX. I was following your setup instructions and picked the vm with the same name. I'll get back to you shortly.

This is the config I used until now:

Operating system:
Linux (ubuntu 20.04)
Size:
Standard DS2 v2 (2 vcpus, 7 GiB memory)

[PieWol]

My bad. I just realized I was choosing the wrong VM config. Sorry for the inconvenience. It's working now.

[PieWol]

Hey @BurnWW , looking for your feedback once more. I was able to complete the procedure as given in the documentation.
Sadly I'm getting the following error when trying to launch my new confidential service within the dApp. The error is visible in the LOG section. Would you mind explaining why this error exists and is also visible in your youtube demo run yet seems to have no effect on the success of the demo? Thanks :)

EGo v1.4.1 (8b99356398dd3bcb5f74e5194d20ce421f607404)
EGo v1.4.1 (8b99356398dd3bcb5f74e5194d20ce421f607404)
[erthost] loading enclave ...
[erthost] entering enclave ...
[ego] starting application ...
WorkerAddr: https://wetee-worker.worker-system.svc.cluster.local:8883
GetKey: Key not found. Generating and storing a new Key open /wetee/sid: no such file or directory
Start entry secret listening https://0.0.0.0:8888 ...
Secrets: &{map[] map[:]}

[BurnWW]

@PieWol Thank you for your reply.

GetKey: Key not found. Generating and storing a new Key open /wetee/sid: no such file or directory

This log entry is not an error; the key created in this log is provisioned for a future feature. Each confidential application will generate its own sr25519 key, and we envision that in the future, these confidential applications will use this key to authenticate their identities and eventually invoke on-chain functions of pallets and ink! smart contracts. At present, this key has no practical use, so it’s safe to ignore this particular log entry.

The log appears within the code of this file
https://github.com/wetee-dao/libos-entry/blob/main/util/key.go

[PieWol]

Hey @BurnWW,
my evaluation is complete. I have a question regarding the documentation which you will find in the notes. Also looking forward to your reply to my question in the post directly above. Minor things.

Here is the link to the evaluation.

[BurnWW]

@PieWol Thank you for the reminder, this is indeed a bug because our project is still in the POC stage. We have been constantly restarting services and rebuilding the entire k3s cluster. We were not aware that each time the blockchain container restarts, it automatically starts again, and the container files of substrate are not being persisted.

This will cause all the containers to fail to run properly.

This is my modification to the yaml file

https://github.com/wetee-dao/worker/blob/main/hack/chain.yaml

--------
          volumeMounts:
            - name: wetee-chain
              mountPath: /tmp/chain
      volumes:
        - name: wetee-chain
          hostPath:
            type: DirectoryOrCreate
            path: /mnt/chain

Get the latest code for worker (https://github.com/wetee-dao/worker/) by running git pull

Testing new effects requires updating the container of blockchain nodes.

$ git pull
$ kubectl replace -f ./hack/chain.yaml 
$ make undeploy 
$ make deploy IMG=wetee/worker:2024-03-27-12_57

You need to re-execute the operations in the following document

https://github.com/wetee-dao/chain/blob/main/docs/grants/milestone-3-documentation.md#1-as-cluster-start-quick-mint-the-normal-mining-process-has-been-validated-in-m2

after which, whether you restart your computer or k3s, all contents within the cluster will automatically recover, including all TEE applications which will also be restarted.

If it’s a recreated cluster, this bug will no longer exist, and the programs will continue to run normally after the server restarts.

[BurnWW]

@PieWol I have placed the check step of cpuid | grep SGX at the very beginning of the process.

https://github.com/wetee-dao/chain/blob/main/docs/grants/milestone-3-documentation.md#check-intel-sgx-support

[PieWol]

Hey @BurnWW , thanks for delivering the changes.

I wonder why you have created a documentation dedicated to the milestone delivery but offer so little general documentation in the repo itself? Or is their additional documentation to the readme within the same repo?

I assume it is because you are in the POC stage without having an actual live network that could be used to deploy TEE resources towards. Still for the current stage of development I think the documentation you created for the milestone delivery could also be used as regular documentation right now. Given you already put in the work anyways. This would allow the public to try out your tech locally. Linking the article at the start of the documentation / readme would be great too.

[BurnWW]

@PieWol hello
We also maintain a document on gitbook that is quite similar to the one delivered by grant. Here’s the address of the document we’re currently maintaining: https://wetee.gitbook.io/docment. This document isn’t particularly detailed at present, but that will change soon as we’re actively thinking and planning, identifying our issues, and formulating subsequent plans:

• Develop deployment scripts for miners akin to those of k3s, create a UI specifically for miners (which will be entirely distinct from the current user-end UI), enabling miners to operate/view nodes at any time;
• Revise the documentation, secure our first miner partnership, acquire 10 seed users, and proceed with product beta testing and launch;
• Adjust the code to integrate with coretime, utilizing other Polkadot assets for collateralization and purchasing container services.

Yes, it’s true that we haven’t updated the readme of our project for quite some time now. The reason being that we’ve been keeping all the latest information on our official website at https://wetee.app/. We greatly appreciate the reminder and will update the readme for our project this week. We look forward to welcoming more developers to join our DAO

[BurnWW]

@PieWol Thank you for the reminder. I have updated the existing documents into the readme, but this still feels insufficient. There is also a lack of a minner UI, so at present, it remains incomplete for both users and developers.
https://github.com/wetee-dao/worker/blob/main/README.md

[PieWol]

Thanks very much for clarifying @BurnWW.
Looking forward to the changes you will introduce eventually. I have updated my evaluation so that your delivery will be accepted. A colleague will review it shortly and you will be notified once things are processed and merged. 🎉

[github-actions]

🪙 Please fill out the invoice form in order to initiate the payment process. Thank you!

[github-actions]

We noticed that this is the last milestone of your project. Congratulations on completing your grant! 🎊

So, where to from here? First of all, you should join our Grants Community chat, if you haven't already, so we can stay in touch.
If you are looking for continuative support for your project, there are quite a few options. The main goal of the W3F grants program is to support research as well as early-stage technical projects. If your project still falls under one of those categories, you might want to apply for a follow-up grant. However, depending on your goals and project status, there are other support programs in our ecosystem that might be better suited as the next step. For example, projects with a Business Case/Token should look into the Substrate Builders Program or VC Funding and Common Good projects have a good chance of receiving Treasury Funding. If you are looking for guidance, the team at https://substrate.io/ecosystem/square-one/ has compiled a list of ecosystem support sources and are happy to help you navigate it.

For a more comprehensive list, see our Alternative Funding page. Let us know if you have any questions regarding the above. We are more than happy to point you to additional resources and help you determine the best course of action.
Lastly, we hope your W3F grant was a success and we want to thank you for being part of the journey!

[RouvenP]

hi @BurnWW we sent the payment last Friday