-
Notifications
You must be signed in to change notification settings - Fork 97
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
1. specify the prefetch value with an URL 2. modified some Syntax errors 3. Modify the reading mode of configuration file 4. modify ci.yml and k8s-e2e.yml
- Loading branch information
Showing
10 changed files
with
295 additions
and
38 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,160 @@ | ||
| # User self-defined nydus image files prefetch | ||
| To improve the flexibility of nydus image files prefetch, for the k8s scenario, we can specify a prefetch files list when create a nydus daemon. The prefetch files list is user self-defined. Nydus-snapshotter has implemented a containerd NRI plugin to transmit the path of prefetch files list to nydus-snapshotter. The prefetch plugin requires NRI 2.0, which is available in containerd (>=v1.7.0). The prefetch plugin subscribes pod creation event, obtains the URL address containing the content of the files need to be prefetched, and forwards it to `nydus-snapshotter`. The `nydus-snapshotter` reads the data through the URL and stores it locally. Then when `nydusd` starts, it will pull the files defined in the prefetch files list through lazy loading. This allows the pull of the prefetch files to be done during container creation rather than image convert, improving the flexibility of file prefetching. | ||
|
|
||
| ## Requirements | ||
|
|
||
| - [NRI 2.0](https://github.com/containerd/nri): Which has been integrated into containerd since [v1.7.0](https://github.com/containerd/containerd/tree/v1.7.0-beta.1). | ||
|
|
||
| ## Workflow | ||
|
|
||
| 1. Add information such as image reference and URL address containing prefetch files to annotations in pod configuration file. | ||
| 2. Run the prefetch plugin to monitor RunPodSandbox events. | ||
| 3. The prefetch plugin fetches image reference and URL and forwards them to nydus-snapshotter. | ||
| 4. Nydus-snapshotter specifies the prefetch list when starting nydus daemon. | ||
| 5. Nydusd completes the mounting of the nydus image. | ||
|
|
||
|
|
||
|
|
||
|
|
||
|
|
||
| ## Modify configuration file | ||
|
|
||
| Modify containerd's toml configuration file to enable NRI. | ||
| ```console | ||
| sudo tee -a /etc/containerd/config.toml <<- EOF | ||
| [plugins."io.containerd.nri.v1.nri"] | ||
| config_file = "/etc/nri/nri.conf" | ||
| disable = false | ||
| plugin_path = "/opt/nri/plugins" | ||
| socket_path = "/var/run/nri.sock" | ||
| EOF | ||
| ``` | ||
| Containerd will load all NRI plugins in the `plugin_path` directory on startup. If you want to start an NRI plugin manually, please add the following configuration to allow other NRI plugins to connect via `socket_path`. | ||
|
|
||
| ```console | ||
| sudo tee /etc/nri/nri.conf <<- EOF | ||
| disableConnections: false | ||
| EOF | ||
| ``` | ||
|
|
||
| Restart the containerd service. | ||
|
|
||
| ```console | ||
| sudo systemctl restart containerd | ||
| ``` | ||
| If you want to start the plugin using `pre-connection` mode, that is, `disableConnections: true`. You need to write a configuration file and place the plugin's binary file and configuration file in the correct directories. Here is an example of configuration file: | ||
| ```console | ||
| sudo tee prefetchConfig.conf <<- EOF | ||
| # unix domain socket address | ||
| socket_address = "/run/containerd-nydus/system.sock" | ||
| EOF | ||
| ``` | ||
| Then move the files to the correct directories and set permissions. | ||
| ```console | ||
| go build | ||
| sudo install -D -m 755 ./prefetchfiles-nri-plugin /opt/nri/plugins/03-prefetchfiles-nri-plugin | ||
| sudo install -D -m 755 ./prefetchfiles-nri-plugin.conf /etc/nri/conf.d/03-prefetchfiles-nri-plugin.conf | ||
| ``` | ||
| When manually starting the prefetch NRI plugin, the socket address can be modified through the command line parameter `socket-addr`. | ||
|
|
||
|
|
||
|
|
||
| After start the prefetch plugin, it will monitor pod creation events. Note that NRI plugin can only be called from containerd/CRI. So creation a pod using crictl as below. | ||
| ```console | ||
| sudo tee pod.yaml <<- EOF | ||
| kind: pod | ||
| metadata: | ||
| name: wordpress-sandbox | ||
| namespace: default | ||
| attempt: 1 | ||
| uid: hdishd83djaidwnduwk28bcsb | ||
| log_directory: /tmp | ||
| annotations: | ||
| containerd.io/nydus-prefetch: | | ||
| [ | ||
| {"image": "dockerhub.kubekey.local/dfns/wordpress:nydus_latest", "prefetch": "http://example.com/api/v1/resource/wordpress"} | ||
| ] | ||
|
|
||
| linux: {} | ||
|
|
||
| EOF | ||
|
|
||
| crictl runp pod.yaml | ||
| ``` | ||
| The files that require prefetching are written in a URL, and `nydus-snapshoter` will read the prefetch list based on the URL address and transfer it to a local text file. The specific content of the prefetch files can be customized by the user. | ||
| `http://example.com/api/v1/resource/wordpress` is just an example of URL address, which needs to be replaced with a real URL during actual operation. The following is an example of some prefetch files in URL: | ||
| ```console | ||
| /usr/bin/env | ||
| /lib/x86_64-linux-gnu/ld-2.31.so | ||
| /etc/ld.so.cache | ||
| /lib/x86_64-linux-gnu/libc-2.31.so | ||
| /bin/bash | ||
| /lib/x86_64-linux-gnu/libtinfo.so.6.2 | ||
| /lib/x86_64-linux-gnu/libdl-2.31.so | ||
| /etc/nsswitch.conf | ||
| ``` | ||
|
|
||
| The prefetching NRI plugin also supports the case where the pod configuration file contains multiple containers. In this case, the definition of pod-mutlicontainers.yaml is as follows: | ||
|
|
||
|
|
||
| ```console | ||
| sudo tee pod-mutlicontainers.yaml <<- EOF | ||
| apiVersion: v1 | ||
| kind: Pod | ||
| metadata: | ||
| name: multi-containers-pod | ||
| namespace: default | ||
| attempt: 1 | ||
| uid: hdishd83djaidwnduwk28bcsb | ||
| log_directory: /tmp | ||
| annotations: | ||
| containerd.io/nydus-prefetch: | | ||
| [ | ||
| {"image": "dockerhub.kubekey.local/dfns/busybox:nydus_latest", "prefetch": "http://example.com/api/v1/resource/busybox"}, | ||
| {"image": "dockerhub.kubekey.local/dfns/ubuntu:nydus_latest", "prefetch": "http://example.com/api/v1/resource/ubuntu"}, | ||
| {"image": "dockerhub.kubekey.local/dfns/wordpress:nydus_latest", "prefetch": "http://example.com/api/v1/resource/wordpress"} | ||
| ] | ||
|
|
||
| linux: {} | ||
|
|
||
| spec: | ||
| containers: | ||
| - name: container-1 | ||
| image: dockerhub.kubekey.local/dfns/busybox:nydus_latest | ||
| command: ["ls"] | ||
| - name: container-2 | ||
| image: dockerhub.kubekey.local/dfns/ubuntu:nydus_latest | ||
| command: ["ls"] | ||
| - name: container-3 | ||
| image: dockerhub.kubekey.local/dfns/wordpress:nydus_latest | ||
| command: ["ls"] | ||
| EOF | ||
| ``` | ||
|
|
||
|
|
||
|
|
||
| Note that the naming of keys in annotations is fixed, and the values in annotations are user self-defined. | ||
| After creating a pod, `nydus-snapshotter` will store the image references and paths of the prefetch list. | ||
|
|
||
| ## Nydus-snapshotter starts nydusd | ||
| Nydusd daemon will start when the container is created. We start a container like below. | ||
| ```console | ||
| sudo tee wordpress.yaml <<- EOF | ||
| metadata: | ||
| name: wordpress | ||
| image: | ||
| image: dockerhub.kubekey.local/dfns/wordpress:nydus_latest | ||
| log_path: wordpress.0.log | ||
| linux: {} | ||
| EOF | ||
|
|
||
| crictl pull dockerhub.kubekey.local/dfns/wordpress:nydus_latest | ||
| crictl create <pod_id> wordpress.yaml pod.yaml | ||
| ``` | ||
|
|
||
| Then nydus-snapshotter will start a daemon by command as follows. | ||
| ```editorconfig | ||
| /usr/local/bin/nydusd fuse --thread-num 4 --config /var/lib/containerd-nydus/config/cjmnum3c3al8js5p3740/config.json --bootstrap /var/lib/containerd-nydus/snapshots/22/fs/image/image.boot --mountpoint /var/lib/containerd-nydus/snapshots/22/mnt --apisock /var/lib/containerd-nydus/socket/cjmnum3c3al8js5p3740/api.sock --log-level info --log-rotation-size 100 --prefetch-files /etc/nydus/PrefetchFiles/dockerhub.kubekey.local@dfns@wordpress:nydus_latest | ||
| ``` | ||
|
|
||
| According to the parameter `--prefetch-files`, we have implemented file prefetching through lazy loading. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,3 @@ | ||
| # unix domain socket address | ||
| socket_address = "/run/containerd-nydus/system.sock" | ||
|
|
This file was deleted.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.