diff --git a/docker2fl/README.md b/docker2fl/README.md new file mode 100644 index 0000000..ec3cffc --- /dev/null +++ b/docker2fl/README.md @@ -0,0 +1,137 @@ +# docker2fl + +`docker2fl` is a tool to extract docker images and convert them to flist using [rfs](../rfs) tool. + +## Build + +To build docker2fl make sure you have rust installed then run the following commands: + +```bash +# this is needed to be run once to make sure the musl target is installed +rustup target add x86_64-unknown-linux-musl + +# build the binary +cargo build --release --target=x86_64-unknown-linux-musl +``` + +the binary will be available under `./target/x86_64-unknown-linux-musl/release/docker2fl` you can copy that binary then to `/usr/bin/` +to be able to use from anywhere on your system. + +```bash +sudo mv ./target/x86_64-unknown-linux-musl/release/docker2fl /usr/bin/ +``` + +## Stores + +A store in where the actual data lives. A store can be as simple as a `directory` on your local machine in that case the files on the `fl` are only 'accessible' on your local machine. A store can also be a `zdb` running remotely or a cluster of `zdb`. Right now only `dir`, `zdb` and `s3` stores are supported but this will change in the future to support even more stores. + +## Usage + +### Creating an `fl` + +```bash +docker2fl -i redis -s +``` + +This tells docker2fl to create an `fl` named `redis-latest.fl` using the store defined by the url `` and upload all the files under the temp docker directory that include exported docker image recursively. + +The simplest form of `` is a `url`. the store `url` defines the store to use. Any `url` has a schema that defines the store type. Right now we have support only for: + +- `dir`: dir is a very simple store that is mostly used for testing. A dir store will store the fs blobs in another location defined by the url path. An example of a valid dir url is `dir:///tmp/store` +- `zdb`: [zdb](https://github.com/threefoldtech/0-db) is a append-only key value store and provides a redis like API. An example zdb url can be something like `zdb://[:port][/namespace]` +- `s3`: aws-s3 is used for storing and retrieving large amounts of data (blobs) in buckets (directories). An example `s3://:@:/` + + `region` is an optional param for s3 stores, if you want to provide one you can add it as a query to the url `?region=` + +`` can also be of the form `-=` where `start` and `end` are a hex bytes for partitioning of blob keys. rfs will then store a set of blobs on the defined store if they blob key falls in the `[start:end]` range (inclusive). + +If the `start-end` range is not provided a `00-FF` range is assume basically a catch all range for the blob keys. In other words, all blobs will be written to that store. + +This is only useful because `docker2fl` can accept multiple stores on the command line with different and/or overlapping ranges. + +For example `-s 00-80=dir:///tmp/store0 -s 81-ff=dir://tmp/store1` means all keys that has prefix byte in range `[00-80]` will be written to /tmp/store0 all other keys `00-ff` will be written to store1. + +The same range can appear multiple times, which means the blob will be replicated to all the stores that matches its key prefix. + +To quickly test this operation + +```bash +docker2fl -i redis -s "dir:///tmp/store0" +``` + +this command will use redis image and effectively create the `redis.fl` and store (and shard) the blobs across the location /tmp/store0. + +```bash +#docker2fl --help + +Usage: docker2fl [OPTIONS] --image-name +Options: + --debug... + enable debugging logs + -i, --image-name + name of the docker image to be converted to flist + -s, --store + store url for rfs in the format [xx-xx=]. the range xx-xx is optional and used for sharding. the URL is per store type, please check docs for more information + -h, --help + Print help + -V, --version + Print version +``` + +## Generate an flist using ZDB + +### Deploy a vm + +1. Deploy a vm with a public IP +2. add docker (don't forget to add a disk for it with mountpoint = "/var/lib/docker") +3. add caddy + +### Install zdb and run an instance of it + +1. Execute `git clone -b development-v2 https://github.com/threefoldtech/0-db /zdb` then `cd /zdb` +2. Build + + ```bash + cd libzdb + make + cd .. + + cd zdbd + make STATIC=1 + cd .. + + make + ``` + +3. Install `make install` +4. run `zdb --listen 0.0.0.0` +5. The result info you should know + + ```console + zdbEndpoint = ":" + zdbNameSpace = "default" + zdbPassword = "default" + ``` + +### Install docker2fl + +1. Execute `git clone -b development-v2 https://github.com/threefoldtech/rfs` then `cd /rfs` +2. Execute + + ```bash + rustup target add x86_64-unknown-linux-musl` + cargo build --features build-binary --release --target=x86_64-unknown-linux-musl + mv ./target/x86_64-unknown-linux-musl/release/docker2fl /usr/bin/ + ``` + +### Convert docker image to an fl + +1. Try an image for example `threefolddev/ubuntu:22.04` image +2. Executing `docker2fl -i threefolddev/ubuntu:22.04 -s "zdb://:/default" -d` +3. You will end up having `threefolddev-ubuntu-22.04.fl` (flist) + +### Serve the flist using caddy + +1. In the directory includes the output flist, you can run `caddy file-server --listen 0.0.0.0:2015 --browse` +2. The flist will be available as `http://:2015/threefolddev-ubuntu-22.04.fl` +3. Use the flist to deploy any virtual machine. diff --git a/rfs/src/main.rs b/rfs/src/main.rs index 45bdd32..647191a 100644 --- a/rfs/src/main.rs +++ b/rfs/src/main.rs @@ -11,8 +11,6 @@ use rfs::cache; use rfs::fungi; use rfs::store::{self, Router, Stores}; -use regex::Regex; - mod fs; /// mount flists #[derive(Parser, Debug)]