Skip to content
/ envd Public
forked from tensorchord/envd

πŸ•οΈ Development environment for machine learning

License

Notifications You must be signed in to change notification settings

Gui-Yue/envd

Β 
Β 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 

Repository files navigation

envd

Development environment for data science and AI/ML

discord invitation link trackgit-views all-contributors envd package donwloads continuous integration Coverage Status

⚠️ envd is still under heavy development, and subject to change. it is not feature-complete or production-ready. Please contact us in discord if there is any problem.

envd (ΙͺnˈvdΙͺ) is a machine learning development environment for data science and AI/ML engineering teams.

🐍 No Docker, only Python - Focus on writing Python code, we will take care of Docker and development environment setup.

πŸ–¨οΈ Built-in Jupyter/VSCode - First-class support for Jupyter and VSCode remote extension.

⏱️ Save time - Better cache management to save your time, keep the focus on the model, instead of dependencies.

☁️ Local & cloud - envd integrates seamlessly with Docker so that you can easily share, version, and publish envd environments with Docker Hub or any other OCI image registries.

πŸ” Repeatable builds & reproducible results - You can reproduce the same dev environment on your laptop, public cloud VMs, or Docker containers, without any change in setup.

Why use envd?

It is still too difficult to configure development environments and reproduce results in AI/ML applications.

envd is a machine learning development environment for data science and AI/ML engineering teams. Environments built with envd provide the following features out-of-the-box:

🐍 Life is short, use Python1

Development environments are full of Dockerfiles, bash scripts, Kubernetes YAML manifests, and many other clunky files that are always breaking. envd builds are isolated and clean. You can write simple instructions in Python, instead of Bash / Makefile / Dockerfile / ...

envd

⏱️ Save you plenty of time

envd adopts a multi-level cache mechanism to accelerate the building process. For example, the PyPI cache is shared across builds and thus the package will be cached if it has been downloaded before. It saves plenty of time, especially when you update the environment by trial and error.

envd

Docker2

$ envd build
=> pip install tensorflow       5s
+ => Using cached tensorflow-...-.whl (511.7 MB)
$ docker build
=> pip install tensorflow      278s
- => Downloading tensorflow-...-.whl (511.7 MB)

☁️ Local & cloud native

envd integrates seamlessly with Docker, you can share, version, and publish envd environments with Docker Hub or any other OCI image registries. The envd environments can be run on Docker or Kubernetes.

πŸ” Repeatable builds & reproducible results

You can reproduce the same dev environment, on your laptop, public cloud VMs, or Docker containers, without any change in setup. You can also collaborate with your colleagues without "let me configure the environment in your machine".

πŸ–¨οΈ Seamless experience of Jupyter/VSCode

envd provides first-class support for Jupyter and VSCode remote extension. You benefit without sacrificing any developer experience.

Who should use envd?

We’re focused on helping data scientists and teams that develop AI/ML models. And they may suffer from:

  • building the development environments with Python, CUDA, Docker, SSH, and so on. Do you have a complicated Dockerfile or build script that sets up all your dev environments, but is always breaking?
  • Updating the environment. Do you always need to ask infrastructure engineers how to add a new python package in the Dockerfile?
  • Managing environments and machines. Do you always forget which machines are used for the specific project, because you handle multiple projects concurrently?

Talk with us

πŸ’¬ Interested in talking with us about your experience building or managing AI/ML applications?

Set up a time to chat!


Before envd After envd

Documentation πŸ“

See envd documentation.

Getting Started

Requirements

  • Docker (20.10.0 or above)

Install and bootstrap envd

envd can be installed with pip. After the installation, please run envd bootstrap to bootstrap.

pip install --pre --upgrade envd
envd bootstrap

You can add --dockerhub-mirror or -m flag when running envd bootstrap, to configure the mirror for docker.io registry:

envd bootstrap --dockerhub-mirror https://docker.mirrors.sjtug.sjtu.edu.cn

Create an envd environment

Please clone the envd-quick-start:

git clone https://github.com/tensorchord/envd-quick-start.git

The build manifest build.envd looks like:

def build():
    base(os="ubuntu20.04", language="python3")
    install.python_packages(name = [
        "numpy",
    ])
    shell("zsh")

Then please run the command below to set up a new environment:

cd envd-quick-start && envd up
$ cd envd-quick-start && envd up
[+] ⌚ parse build.envd and download/cache dependencies 2.8s βœ… (finished)     
 => download oh-my-zsh                                                    2.8s 
[+] πŸ‹ build envd environment 18.3s (25/25) βœ… (finished)                      
 => create apt source dir                                                 0.0s 
 => local://cache-dir                                                     0.1s 
 => => transferring cache-dir: 5.12MB                                     0.1s 
...
 => pip install numpy                                                    13.0s 
 => copy /oh-my-zsh /home/envd/.oh-my-zsh                                 0.1s 
 => mkfile /home/envd/install.sh                                          0.0s 
 => install oh-my-zsh                                                     0.1s 
 => mkfile /home/envd/.zshrc                                              0.0s 
 => install shell                                                         0.0s
 => install PyPI packages                                                 0.0s
 => merging all components into one                                       0.3s
 => => merging                                                            0.3s
 => mkfile /home/envd/.gitconfig                                          0.0s 
 => exporting to oci image format                                         2.4s 
 => => exporting layers                                                   2.0s 
 => => exporting manifest sha256:7dbe9494d2a7a39af16d514b997a5a8f08b637f  0.0s
 => => exporting config sha256:1da06b907d53cf8a7312c138c3221e590dedc2717  0.0s
 => => sending tarball                                                    0.4s
(envd) ➜  demo git:(master) βœ— # You are in the container-based environment!

Set up Jupyter notebook

Please edit the build.envd to enable jupyter notebook:

def build():
    base(os="ubuntu20.04", language="python3")
    install.python_packages(name = [
        "numpy",
    ])
    shell("zsh")
    config.jupyter(password="")

You can get the endpoint of the running Jupyter notebook via envd get envs.

$ envd up --detach
$ envd get env
NAME                    JUPYTER                 SSH TARGET              CONTEXT                                 IMAGE                   GPU     CUDA    CUDNN   STATUS          CONTAINER ID 
envd-quick-start        http://localhost:42779   envd-quick-start.envd   /home/gaocegege/code/envd-quick-start   envd-quick-start:dev    false   <none>  <none>  Up 54 seconds   bd3f6a729e94

Roadmap πŸ—‚οΈ

Please checkout ROADMAP.

Contribute πŸš€

We welcome all kinds of contributions from the open-source community, individuals, and partners.

Open in Gitpod

Contributors ✨

Thanks goes to these wonderful people (emoji key):


Friends A.

πŸ“–

Aaron Sun

πŸ““ πŸ’»

Aka.Fido

πŸ“¦ πŸ“– πŸ’»

Ce Gao

πŸ’» πŸ“– 🎨 πŸ“†

Guangyang Li

πŸ’»

Gui-Yue

πŸ’»

Haiker Sun

πŸ’»

Ikko Ashimine

πŸ’»

Jian Zeng

🎨 πŸ€” πŸ”¬

Jinjing Zhou

πŸ› πŸ’» 🎨 πŸ“–

Jun

πŸ“¦ πŸ’»

Keming

πŸ’» πŸ“– πŸ€” πŸš‡

Kevin Su

πŸ’»

Ling Jin

πŸ› πŸš‡

Manjusaka

πŸ’»

Nino

🎨

Pengyu Wang

πŸ“–

Siyuan Wang

πŸ’» πŸš‡ 🚧

Xu Jin

πŸ’»

Xuanwo

πŸ’¬ 🎨 πŸ€” πŸ‘€

Yuan Tang

πŸ’» 🎨 πŸ“– πŸ€”

Yuchen Cheng

πŸ› πŸš‡ 🚧 πŸ”§

Yunchuan Zheng

πŸ’»

Zheming Li

πŸ’»

Zhenzhen Zhao

πŸš‡ πŸ““ πŸ’»

Zhizhen He

πŸ’» πŸ“–

jimoosciuc

πŸ““

kenwoodjw

πŸ’»

nullday

πŸ€” πŸ’»

wyq

πŸ› 🎨 πŸ’»

xing0821

πŸ€” πŸ““ πŸ’»

zhyon404

πŸ’»

This project follows the all-contributors specification. Contributions of any kind welcome!

License πŸ“‹

Apache 2.0

trackgit-views

Footnotes

  1. The build language is starlark, which is a dialect of Python. ↩

  2. Docker without buildkit ↩

About

πŸ•οΈ Development environment for machine learning

Resources

License

Code of conduct

Stars

Watchers

Forks

Packages

No packages published

Languages

  • Go 89.5%
  • Shell 4.6%
  • Dockerfile 2.5%
  • Python 2.1%
  • Makefile 1.3%