Devcontainer #29259

robbederks · 2023-08-06T10:06:17Z

This makes it super easy to set up a dev environment with vscode.
It prepares the docker, and automatically mounts in the work dir.

.devcontainer/devcontainer.json

Dockerfile.openpilot_dev

.devcontainer/devcontainer.json

robbederks · 2023-08-08T03:51:58Z

@fredyshox Mind writing a small README for this? Think it's already in a pretty usable state

adeebshihadeh · 2023-08-08T22:13:06Z

I imagine some contributors are going to be relying on this, so think we should add a small test before merging this.

adeebshihadeh · 2023-08-08T22:17:27Z

Dockerfile.openpilot_dev

@@ -0,0 +1,5 @@
+FROM ghcr.io/commaai/openpilot-base:latest


would put this file in .devcontainer/

tools/README.md

fredyshox · 2023-08-09T02:48:45Z

Btw I've checked how slow would it be to build aarch64 images using QEMU and docker buildx on x64 machine. It's way too slow, after an hour it only made it to py3.11 installation (and crashed while doing so).

That's very unfortunate, because this feature makes it super easy to have stable development experience on a mac.

fredyshox · 2023-08-09T03:16:02Z

I've just found out that Azure pipelines seem to support arm64. We're already using that in some places, maybe we could utilize that for building such images.

adeebshihadeh · 2023-08-09T03:22:36Z

I don't want to introduce another CI system for this. QEMU should be fine, or BuildJet (#29277) if we end up merging that.

robbederks · 2023-08-10T00:29:12Z

tools/README.md

+```
+xhost +localhost
+```
+Note that this is temporary and only affects current XQuartz session. To make it pernament, add the above line to shell rc file.


typo: permanent

fredyshox · 2023-08-10T18:35:32Z

Added simple CI test for this which builds an image and runs the container using @devcontainers/cli, which should act very similar to vscode extension.

adeebshihadeh · 2023-08-10T21:03:56Z

.devcontainer/Dockerfile

+# remove gitconfig if exists, since its gonna be replaced by host one
+RUN rm -f /root/.gitconfig


won't it be shadowed anyway when it's mounted in?

We are not mounting it ourselves, vscode/devcontainer does that for us, and it fails silently if it exists already.

adeebshihadeh · 2023-08-10T21:04:25Z

.devcontainer/Dockerfile

+# remove gitconfig if exists, since its gonna be replaced by host one
+RUN rm -f /root/.gitconfig
+RUN apt update && apt install -y vim net-tools usbutils htop
+RUN pip install ipython jupyter jupyterlab


we should consider adding this as a poetry group

adeebshihadeh · 2023-08-10T21:14:49Z

tools/README.md

+
+openpilot supports [Dev Containers](https://containers.dev/). Dev containers provide customizable and consistent development environment wrapped inside a container. This means you can develop in a designated environment matching our primary development target, regardless of you local setup.
+
+Dev containers are supported in [multiple editors and IDEs](https://containers.dev/supporting), including Visual Studio Code. To start using it in VS Code, make sure you have `docker` installed. Then install `Dev Containers` extension. More detailed installation instructions can be found [here](https://code.visualstudio.com/docs/devcontainers/containers#_installation).


this new section is quite long. we should just link to the real instructions for each thing, i.e. https://code.visualstudio.com/docs/devcontainers/create-dev-container for this one

adeebshihadeh · 2023-08-16T18:01:41Z

tools/README.md

@@ -36,6 +36,32 @@ Build openpilot with this command:
 scons -u -j$(nproc)
 ```

+### Dev Container
+
+openpilot supports [Dev Containers](https://containers.dev/). Dev containers provide customizable and consistent development environment wrapped inside a container. This means you can develop in a designated environment matching our primary development target, regardless of you local setup.


you -> your

adeebshihadeh · 2023-08-16T18:03:37Z

tools/README.md

+
+Dev containers are supported in [multiple editors and IDEs](https://containers.dev/supporting), including [Visual Studio Code](https://code.visualstudio.com/docs/devcontainers/containers).
+
+#### X11 forwarding on macOS


seems like we can link away this whole section

adeebshihadeh · 2023-08-16T18:05:12Z

.github/workflows/dev_tools_tests.yaml

@@ -0,0 +1,24 @@
+name: development tools


we've already got tools_tests.yaml

adeebshihadeh · 2023-08-16T18:05:52Z

.github/workflows/dev_tools_tests.yaml

+    - name: Stop containers
+      run: docker kill $(docker ps -q)


doesn't seem necessary

fredyshox · 2023-08-21T20:47:22Z

@adeebshihadeh is this mergeable now?

adeebshihadeh · 2023-08-21T20:50:17Z

.github/workflows/tools_tests.yaml

+    name: devcontainer
+    runs-on: ubuntu-latest
+    if: github.repository == 'commaai/openpilot'


what's this for?

not needed in this case

adeebshihadeh · 2023-08-21T20:51:34Z

.github/workflows/tools_tests.yaml

+      run: devcontainer build --workspace-folder .
+    - name: Run dev container
+      run: devcontainer up --workspace-folder .


can we also run something like "scons --dry-run" to get some coverage on the environment?

adeebshihadeh · 2023-08-21T20:52:19Z

Just got this while trying it out

fredyshox · 2023-08-21T21:10:13Z

Just got this while trying it out

Seems something is wrong with permission to submodules. Gonna see what i can do

fredyshox · 2023-08-21T21:41:31Z

Should be fine now, can you try again?

adeebshihadeh · 2023-08-22T00:33:50Z

Same thing (this is not an efficient loop)

adeebshihadeh · 2023-08-22T00:40:58Z

New issue

bash: /tmp/tools/openpilot_env.sh: No such file or directory
bash: /workspaces/openpilot/tools/openpilot_env.sh: No such file or directory

adeebshihadeh · 2023-08-22T00:44:42Z

Seems it's missing the scons cache

fredyshox · 2023-08-22T02:26:18Z

New issue

bash: /tmp/tools/openpilot_env.sh: No such file or directory
bash: /workspaces/openpilot/tools/openpilot_env.sh: No such file or directory

Removed references to openpilot_env since it got removed

fredyshox · 2023-08-22T03:37:40Z

Seems it's missing the scons cache

I don't think that causes big inconvenience, as it's relatively quick to build. Including it would significantly inflate image size, which is already quite big.

adeebshihadeh · 2023-08-22T03:54:33Z

It should be volume mounted, not copied in

adeebshihadeh · 2023-08-22T18:52:32Z

.devcontainer/devcontainer.json

+    "--volume=${localEnv:XAUTHORITY}:/root/.Xauthority",
+    "--volume=${localEnv:HOME}/.comma:/root/.comma",
+    "--volume=/tmp/comma_download_cache:/tmp/comma_download_cache",
+    "--volume=/tmp/devcontainer_scons_cache:/tmp/scons_cache",


why is there a special devcontainer cache?

I didn't want to mess with hosts cache, which may be incompatible (like on macos)

hmm, but this is also true for the build targets

robbederks commented Aug 7, 2023

View reviewed changes

.devcontainer/devcontainer.json Outdated Show resolved Hide resolved

fredyshox reviewed Aug 7, 2023

View reviewed changes

Dockerfile.openpilot_dev Outdated Show resolved Hide resolved

.devcontainer/devcontainer.json Outdated Show resolved Hide resolved

.devcontainer/devcontainer.json Show resolved Hide resolved

adeebshihadeh added the PC Issues related to running openpilot on PC label Aug 8, 2023

adeebshihadeh reviewed Aug 8, 2023

View reviewed changes

robbederks commented Aug 8, 2023

View reviewed changes

tools/README.md Outdated Show resolved Hide resolved

robbederks commented Aug 10, 2023

View reviewed changes

adeebshihadeh reviewed Aug 10, 2023

View reviewed changes

Robbe Derks and others added 15 commits August 14, 2023 13:04

mvp devcontainer. builds, runs and UI works

4de62a0

fix indentation

8c0ea4d

enable color and fix openpilot_env path

c6c8cf8

separate dockerfile for devcontainer

ba2177d

use prebuilt image

d1a3a75

Fix openpilot_env setup

40b559a

Add dev container section in readme

e734099

Fix typo

d5f48b9

Fix typo again

db19249

Move dev dockerfile to .devcontainer dir

ad9a87b

Update README

2d0456f

Add note about x11 forwarding on macos

33e7ebc

Update dev container json

bd611bf

Fix typo

f98ae2f

Override DISPLAY on macOS

a62a9ba

robbederks force-pushed the devcontainer branch from 5abd18b to 1d1c822 Compare August 14, 2023 18:05

robbederks marked this pull request as ready for review August 14, 2023 18:24

robbederks and others added 3 commits August 14, 2023 18:33

ripgrep

27a5f12

Link instead of vscode manual

949275a

Replace raw path with containerWorkspaceFolder

0a84a57

adeebshihadeh reviewed Aug 16, 2023

View reviewed changes

fredyshox added 5 commits August 16, 2023 21:59

Fix typo

5e5f53a

Remove dev tools workflow

17768e5

Merge branch 'master' into devcontainer

d3aa650

Link away X11 forwarding setup isntructions for mac

aa75b3f

Merge remote-tracking branch 'origin/master' into devcontainer

6c83c1b

adeebshihadeh approved these changes Aug 21, 2023

View reviewed changes

Remove fork consrtaint, add test run of scons

11b82d5

Add submodules as safe.directories in postStartCommand

bd8d87b

Remove openpilot_env references

4972a3b

Merge remote-tracking branch 'origin/master' into devcontainer

0126999

Add scons cache volume

15305c1

fredyshox merged commit d71a719 into master Aug 22, 2023
20 checks passed

fredyshox deleted the devcontainer branch August 22, 2023 18:47

adeebshihadeh reviewed Aug 22, 2023

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Devcontainer #29259

Devcontainer #29259

robbederks commented Aug 6, 2023

robbederks commented Aug 8, 2023

adeebshihadeh commented Aug 8, 2023

adeebshihadeh Aug 8, 2023

fredyshox commented Aug 9, 2023 •

edited

Loading

fredyshox commented Aug 9, 2023 •

edited

Loading

adeebshihadeh commented Aug 9, 2023

robbederks Aug 10, 2023

fredyshox commented Aug 10, 2023

adeebshihadeh Aug 10, 2023

fredyshox Aug 10, 2023 •

edited

Loading

adeebshihadeh Aug 10, 2023

adeebshihadeh Aug 10, 2023

adeebshihadeh Aug 16, 2023

adeebshihadeh Aug 16, 2023

adeebshihadeh Aug 16, 2023

adeebshihadeh Aug 16, 2023

fredyshox commented Aug 21, 2023

adeebshihadeh Aug 21, 2023

fredyshox Aug 21, 2023

adeebshihadeh Aug 21, 2023

adeebshihadeh commented Aug 21, 2023

fredyshox commented Aug 21, 2023

fredyshox commented Aug 21, 2023

adeebshihadeh commented Aug 22, 2023

adeebshihadeh commented Aug 22, 2023

adeebshihadeh commented Aug 22, 2023

fredyshox commented Aug 22, 2023

fredyshox commented Aug 22, 2023

adeebshihadeh commented Aug 22, 2023

adeebshihadeh Aug 22, 2023

fredyshox Aug 22, 2023

adeebshihadeh Aug 22, 2023

		# remove gitconfig if exists, since its gonna be replaced by host one
		RUN rm -f /root/.gitconfig


		openpilot supports [Dev Containers](https://containers.dev/). Dev containers provide customizable and consistent development environment wrapped inside a container. This means you can develop in a designated environment matching our primary development target, regardless of you local setup.

		Dev containers are supported in [multiple editors and IDEs](https://containers.dev/supporting), including Visual Studio Code. To start using it in VS Code, make sure you have `docker` installed. Then install `Dev Containers` extension. More detailed installation instructions can be found [here](https://code.visualstudio.com/docs/devcontainers/containers#_installation).


		Dev containers are supported in [multiple editors and IDEs](https://containers.dev/supporting), including [Visual Studio Code](https://code.visualstudio.com/docs/devcontainers/containers).

		#### X11 forwarding on macOS

Devcontainer #29259

Devcontainer #29259

Conversation

robbederks commented Aug 6, 2023

robbederks commented Aug 8, 2023

adeebshihadeh commented Aug 8, 2023

Choose a reason for hiding this comment

fredyshox commented Aug 9, 2023 • edited Loading

fredyshox commented Aug 9, 2023 • edited Loading

adeebshihadeh commented Aug 9, 2023

Choose a reason for hiding this comment

fredyshox commented Aug 10, 2023

Choose a reason for hiding this comment

fredyshox Aug 10, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fredyshox commented Aug 21, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

adeebshihadeh commented Aug 21, 2023

fredyshox commented Aug 21, 2023

fredyshox commented Aug 21, 2023

adeebshihadeh commented Aug 22, 2023

adeebshihadeh commented Aug 22, 2023

adeebshihadeh commented Aug 22, 2023

fredyshox commented Aug 22, 2023

fredyshox commented Aug 22, 2023

adeebshihadeh commented Aug 22, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fredyshox commented Aug 9, 2023 •

edited

Loading

fredyshox commented Aug 9, 2023 •

edited

Loading

fredyshox Aug 10, 2023 •

edited

Loading