Proxy for accessories #981

igor-alexandrov · 2024-09-26T19:05:29Z

This PR adds functionality similar to what Traefik provided Host middleware for accessories.
Some accessories like PGHero require to be deployed to kamal-proxy to provide web interface.

This PR also includes changes from #988.

djmb · 2024-09-30T12:21:53Z

I don't think we want to start supporting accessories with kamal - if there are two things that require proper Kamal deployments, we could have two separate sets of deploy files.

igor-alexandrov · 2024-09-30T12:28:06Z

I didn't think about having a separate deploy file. I will check and come back to you.

luizkowalski · 2024-09-30T17:49:48Z

just to add to the convo: I'm running a couple of accessories on web role, like Grafana and Umami. This would be a great addition! Right now it works smoothly with Traefik, I just add some labels and move on

igor-alexandrov · 2024-09-30T17:55:31Z

@luizkowalski I also run Grafana and PgHero. As a workaround for now, you can deploy manually after the accessory was started.

docker exec 3a34b6c08923 kamal-proxy deploy onetribe-pghero --target c67f2259dce6:8080 --host pghero.onetribe.team --health-check-path /

luizkowalski · 2024-09-30T17:58:12Z

which container is 3a34b6c08923 exactly? I assume c67f2259dce6 would be pghero

igor-alexandrov · 2024-09-30T18:03:27Z

Sorry it wasn't clear, so

3a34b6c08923 is a container with kamal-proxy
c67f2259dce6 is a PgHero container
The command should be run from the host machine

luizkowalski · 2024-09-30T20:42:28Z

I can't do it for some reason:

kamal@web:~$ docker exec bcb17df9ad18 kamal-proxy deploy sumiu-umami --target sumiu-umami:3000 --host msg.sumiu.link --health-check-path /
Error: target failed to become healthy

and the logs are just

2024-09-30T20:41:18.620448260Z {"time":"2024-09-30T20:41:18.619622857Z","level":"INFO","msg":"Healthcheck failed","status":307}
2024-09-30T20:41:18.620726703Z {"time":"2024-09-30T20:41:18.619706895Z","level":"INFO","msg":"Target health updated","target":"sumiu-umami:3000","success":false,"state":"adding"}
2024-09-30T20:41:19.621417953Z {"time":"2024-09-30T20:41:19.621048761Z","level":"INFO","msg":"Healthcheck failed","status":307}
2024-09-30T20:41:19.621483430Z {"time":"2024-09-30T20:41:19.62108796Z","level":"INFO","msg":"Target health updated","target":"sumiu-umami:3000","success":false,"state":"adding"}
2024-09-30T20:41:20.626210209Z {"time":"2024-09-30T20:41:20.625598445Z","level":"INFO","msg":"Healthcheck failed","status":307}
2024-09-30T20:41:20.626277910Z {"time":"2024-09-30T20:41:20.625647628Z","level":"INFO","msg":"Target health updated","target":"sumiu-umami:3000","success":false,"state":"adding"}
2024-09-30T20:41:21.617063999Z {"time":"2024-09-30T20:41:21.616750402Z","level":"INFO","msg":"Healthcheck failed","status":307}
2024-09-30T20:41:21.617101013Z {"time":"2024-09-30T20:41:21.616789156Z","level":"INFO","msg":"Target health updated","target":"sumiu-umami:3000","success":false,"state":"adding"}
2024-09-30T20:41:22.620533515Z {"time":"2024-09-30T20:41:22.620073727Z","level":"INFO","msg":"Healthcheck failed","status":307}

I can curl "sumiu-umami" from any other container tho

igor-alexandrov · 2024-10-01T05:49:48Z

Maybe you are using the wrong health check path?

luizkowalski · 2024-10-01T08:19:22Z

yup, that was it. I was pointing to / but it actually does not respond with 200.
looking forward for this PR to get merged

djmb · 2024-10-01T14:47:15Z

@igor-alexandrov, @luizkowalski - what were you using Traefik for with those accessories, compared to just publishing them directly?

You can run Traefik as an accessory as well. You can either run it in front of everything if you want it on ports 80/443, so you could just stand it up an different ports and configure the accessories to use it (see https://kamal-deploy.org/docs/upgrading/continuing-to-use-traefik/ for more details).

igor-alexandrov · 2024-10-01T14:53:58Z

@djmb Donal, having PgHero and probably other monitoring dashboards as accessories is easier compared to deploying them as a separate application (however I think this is possible too).

So the question is, why the database is an accessory, but a monitoring tool for this database should be a separate application?

luizkowalski · 2024-10-01T14:55:59Z

were you using Traefik for with those accessories, compared to just publishing them directly?

I deployed them to web role and put them on a subdomain:

umami:
  image: umamisoftware/umami:postgresql-v2.13.2
  hosts:
    - web
  env:
    clear:
      DATABASE_URL: <%= ENV["UMAMI_DATABASE_URL"] %>
      APP_SECRET: <%= ENV["UMAMI_SECRET_KEY_BASE"] %>
      DATABASE_TYPE: postgresql
  labels:
    traefik.http.routers.umami.rule: Host(`msg.domain.com`)
    traefik.http.routers.umami.service: sumiu-umami@docker
    traefik.http.routers.umami.tls: true

it is incredibly helpful

can run Traefik as an accessory as well

I know but not only feels like a hack, I'd need to add one more accessory and maintain it. would be much easier if this behavior was native

luizkowalski · 2024-10-01T17:39:54Z

ah one more thing i remembered: the "args" option is not available for accessories so all the configurations im currently doing for traefik would have to be clumped under "cmd" and i found it difficult to do it

luizkowalski · 2024-10-01T19:15:33Z

been giving a try with Traefik but I can't tell if this is something on my end or the changes on v2 are not compatible with the previous way.

First, I started with setting proxy: false and configuring the pre-deploy hook with the instructions from the upgrade guide.

for my web app, I have the following configuration:

servers:
  web:
    hosts:
      - web
    labels:
      traefik.enable: true
      traefik.http.routers.sumiu.rule: Host(`domain.com`)
      traefik.http.routers.sumiu.entrypoints: websecure
      traefik.http.routers.sumiu.tls.certresolver: letsencrypt

      traefik.http.routers.sumiu-metrics.rule: Host(`domain.com`) && PathPrefix(`/metrics`)
      traefik.http.routers.sumiu-metrics.service: sumiu-web@docker
      traefik.http.routers.sumiu-metrics.tls: true
      traefik.http.routers.sumiu-metrics.middlewares: metrics-auth
      traefik.http.middlewares.metrics-auth.basicauth.users: admin:$apr1$5QeDhE9N$wfLTmPYcFmoHiFbmv.RC1/

previously, the service name used to be stable across deployment, so sumiu-web would definitely work. now it doesn't because the service name is something like sumiu-web-58692e90c3e42f6a41490b555092ad2c804dcac1-uncommitted-eb44aaa2018ef063@docker so it is impossible to predict the service name for the service under sumiu-metrics here. This is not the case for other services:

I noticed also one more problem: downtime between deploys. (Can't tell why is this happening, maybe it has to do with the proxy being disabled and there are not healthchecks?)

This is the output of the deployment during a downtime

  INFO [77859a44] Running docker login -u [REDACTED] -p [REDACTED] on web
  INFO [77859a44] Finished in 1.010 seconds with exit status 0 (successful).
  INFO [fcd6be6a] Running docker inspect kamal-proxy --format '{{.Config.Image}}' | cut -d: -f2 on web
  INFO [fcd6be6a] Finished in 0.167 seconds with exit status 0 (successful).
  INFO [90e2e19e] Running docker container start kamal-proxy || docker run --name kamal-proxy --network kamal --detach --restart unless-stopped --volume kamal-proxy-config:/home/kamal-proxy/.config/kamal-proxy $(cat .kamal/proxy/options || echo "--publish 80:80 --publish 443:443") basecamp/kamal-proxy:v0.6.0 on web
  INFO [90e2e19e] Finished in 0.159 seconds with exit status 0 (successful).
Detect stale containers...
  INFO [d3a0bf00] Running docker ps --filter label=service=sumiu --filter label=role=web --format "{{.Names}}" | while read line; do echo ${line#sumiu-web-}; done on web
  INFO [d3a0bf00] Finished in 0.154 seconds with exit status 0 (successful).
  INFO [b8dba2ed] Running /usr/bin/env sh -c 'docker ps --latest --format '\''{{.Names}}'\'' --filter label=service=sumiu --filter label=role=web --filter status=running --filter status=restarting --filter ancestor=$(docker image ls --filter reference=luizkowalski/sumiu-app:latest --format '\''{{.ID}}'\'') ; docker ps --latest --format '\''{{.Names}}'\'' --filter label=service=sumiu --filter label=role=web --filter status=running --filter status=restarting' | head -1 | while read line; do echo ${line#sumiu-web-}; done on web
  INFO [b8dba2ed] Finished in 0.232 seconds with exit status 0 (successful).
  INFO [c89cfa32] Running docker ps --filter label=service=sumiu --filter label=role=job --format "{{.Names}}" | while read line; do echo ${line#sumiu-job-}; done on web
  INFO [c89cfa32] Finished in 0.166 seconds with exit status 0 (successful).
  INFO [ff0a00e4] Running /usr/bin/env sh -c 'docker ps --latest --format '\''{{.Names}}'\'' --filter label=service=sumiu --filter label=role=job --filter status=running --filter status=restarting --filter ancestor=$(docker image ls --filter reference=luizkowalski/sumiu-app:latest --format '\''{{.ID}}'\'') ; docker ps --latest --format '\''{{.Names}}'\'' --filter label=service=sumiu --filter label=role=job --filter status=running --filter status=restarting' | head -1 | while read line; do echo ${line#sumiu-job-}; done on web
  INFO [ff0a00e4] Finished in 0.260 seconds with exit status 0 (successful).
Start container with version 47180c1aab1d0dd56f986f90126523bca18468a3 (or reboot if already running)...
  INFO [0c456bf6] Running /usr/bin/env mkdir -p .kamal/apps/sumiu/assets/extracted/web-47180c1aab1d0dd56f986f90126523bca18468a3 && docker stop -t 1 sumiu-web-assets 2> /dev/null || true && docker run --name sumiu-web-assets --detach --rm --entrypoint sleep luizkowalski/sumiu-app:47180c1aab1d0dd56f986f90126523bca18468a3 1000000 && docker cp -L sumiu-web-assets:/rails/public/assets/. .kamal/apps/sumiu/assets/extracted/web-47180c1aab1d0dd56f986f90126523bca18468a3 && docker stop -t 1 sumiu-web-assets on web
  INFO [0c456bf6] Finished in 12.956 seconds with exit status 0 (successful).
  INFO [8dbf5732] Running /usr/bin/env sh -c 'docker ps --latest --format '\''{{.Names}}'\'' --filter label=service=sumiu --filter label=role=web --filter status=running --filter status=restarting --filter ancestor=$(docker image ls --filter reference=luizkowalski/sumiu-app:latest --format '\''{{.ID}}'\'') ; docker ps --latest --format '\''{{.Names}}'\'' --filter label=service=sumiu --filter label=role=web --filter status=running --filter status=restarting' | head -1 | while read line; do echo ${line#sumiu-web-}; done on web
  INFO [8dbf5732] Finished in 0.185 seconds with exit status 0 (successful).
  INFO [569f2192] Running /usr/bin/env mkdir -p .kamal/apps/sumiu/assets/volumes/web-47180c1aab1d0dd56f986f90126523bca18468a3 ; cp -rnT .kamal/apps/sumiu/assets/extracted/web-47180c1aab1d0dd56f986f90126523bca18468a3 .kamal/apps/sumiu/assets/volumes/web-47180c1aab1d0dd56f986f90126523bca18468a3 ; cp -rnT .kamal/apps/sumiu/assets/extracted/web-47180c1aab1d0dd56f986f90126523bca18468a3 .kamal/apps/sumiu/assets/volumes/web-58692e90c3e42f6a41490b555092ad2c804dcac1_uncommitted_eb44aaa2018ef063 || true ; cp -rnT .kamal/apps/sumiu/assets/extracted/web-58692e90c3e42f6a41490b555092ad2c804dcac1_uncommitted_eb44aaa2018ef063 .kamal/apps/sumiu/assets/volumes/web-47180c1aab1d0dd56f986f90126523bca18468a3 || true on web
  INFO [569f2192] Finished in 0.143 seconds with exit status 0 (successful).
  INFO [7e05dc95] Running docker container ls --all --filter name=^sumiu-web-47180c1aab1d0dd56f986f90126523bca18468a3$ --quiet on web
  INFO [7e05dc95] Finished in 0.157 seconds with exit status 0 (successful).
  INFO [ffa6a90f] Running /usr/bin/env sh -c 'docker ps --latest --format '\''{{.Names}}'\'' --filter label=service=sumiu --filter label=role=web --filter status=running --filter status=restarting --filter ancestor=$(docker image ls --filter reference=luizkowalski/sumiu-app:latest --format '\''{{.ID}}'\'') ; docker ps --latest --format '\''{{.Names}}'\'' --filter label=service=sumiu --filter label=role=web --filter status=running --filter status=restarting' | head -1 | while read line; do echo ${line#sumiu-web-}; done on web
  INFO [ffa6a90f] Finished in 0.232 seconds with exit status 0 (successful).
  INFO [dc79208e] Running /usr/bin/env mkdir -p .kamal/apps/sumiu/env/roles on web
  INFO [dc79208e] Finished in 0.129 seconds with exit status 0 (successful).
  INFO Uploading .kamal/apps/sumiu/env/roles/web.env 100.0%
  INFO [2278229e] Running docker run --detach --restart unless-stopped --name sumiu-web-47180c1aab1d0dd56f986f90126523bca18468a3 --network kamal --hostname web-9145ff71092c -e KAMAL_CONTAINER_NAME="sumiu-web-47180c1aab1d0dd56f986f90126523bca18468a3" -e KAMAL_VERSION="47180c1aab1d0dd56f986f90126523bca18468a3" --env RAILS_MAX_THREADS="3" --env WEB_CONCURRENCY="2" --env RAILS_ENV="production" --env-file .kamal/apps/sumiu/env/roles/web.env --log-opt max-size="10m" --volume /data/thruster:/rails/storage/thruster --volume $(pwd)/.kamal/apps/sumiu/assets/volumes/web-47180c1aab1d0dd56f986f90126523bca18468a3:/rails/public/assets --label service="sumiu" --label role="web" --label destination --label traefik.enable="true" --label traefik.http.routers.sumiu.rule="Host(\`sumiu.link\`)" --label traefik.http.routers.sumiu.entrypoints="websecure" --label traefik.http.routers.sumiu.tls.certresolver="letsencrypt" --label traefik.http.routers.sumiu-metrics.rule="Host(\`sumiu.link\`) && PathPrefix(\`/metrics\`)" --label traefik.http.routers.sumiu-metrics.service="sumiu-web@docker" --label traefik.http.routers.sumiu-metrics.tls="true" --label traefik.http.routers.sumiu-metrics.middlewares="metrics-auth" --label traefik.http.middlewares.metrics-auth.basicauth.users="admin:\$apr1\$5QeDhE9N\$wfLTmPYcFmoHiFbmv.RC1/" luizkowalski/sumiu-app:47180c1aab1d0dd56f986f90126523bca18468a3 on web
  INFO [2278229e] Finished in 0.562 seconds with exit status 0 (successful).
  INFO [9c474a4b] Running docker container ls --all --filter name=^sumiu-web-47180c1aab1d0dd56f986f90126523bca18468a3$ --quiet on web
  INFO [9c474a4b] Finished in 0.176 seconds with exit status 0 (successful).
  INFO [2be89293] Running docker exec kamal-proxy kamal-proxy deploy sumiu-web --target "d1b769050226:80" --deploy-timeout "30s" --drain-timeout "30s" --buffer-requests --buffer-responses --log-request-header "Cache-Control" --log-request-header "Last-Modified" --log-request-header "User-Agent" on web
  INFO [2be89293] Finished in 11.345 seconds with exit status 0 (successful).
  INFO First web container is healthy on web, booting any other roles
  INFO [2a0ac0ca] Running docker container ls --all --filter name=^sumiu-web-58692e90c3e42f6a41490b555092ad2c804dcac1_uncommitted_eb44aaa2018ef063$ --quiet | xargs docker stop on web
  INFO [2a0ac0ca] Finished in 12.462 seconds with exit status 0 (successful).
  INFO [98639c67] Running /usr/bin/env find .kamal/apps/sumiu/assets/extracted -maxdepth 1 -name 'web-*' ! -name web-47180c1aab1d0dd56f986f90126523bca18468a3 -exec rm -rf "{}" + ; find .kamal/apps/sumiu/assets/volumes -maxdepth 1 -name 'web-*' ! -name web-47180c1aab1d0dd56f986f90126523bca18468a3 -exec rm -rf "{}" + on web
  INFO [98639c67] Finished in 0.092 seconds with exit status 0 (successful).
  INFO [523f1780] Running docker container ls --all --filter name=^sumiu-job-47180c1aab1d0dd56f986f90126523bca18468a3$ --quiet on web
  INFO [523f1780] Finished in 0.171 seconds with exit status 0 (successful).
  INFO [606f64c3] Running /usr/bin/env sh -c 'docker ps --latest --format '\''{{.Names}}'\'' --filter label=service=sumiu --filter label=role=job --filter status=running --filter status=restarting --filter ancestor=$(docker image ls --filter reference=luizkowalski/sumiu-app:latest --format '\''{{.ID}}'\'') ; docker ps --latest --format '\''{{.Names}}'\'' --filter label=service=sumiu --filter label=role=job --filter status=running --filter status=restarting' | head -1 | while read line; do echo ${line#sumiu-job-}; done on web
  INFO [606f64c3] Finished in 0.265 seconds with exit status 0 (successful).
  INFO Waiting for the first healthy web container before booting job on web...
  INFO First web container is healthy, booting job on web...
  INFO [720688f2] Running /usr/bin/env mkdir -p .kamal/apps/sumiu/env/roles on web
  INFO [720688f2] Finished in 0.124 seconds with exit status 0 (successful).
  INFO Uploading .kamal/apps/sumiu/env/roles/job.env 100.0%
  INFO [d8706509] Running docker run --detach --restart unless-stopped --name sumiu-job-47180c1aab1d0dd56f986f90126523bca18468a3 --network kamal --hostname web-661c3a4e6625 -e KAMAL_CONTAINER_NAME="sumiu-job-47180c1aab1d0dd56f986f90126523bca18468a3" -e KAMAL_VERSION="47180c1aab1d0dd56f986f90126523bca18468a3" --env RAILS_MAX_THREADS="3" --env WEB_CONCURRENCY="2" --env RAILS_ENV="production" --env CRON_ENABLED="1" --env GOOD_JOB_MAX_THREADS="10" --env DB_POOL="13" --env-file .kamal/apps/sumiu/env/roles/job.env --log-opt max-size="10m" --volume /data/thruster:/rails/storage/thruster --label service="sumiu" --label role="job" --label destination --label traefik.enable luizkowalski/sumiu-app:47180c1aab1d0dd56f986f90126523bca18468a3 bin/good_job --probe-port=7001 on web
  INFO [d8706509] Finished in 0.579 seconds with exit status 0 (successful).
  INFO [76bdac33] Running docker container ls --all --filter name=^sumiu-job-47180c1aab1d0dd56f986f90126523bca18468a3$ --quiet | xargs docker inspect --format '{{if .State.Health}}{{.State.Health.Status}}{{else}}{{.State.Status}}{{end}}' on web
  INFO [76bdac33] Finished in 0.194 seconds with exit status 0 (successful).
  INFO Container is running, waiting for readiness delay of 7 seconds
  INFO [69a72a88] Running docker container ls --all --filter name=^sumiu-job-47180c1aab1d0dd56f986f90126523bca18468a3$ --quiet | xargs docker inspect --format '{{if .State.Health}}{{.State.Health.Status}}{{else}}{{.State.Status}}{{end}}' on web
  INFO [69a72a88] Finished in 0.162 seconds with exit status 0 (successful).
  INFO Container is healthy!
  INFO [81bf21bf] Running docker container ls --all --filter name=^sumiu-job-58692e90c3e42f6a41490b555092ad2c804dcac1_uncommitted_eb44aaa2018ef063$ --quiet | xargs docker stop -t 30 on web
  INFO [81bf21bf] Finished in 12.437 seconds with exit status 0 (successful).
  INFO [6ccf3e48] Running docker tag luizkowalski/sumiu-app:47180c1aab1d0dd56f986f90126523bca18468a3 luizkowalski/sumiu-app:latest on web
  INFO [6ccf3e48] Finished in 0.159 seconds with exit status 0 (successful).
Prune old containers and images...
  INFO [f05dedfb] Running docker ps -q -a --filter label=service=sumiu --filter status=created --filter status=exited --filter status=dead | tail -n +6 | while read container_id; do docker rm $container_id; done on web
  INFO [f05dedfb] Finished in 0.192 seconds with exit status 0 (successful).
  INFO [2daf8d83] Running docker image prune --force --filter label=service=sumiu on web
  INFO [2daf8d83] Finished in 0.168 seconds with exit status 0 (successful).
  INFO [5a491889] Running docker image ls --filter label=service=sumiu --format '{{.ID}} {{.Repository}}:{{.Tag}}' | grep -v -w "$(docker container ls -a --format '{{.Image}}\|' --filter label=service=sumiu | tr -d '\n')luizkowalski/sumiu-app:latest\|luizkowalski/sumiu-app:<none>" | while read image tag; do docker rmi $tag; done on web
  INFO [5a491889] Finished in 0.174 seconds with exit status 0 (successful).
Releasing the deploy lock...
  Finished all in 420.0 seconds
Running the post-deploy hook...
  INFO [56748980] Running /usr/bin/env .kamal/hooks/post-deploy as luiz@localhost
  INFO [56748980] Finished in 1.004 seconds with exit status 0 (successful).

so it kinda works

igor-alexandrov · 2024-10-09T09:11:46Z

This has been rebased.

a3kov · 2024-10-28T00:51:20Z

@djmb For me there would be 2 advantages of using kamal proxy for accessories:

no need to use custom tcp port (it's actually nice because then you don't need to remember it when accessing from a new browser)
no need to have another piece of software - even if it's lightweight it still consumes resources and requires maintenance/security etc

nickhammond · 2024-10-31T00:17:51Z

I'm starting to run into this now as well with a project I'm upgrading from 1.8 to 2. It'd be great to be able to easily support accessories connecting to kamal-proxy instead of having to bring back traefik or caddy to accomplish this.

igor-alexandrov · 2024-10-31T07:24:16Z

I rebased the PR and solved the conflicts.

djmb · 2024-10-31T08:30:39Z

Thanks for the thoughts on this everyone. I've been pushing back against this mainly because there are other ways to achieve the same thing and also once you make a change like this you can't go back.

But the compelling argument for me is that this will make it easy to set up your own registry. You'd still need to supply a hostname and authentication for the registry so it's not completely plug and play yet but still a big improvement.

@igor-alexandrov - I don't have time to review this right now but I'll get to it soon 👍

igor-alexandrov · 2024-10-31T18:35:26Z

@djmb Donal, if you need any clarifications on what has been done – please let me know. We can do a call if required.

igor-alexandrov force-pushed the proxy-accessories branch from 2c6010d to 63bf121 Compare September 27, 2024 08:26

igor-alexandrov changed the title ~~Proxy accessories~~ Proxy for accessories Sep 28, 2024

igor-alexandrov marked this pull request as ready for review September 28, 2024 17:09

igor-alexandrov force-pushed the proxy-accessories branch from c2f8600 to c17bf6c Compare October 9, 2024 08:33

luizkowalski mentioned this pull request Oct 22, 2024

Using kamal-proxy for accessories #1122

Open

igor-alexandrov added 8 commits October 31, 2024 10:50

Added commands to deploy accessory to kamal-proxy

1e8567c

Added tests for accessory configuration with proxy

5917ece

Added tests for accessory deploy and remove commands

9ff509c

Added host to the expected accessory deploy command result

980209a

Fixed kamal-proxy remove command

a26ebd6

Fixed kamal-proxy remove command

6cd1eb5

Extracted proxy commands to a module

5c21c4c

Removed duplicated method

f6ec769

Updated accessory proxy to support hosts option

bdd111b

igor-alexandrov force-pushed the proxy-accessories branch from 2b4cb8f to bdd111b Compare October 31, 2024 07:22

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Proxy for accessories #981

Proxy for accessories #981

igor-alexandrov commented Sep 26, 2024 •

edited

Loading

djmb commented Sep 30, 2024

igor-alexandrov commented Sep 30, 2024

luizkowalski commented Sep 30, 2024

igor-alexandrov commented Sep 30, 2024

luizkowalski commented Sep 30, 2024

igor-alexandrov commented Sep 30, 2024

luizkowalski commented Sep 30, 2024

igor-alexandrov commented Oct 1, 2024

luizkowalski commented Oct 1, 2024

djmb commented Oct 1, 2024

igor-alexandrov commented Oct 1, 2024

luizkowalski commented Oct 1, 2024 •

edited

Loading

luizkowalski commented Oct 1, 2024

luizkowalski commented Oct 1, 2024

igor-alexandrov commented Oct 9, 2024

a3kov commented Oct 28, 2024

nickhammond commented Oct 31, 2024

igor-alexandrov commented Oct 31, 2024

djmb commented Oct 31, 2024

igor-alexandrov commented Oct 31, 2024

Proxy for accessories #981

Are you sure you want to change the base?

Proxy for accessories #981

Conversation

igor-alexandrov commented Sep 26, 2024 • edited Loading

djmb commented Sep 30, 2024

igor-alexandrov commented Sep 30, 2024

luizkowalski commented Sep 30, 2024

igor-alexandrov commented Sep 30, 2024

luizkowalski commented Sep 30, 2024

igor-alexandrov commented Sep 30, 2024

luizkowalski commented Sep 30, 2024

igor-alexandrov commented Oct 1, 2024

luizkowalski commented Oct 1, 2024

djmb commented Oct 1, 2024

igor-alexandrov commented Oct 1, 2024

luizkowalski commented Oct 1, 2024 • edited Loading

luizkowalski commented Oct 1, 2024

luizkowalski commented Oct 1, 2024

igor-alexandrov commented Oct 9, 2024

a3kov commented Oct 28, 2024

nickhammond commented Oct 31, 2024

igor-alexandrov commented Oct 31, 2024

djmb commented Oct 31, 2024

igor-alexandrov commented Oct 31, 2024

igor-alexandrov commented Sep 26, 2024 •

edited

Loading

luizkowalski commented Oct 1, 2024 •

edited

Loading