-
-
Notifications
You must be signed in to change notification settings - Fork 2
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Kubernetes for local dev #118
Commits on Jun 16, 2023
-
Using Kubernetes from https://k3s.io/ for local development, we don't have a weekly-snapshots-retain-4 storage class, so we need local-path. Also, local-path doesn't allow ReadWriteMany -- but we don't need that anyway since all of our containers are running in the same pod, and therefore on the same node. ReadWriteMany is only required for volumes being mounted from different nodes.
Configuration menu - View commit details
-
Copy full SHA for 74049bf - Browse repository at this point
Copy the full SHA 74049bfView commit details -
Remove namespace from secrets for now
I'll put the namespace back everywhere eventually. But for local dev, we don't (yet) need a namespace.
Configuration menu - View commit details
-
Copy full SHA for 7174aad - Browse repository at this point
Copy the full SHA 7174aadView commit details -
Fix "invalid base64" error in secrets.yaml
The `data:` key tells Kubernetes these are base64-encoded values. If they are not encoded, the `stringData` key is correct.
Configuration menu - View commit details
-
Copy full SHA for dd62dc6 - Browse repository at this point
Copy the full SHA dd62dc6View commit details -
Populate Kubernetes dev secrets
These are the same values found in the .env file checked into the repo, so this commit is not exposing any actual secrets, just the ones we use during local development.
Configuration menu - View commit details
-
Copy full SHA for e37a7b3 - Browse repository at this point
Copy the full SHA e37a7b3View commit details -
First pass at a Kubernetes setup for local dev
This creates a container that will hold the Git repo and allow us to push to it. Next we'll set up the frontend and backend containers to watch that repo.
Configuration menu - View commit details
-
Copy full SHA for 34a824b - Browse repository at this point
Copy the full SHA 34a824bView commit details -
Add "localdev" task to get git container running
The "localdev" task does everything needed to get a Git container up and running, but does not (yet) create a .ssh/config entry for you.
Configuration menu - View commit details
-
Copy full SHA for 7adca7c - Browse repository at this point
Copy the full SHA 7adca7cView commit details -
Configuration menu - View commit details
-
Copy full SHA for d8ec41a - Browse repository at this point
Copy the full SHA d8ec41aView commit details -
Use Skaffold instead of custom-built solution
This will enable us to run `skaffold dev` and have a live-reloading version of our app running in Kubernetes.
Configuration menu - View commit details
-
Copy full SHA for c24372a - Browse repository at this point
Copy the full SHA c24372aView commit details -
Increase memory limits so Hasura isn't OOM killed
The hasura container was being OOMKilled, so we'll try bumping its memory allocation and its hard limit.
Configuration menu - View commit details
-
Copy full SHA for e9d475c - Browse repository at this point
Copy the full SHA e9d475cView commit details -
Fix CORS error in local k8s dev
Long-term, we need to specify exactly which clients (e.g., the frontend box) are allowed to connect, but AllowAnyOrigin is good enough during dev work.
Configuration menu - View commit details
-
Copy full SHA for 2bb077e - Browse repository at this point
Copy the full SHA 2bb077eView commit details -
Configuration menu - View commit details
-
Copy full SHA for 08f5f88 - Browse repository at this point
Copy the full SHA 08f5f88View commit details -
Restore the k8s "languagedepot" namespace
All k8s deployment files now use the "languagedepot" namespace. To use that namespace by default and not have to specify `-n lanugagedepot` all the time, add `namespace: languagedepot` to your ~/.kube/config file in the appropriate context stanza, just after the cluster: and user: lines.
Configuration menu - View commit details
-
Copy full SHA for 351cb12 - Browse repository at this point
Copy the full SHA 351cb12View commit details
Commits on Jun 19, 2023
-
Fix login process - use Svelte-Kit load properly
Now the login process will redirect to a server page, forcing a cookie reload so that the rest of the app will correctly get the user from the load() function in src/routes. This will mean that the user store should no longer be stale.
Configuration menu - View commit details
-
Copy full SHA for 37032f5 - Browse repository at this point
Copy the full SHA 37032f5View commit details -
The `crypto.subtle` module is a browser-only module; if we're doing server-side rendering then we need the NodeJS equivalent.
Configuration menu - View commit details
-
Copy full SHA for 9d3eb39 - Browse repository at this point
Copy the full SHA 9d3eb39View commit details -
Add k8s deployment setup for docker-desktop
Docker desktop's embedded Kubernetes doesn't have an ingress controller, so part of local k8s dev must include deploying one. It also doesn't have local-path storageClass, so we have to use hostpath.
Configuration menu - View commit details
-
Copy full SHA for c87d0a3 - Browse repository at this point
Copy the full SHA c87d0a3View commit details -
Work around issue with recent Node Docker images
Recent docker images of Node 20 have an issue where node-gyp's postinstall step can fail because it's trying to replace a binary while it's still running. Moving to Node 18 Docker images, just for now, to avoid the problem.
Configuration menu - View commit details
-
Copy full SHA for f28baa0 - Browse repository at this point
Copy the full SHA f28baa0View commit details -
Node 18 doesn't have webcrypto module
Can't pre-hash the password if the page was SSR rendered, so as long as we're on Node 18, we need to turn off password prehashing.
Configuration menu - View commit details
-
Copy full SHA for f83ee84 - Browse repository at this point
Copy the full SHA f83ee84View commit details -
Turn off https during local dev
If there's a tls config block in the ingress deployment YAML, it sets HSTS for dev.languagedepot.org even though the TLS is only configured for staging.languagedepot.org, and that causes issues. In the future we'll use kustomize.yaml to configure this.
Configuration menu - View commit details
-
Copy full SHA for 64d98e8 - Browse repository at this point
Copy the full SHA 64d98e8View commit details -
Configuration menu - View commit details
-
Copy full SHA for a8d5ad6 - Browse repository at this point
Copy the full SHA a8d5ad6View commit details
Commits on Jun 20, 2023
-
Configuration menu - View commit details
-
Copy full SHA for b1c9efb - Browse repository at this point
Copy the full SHA b1c9efbView commit details
Commits on Jun 21, 2023
-
Configuration menu - View commit details
-
Copy full SHA for ee8ebc6 - Browse repository at this point
Copy the full SHA ee8ebc6View commit details
Commits on Jun 23, 2023
-
Attempt to split Hasura into separate deployment
Currently not working: Hasura is saying "inconsistent metadata" and claiming that the tables Projects and Users don't exist.
Configuration menu - View commit details
-
Copy full SHA for 2775ff8 - Browse repository at this point
Copy the full SHA 2775ff8View commit details -
Configuration menu - View commit details
-
Copy full SHA for e03e891 - Browse repository at this point
Copy the full SHA e03e891View commit details
Commits on Jun 26, 2023
-
Configuration menu - View commit details
-
Copy full SHA for 2187c87 - Browse repository at this point
Copy the full SHA 2187c87View commit details -
Have hasura pod wait for backend to be healthy
Since Hasura needs the backend to be up and healthy before it will accept its metadata, we change `wait-db` to `wait-lexbox` in the Hasura container (and the lexbox deployment no longer waits for Hasura, but instead waits for the database to be up and running).
Configuration menu - View commit details
-
Copy full SHA for 8d99fb3 - Browse repository at this point
Copy the full SHA 8d99fb3View commit details -
Rename hg-repos-pvc.yaml to just pvc.yaml
This file will end up containing all PersistentVolumeClaims, so that it can be deployed separately from skaffold
Configuration menu - View commit details
-
Copy full SHA for d1767ef - Browse repository at this point
Copy the full SHA d1767efView commit details -
Move db PVC into pvc.yaml and create skaffold hook
This way skaffold won't consider that it deployed the persistent volume claims in pvc.yaml and won't clean them up after skaffold dev.
Configuration menu - View commit details
-
Copy full SHA for d94f200 - Browse repository at this point
Copy the full SHA d94f200View commit details -
So that if it's run first, it can still deploy into the languagedepot namespace. The duplicate namespace.yaml won't cause any issues.
Configuration menu - View commit details
-
Copy full SHA for 69fa1db - Browse repository at this point
Copy the full SHA 69fa1dbView commit details -
Configuration menu - View commit details
-
Copy full SHA for de86efb - Browse repository at this point
Copy the full SHA de86efbView commit details -
Increase memory for dev containers
Backend and ui containers were getting OOM killed because the build process uses more memory than we had allocated. Eventually we'll want these values to be dynamically replaced by skaffold dev so that the memory allocations for the deployed app stay low.
Configuration menu - View commit details
-
Copy full SHA for 2564637 - Browse repository at this point
Copy the full SHA 2564637View commit details -
Greatly increase memory request for dotnet watch
The dotnet watch command is memory-hungry, and if the memory request is too low, the container gets OOM killed and then the new dotnet watch has to re-restore all the packages, which takes a while. Upping the memory limit means that dotnet watch is able to do its job properly.
Configuration menu - View commit details
-
Copy full SHA for 52574d6 - Browse repository at this point
Copy the full SHA 52574d6View commit details -
Configuration menu - View commit details
-
Copy full SHA for 89c73c2 - Browse repository at this point
Copy the full SHA 89c73c2View commit details
Commits on Jun 27, 2023
-
Pre-populate Sena-3 repo in k8s deployment
Eventually we'll want this to be done only on dev and staging.
Configuration menu - View commit details
-
Copy full SHA for 172c7c2 - Browse repository at this point
Copy the full SHA 172c7c2View commit details -
Configuration menu - View commit details
-
Copy full SHA for 1942832 - Browse repository at this point
Copy the full SHA 1942832View commit details -
Deploy Hasura metadata as part of container image
Instead of mounting the hasura-metadata volume, we'll just go back to inserting it as part of the Docker metadata. There's no reason to add the complexity of trying to sync the volume. If the metadata changes, we'll just let skaffold rebuild the Docker image.
Configuration menu - View commit details
-
Copy full SHA for e146b51 - Browse repository at this point
Copy the full SHA e146b51View commit details -
Remove db deployment from skaffold dev
Now you will run `task k8s:db-up` (which may become just `task db-up` and will eventually be folded into `task infra-up`) to bring up the database, so that it will be persistent and will not get torn down when you exit skaffold dev. Then once you run `task db-up`, you can run `skaffold dev` and it will have the database ready for you.
Configuration menu - View commit details
-
Copy full SHA for 28bc26c - Browse repository at this point
Copy the full SHA 28bc26cView commit details -
Hasura no longer needs to wait on backend
Now that the database is persistent, the only time Hasura will give an error is the first time you spin up the whole system, when the database is completely empty. And even there, we may be able to avoid that by having an initContainer to check that `SELECT * FROM Projects LIMIT 1` returns one line rather than zero, which means that the database has been populated by the backend's SeedingData.SeedDatabase task.
Configuration menu - View commit details
-
Copy full SHA for 6c0ea9b - Browse repository at this point
Copy the full SHA 6c0ea9bView commit details
Commits on Jun 28, 2023
-
Attempt to use Kustomize for dev mode
This allows us to run `skaffold dev` using docker-desktop, but leave the original config in place for staging.
Configuration menu - View commit details
-
Copy full SHA for 78ada23 - Browse repository at this point
Copy the full SHA 78ada23View commit details -
Configuration menu - View commit details
-
Copy full SHA for 67b1c6c - Browse repository at this point
Copy the full SHA 67b1c6cView commit details -
Configuration menu - View commit details
-
Copy full SHA for 90b5550 - Browse repository at this point
Copy the full SHA 90b5550View commit details -
Configuration menu - View commit details
-
Copy full SHA for aee8942 - Browse repository at this point
Copy the full SHA aee8942View commit details -
Configuration menu - View commit details
-
Copy full SHA for aa3f735 - Browse repository at this point
Copy the full SHA aa3f735View commit details
Commits on Jun 29, 2023
-
Update Database Startup procedures
Updated database startup procedures to handle migration requests with added condition on development environment or command line argument. Introduced conditional autoExecuteMigrations in AddLexData to only AddHostedService when in development environment. Updated TestServicesFixtures to build service providers with migrations applied. Adjusted DbStartupService to throw exception when migrations are not executed, improving error visibility. Furthermore, added logging to track migration application duration.
Configuration menu - View commit details
-
Copy full SHA for 4cf5faa - Browse repository at this point
Copy the full SHA 4cf5faaView commit details -
Configuration menu - View commit details
-
Copy full SHA for ced78b8 - Browse repository at this point
Copy the full SHA ced78b8View commit details -
Move Hasura container back inside lexbox pod
The lexbox pod will be running DB migrations on startup, and the Hasura container needs to see a current DB version (with all migrations run) at the time it starts up. So we put both containers in the same pod and use an initContainer to run the migrations.
Configuration menu - View commit details
-
Copy full SHA for 212ad16 - Browse repository at this point
Copy the full SHA 212ad16View commit details -
Configuration menu - View commit details
-
Copy full SHA for 60ac444 - Browse repository at this point
Copy the full SHA 60ac444View commit details
Commits on Jun 30, 2023
-
Speed up Docker builds by caching packages
Both the backend and frontend builds benefit greatly by caching NuGet or NPM packages from build to build, using Docker's built-in caching mechanism. You get warnings from pnpm that cross-device symlinks are not allowed (becuse the pnpm store is located on the cache volume), but it falls back to copying the NPM package files rather than symlinking them, which still saves a lot of time compared to downloading them.
Configuration menu - View commit details
-
Copy full SHA for 0547001 - Browse repository at this point
Copy the full SHA 0547001View commit details -
Attempt to cache nuget packages in k8s dev
This doesn't work: the custom storageClass does retain the data, but then the next PVC is considered a *different* claim and doesn't grab the retained data. We'll go for a simpler solution where we just cache the packages in the Docker image.
Configuration menu - View commit details
-
Copy full SHA for eecf9af - Browse repository at this point
Copy the full SHA eecf9afView commit details -
Configuration menu - View commit details
-
Copy full SHA for f7a88cb - Browse repository at this point
Copy the full SHA f7a88cbView commit details -
Reorder skaffold.yaml to be slightly more robust
With the backend and frontend in slots 0 and 1 of the dockerfile definitions, it's far less likely to accidentally mess up which Dockerfile is associated with which container.
Configuration menu - View commit details
-
Copy full SHA for f09a80f - Browse repository at this point
Copy the full SHA f09a80fView commit details -
Put backend before frontend in skaffold.yaml
It makes a little more sense for the backend to be built first and the frontend second.
Configuration menu - View commit details
-
Copy full SHA for 95c1bae - Browse repository at this point
Copy the full SHA 95c1baeView commit details
Commits on Jul 3, 2023
-
Put staging deployment back how it was
We've decided to run skaffold with `--cleanup=false` so that it won't delete the dpeloyments (and persistent volumes!) that it creates. That way our task of setting it up is much simpler.
Configuration menu - View commit details
-
Copy full SHA for bc031c6 - Browse repository at this point
Copy the full SHA bc031c6View commit details
Commits on Jul 4, 2023
-
Update skaffold to version 2.6.0
Released last week, has some bugfixes that might help.
Configuration menu - View commit details
-
Copy full SHA for 3c70976 - Browse repository at this point
Copy the full SHA 3c70976View commit details -
Remove dev.languageforge.org from staging k8s proxy
Staging config should only know about staging.languagedepot.org; we can add the dev.languageforge.org in a patch in dev/kustomize.yaml.
Configuration menu - View commit details
-
Copy full SHA for ac02e28 - Browse repository at this point
Copy the full SHA ac02e28View commit details -
Node bug is fixed, revert to current Node version
The Node bug only triggered on certain Linux kernel versions, and the latest Docker Desktop release has a version that doesn't trigger the bug any more. So we can go back to using the latest version of Node in the frontend build.
Configuration menu - View commit details
-
Copy full SHA for 5737454 - Browse repository at this point
Copy the full SHA 5737454View commit details -
Have Vite listen on all interfaces
Vite only listens on the loopback interface by default, which is fine for local development but doesn't work in a Kubernetes container, where the container is assigned its own cluster-local IP address that you're supposed to listen to. Telling vite to listen to 0.0.0.0 means "all available interfaces" which is what we want for k8s dev.
Configuration menu - View commit details
-
Copy full SHA for 77e0155 - Browse repository at this point
Copy the full SHA 77e0155View commit details -
Use correct port for ingress-lexbox communication
On dev, the lexbox container is running on port 5158, **not** on port 80. So the ingress controller couldn't talk to it, which explains all the 502 errors I'd been seeing until now.
Configuration menu - View commit details
-
Copy full SHA for c0f2ad4 - Browse repository at this point
Copy the full SHA c0f2ad4View commit details
Commits on Jul 5, 2023
-
Add skaffold profiles for running locally
These profiles let you run the frontend, or the frontend *and* the backend, locally.
Configuration menu - View commit details
-
Copy full SHA for 8d9fe0a - Browse repository at this point
Copy the full SHA 8d9fe0aView commit details -
Add DOTNET_URLS env to lexbox in dev mode
Should avoid a problem where ASP.NET Core listens on the wrong interface
Configuration menu - View commit details
-
Copy full SHA for 9ca7d76 - Browse repository at this point
Copy the full SHA 9ca7d76View commit details -
Configuration menu - View commit details
-
Copy full SHA for bd4e905 - Browse repository at this point
Copy the full SHA bd4e905View commit details -
Upgrade NuGet packages in Testing.csproj
Some of the changes in develop created package conflicts, which are solved by upgrading the SIL.Core and SIL.Chorus packages.
Configuration menu - View commit details
-
Copy full SHA for 9f0a038 - Browse repository at this point
Copy the full SHA 9f0a038View commit details -
Configuration menu - View commit details
-
Copy full SHA for 4f19dfb - Browse repository at this point
Copy the full SHA 4f19dfbView commit details
Commits on Jul 6, 2023
-
only proxy from vite in local dev not in a container. Forward some mi…
…ssing ports for doing dotnet locally.
Configuration menu - View commit details
-
Copy full SHA for 1859901 - Browse repository at this point
Copy the full SHA 1859901View commit details -
Configuration menu - View commit details
-
Copy full SHA for 8784cb5 - Browse repository at this point
Copy the full SHA 8784cb5View commit details -
Configuration menu - View commit details
-
Copy full SHA for 311ff21 - Browse repository at this point
Copy the full SHA 311ff21View commit details -
Configuration menu - View commit details
-
Copy full SHA for 345cd3c - Browse repository at this point
Copy the full SHA 345cd3cView commit details -
Configuration menu - View commit details
-
Copy full SHA for 3d51cc4 - Browse repository at this point
Copy the full SHA 3d51cc4View commit details -
Configuration menu - View commit details
-
Copy full SHA for 560bacd - Browse repository at this point
Copy the full SHA 560bacdView commit details -
Configuration menu - View commit details
-
Copy full SHA for 3a32b67 - Browse repository at this point
Copy the full SHA 3a32b67View commit details
Commits on Jul 7, 2023
-
Configuration menu - View commit details
-
Copy full SHA for e686f3e - Browse repository at this point
Copy the full SHA e686f3eView commit details
Commits on Jul 17, 2023
-
Merge branch 'develop' into chore/k8s-for-local-dev
# Conflicts: # deployment/base/lexbox-deployment.yaml
Configuration menu - View commit details
-
Copy full SHA for 0a5f005 - Browse repository at this point
Copy the full SHA 0a5f005View commit details -
Configuration menu - View commit details
-
Copy full SHA for 2cba39e - Browse repository at this point
Copy the full SHA 2cba39eView commit details -
Configuration menu - View commit details
-
Copy full SHA for 123cd1b - Browse repository at this point
Copy the full SHA 123cd1bView commit details -
Configuration menu - View commit details
-
Copy full SHA for 4d914cf - Browse repository at this point
Copy the full SHA 4d914cfView commit details -
Configuration menu - View commit details
-
Copy full SHA for 50f6279 - Browse repository at this point
Copy the full SHA 50f6279View commit details -
Configuration menu - View commit details
-
Copy full SHA for 7032010 - Browse repository at this point
Copy the full SHA 7032010View commit details -
Configuration menu - View commit details
-
Copy full SHA for 17775dd - Browse repository at this point
Copy the full SHA 17775ddView commit details -
Configuration menu - View commit details
-
Copy full SHA for 1658b5f - Browse repository at this point
Copy the full SHA 1658b5fView commit details -
Configuration menu - View commit details
-
Copy full SHA for 0979a42 - Browse repository at this point
Copy the full SHA 0979a42View commit details -
fix issue with deployments still trying to reference secrets. Fix mis…
…sing env variable for hgweb
Configuration menu - View commit details
-
Copy full SHA for aa2ed76 - Browse repository at this point
Copy the full SHA aa2ed76View commit details
Commits on Jul 18, 2023
-
Configuration menu - View commit details
-
Copy full SHA for fcf0d8c - Browse repository at this point
Copy the full SHA fcf0d8cView commit details -
change dotnet port to 5158 instead of 80 across the board. Expose db …
…via port forwarding in dev
Configuration menu - View commit details
-
Copy full SHA for 56394d4 - Browse repository at this point
Copy the full SHA 56394d4View commit details -
Rearrange README a bit, mention extra Windows setup
Adding lines to /etc/hosts isn't actually needed on Linux, but is necessary on Windows, so we'll mention that. And we'll rearrage the README to put the extra setup steps immediately after the setup.
Configuration menu - View commit details
-
Copy full SHA for b21be5b - Browse repository at this point
Copy the full SHA b21be5bView commit details -
Revert "Add "localdev" task to get git container running"
This reverts commit 7adca7c.
Configuration menu - View commit details
-
Copy full SHA for f152551 - Browse repository at this point
Copy the full SHA f152551View commit details -
Remove localdev-deployment.yaml
This was an early attempt at getting k8s working for local dev, obsoleted by using skaffold.
Configuration menu - View commit details
-
Copy full SHA for 73901a5 - Browse repository at this point
Copy the full SHA 73901a5View commit details