Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update Dockerfile & docker-compose for faster, usable builds #2942

Merged
merged 4 commits into from
Jan 24, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 25 additions & 18 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -1,30 +1,37 @@
# Mozilla Kinto server
FROM python:3.10-slim

FROM node:lts-bullseye-slim as node-builder
COPY /kinto/plugins/admin/package.json /kinto/plugins/admin/package-lock.json ./
RUN npm ci
COPY /kinto/plugins/admin ./
RUN npm run build

FROM python:3.10-slim-bullseye as python-builder
RUN python -m venv /opt/venv
ENV PATH="/opt/venv/bin:$PATH"
RUN apt-get update && apt-get install -y --no-install-recommends build-essential libpq-dev
COPY requirements.txt .
RUN pip install --upgrade pip && \
pip install -r requirements.txt && \
pip install kinto-attachment kinto-emailer kinto-elasticsearch kinto-portier kinto-redis httpie

FROM python:3.10-slim-bullseye
RUN apt-get update && apt-get install -y --no-install-recommends libpq-dev
RUN groupadd --gid 10001 app && \
useradd --uid 10001 --gid 10001 --home /app --create-home app

WORKDIR /app
COPY --from=python-builder /opt/venv /opt/venv
COPY . /app
COPY --from=node-builder /build ./kinto/plugins/admin/build

ENV KINTO_INI /etc/kinto/kinto.ini
ENV PORT 8888
ENV KINTO_INI=/etc/kinto/kinto.ini \
PORT=8888 \
PATH="/opt/venv/bin:$PATH"

# Install build dependencies, build the virtualenv and remove build
# dependencies all at once to build a small image.
RUN \
apt-get update && \
apt-get install -y g++ gcc libpq5 curl libssl-dev libffi-dev libpq-dev gnupg2 && \
curl -sL https://deb.nodesource.com/setup_10.x | bash - && \
apt-get install -y nodejs && \
cd kinto/plugins/admin && npm install && npm run build && \
pip3 install --upgrade pip && \
pip3 install -e /app[postgresql,memcached,monitoring] -c /app/requirements.txt && \
pip3 install kinto-attachment kinto-emailer kinto-elasticsearch kinto-portier kinto-redis && \
pip3 install httpie && \
kinto init --ini $KINTO_INI --host 0.0.0.0 --backend=memory --cache-backend=memory && \
apt-get purge -y -qq g++ gcc libssl-dev libffi-dev libpq-dev curl nodejs && \
apt-get autoremove -y -qq && \
apt-get clean -y
pip install -e /app[postgresql,memcached,monitoring] -c /app/requirements.txt && \
kinto init --ini $KINTO_INI --host 0.0.0.0 --backend=memory --cache-backend=memory

USER app
# Run database migrations and start the kinto server
Expand Down
48 changes: 27 additions & 21 deletions docker-compose.yml
Original file line number Diff line number Diff line change
@@ -1,21 +1,27 @@
db:
image: postgres
environment:
POSTGRES_USER: postgres
POSTGRES_PASSWORD: postgres
cache:
image: library/memcached
web:
image: kinto/kinto-server
links:
- db
- cache
ports:
- "8888:8888"
environment:
KINTO_CACHE_BACKEND: kinto.core.cache.memcached
KINTO_CACHE_HOSTS: cache:11211 cache:11212
KINTO_STORAGE_BACKEND: kinto.core.storage.postgresql
KINTO_STORAGE_URL: postgresql://postgres:postgres@db/postgres
KINTO_PERMISSION_BACKEND: kinto.core.permission.postgresql
KINTO_PERMISSION_URL: postgresql://postgres:postgres@db/postgres
version: "3"
services:
db:
image: postgres:14
environment:
POSTGRES_NAME: postgres
POSTGRES_USER: postgres
POSTGRES_PASSWORD: postgres
cache:
image: memcached:1
web:
build:
context: .
dockerfile: Dockerfile
image: kinto/kinto-server:latest
depends_on:
- db
- cache
ports:
- "8888:8888"
environment:
KINTO_CACHE_BACKEND: kinto.core.cache.memcached
KINTO_CACHE_HOSTS: cache:11211 cache:11212
KINTO_STORAGE_BACKEND: kinto.core.storage.postgresql
KINTO_STORAGE_URL: postgresql://postgres:postgres@db/postgres
KINTO_PERMISSION_BACKEND: kinto.core.permission.postgresql
KINTO_PERMISSION_URL: postgresql://postgres:postgres@db/postgres
2 changes: 1 addition & 1 deletion requirements.txt
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ jsonpatch==1.32
jsonschema==4.4.0
logging-color-formatter==1.0.2
newrelic==7.2.4.171
psycopg2-binary==2.9.3
psycopg2==2.9.3
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change is because kinto[postgresql] requires psycopg2 (not psycopg2-binary)

Copy link
Member

@Natim Natim Jan 21, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

psycopg2-binary is psycopg2 precompiled, we should keep the binary version IMHO.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

psycopg2-binary is psycopg2 precompiled, we should keep the binary version IMHO.

I would prefer that as well, but I think in order to do this, we'll have to change psycopg2 to psycopg2-binary in setup.cfg. Would that be acceptable? I didn't know if there was a good reason we needed to build psycopg2 when installing kinto.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it makes sense to align the dependencies names. Shall we try psycopg2-binary everywhere?

Copy link
Contributor Author

@grahamalama grahamalama Jan 24, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shall we try psycopg2-binary everywhere?

Yep, we certainly can. I was just ensuring there wasn't a good reason to build psycopg2. From their docs:

If you are the maintainer of a published package depending on psycopg2 you shouldn’t use psycopg2-binary as a module dependency. For production use you are advised to use the source distribution.

and

Warning
The psycopg2 wheel package comes packaged, among the others, with its own libssl binary. This may create conflicts with other extension modules binding with libssl as well, for instance with the Python ssl module: in some cases, under concurrency, the interaction between the two libraries may result in a segfault. In case of doubts you are advised to use a package built from source.

Copy link
Contributor Author

@grahamalama grahamalama Jan 24, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Natim I'm going to merge this PR as-is, then open a follow-up issue where we can further discuss using psycopg2 vs psycopg2-binary

EDIT:
Follow-up issue #2943

pyramid==2.0
pyramid-mailer==0.15.1
pyramid-multiauth==1.0.1
Expand Down