Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ugrade base system to Ubuntu 24.04 #1067

Closed
wants to merge 9 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -40,9 +40,11 @@ RUN apt-get update && DEBIAN_FRONTEND="noninteractive" apt-get install -y --no-i
xlsx2csv \
gh \
nodejs \
npm \
graphviz \
python3-psycopg2 \
swi-prolog
swi-prolog \
libpcre3

# Install run-time dependencies for Soufflé.
RUN DEBIAN_FRONTEND="noninteractive" apt-get install -y --no-install-recommends \
Expand Down Expand Up @@ -95,7 +97,7 @@ COPY scripts/obodash /tools
RUN chmod +x /tools/obodash && \
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fuck it, we are not going to create a virtual environment just for the
OBO Dashboard. If the OBO Dashboard dependencies end up breaking the
rest of our Python packages, we might as well just create a separate
Docker image just for the dashboard.

Your commit messages are great, but it's important to avoid using offensive language in them. Thank you!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry but I won’t take lessons in how to write commit messages from folks who can’t be bothered to write decent commit messages themselves.

If people complain about an occasional profanity in one of my commit messages, at least I will know that there are some people who read them.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct me if I am wrong but I dont think this was meant as a lesson, more like a statement of affectedness.

I also prefer to avoid strong language in the spirit of conduct (as the Covenant Code of Conduct suggests), I had endless discussions of misconduct in some other projects which just distract from our common goals.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct me if I am wrong

My pleasure.

I dont think this was meant as a lesson, more like a statement of affectedness.

A “statement of affectedness” would have been something like, “I felt offended by the presence of that F-word, I don’t like reading such things when I am reviewing a PR. I’d appreciate if you abstained to do that“ or similar.

“It’s important to avoid using offensive language in [commit messages]” is absolutely a lesson in how one should write a commit message – one that I find hard to accept coming from someone who I can’t remember having ever seen write a single decent commit message that actually explains what they are trying to do.

as the Covenant Code of Conduct suggests

The ODK does not have a code of conduct. Maybe it should (though I’d much rather have a “code on how to write efficient commit messages so that future contributors don’t have to guess what the f... the developers had in mind when they committed that change”, but it’s OK, we can have different priorities), but currently it does not.

Even if it had, according to the most popular code of conducts in use out there, the correct response to the occasional use of inappropriate language is a “private, written warning from community leaders, providing clarity around the nature of the violation and an explanation of why the behavior was inappropriate” – not a public rebuke.

I had endless discussions of misconduct in some other projects which just distract from our common goals

Fine. I hereby publicly apologise for having expressed my state of mind, as to why I believe installing the OBO Dashboard in a virtual environment just to circumvent pip’s default behaviour would be a unnecessary complication, using a 6-letters profanity instead of a more elaborate, professional, and polite phrasing. I did that for the sake of efficiency, but I didn’t consider that indeed people might not want to read a F-word when they are busy reviewing a PR.

I hope this will end the current distraction, and I will temporarily withdraw for all ODK maintenance to be sure you are not distracted again.

git clone --depth 1 https://github.com/OBOFoundry/OBO-Dashboard.git && \
cd OBO-Dashboard && \
python3 -m pip install -r requirements.txt && \
python3 -m pip install -r requirements.txt --break-system-packages && \
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The OBO Dashboard must now be installed with --break-system-packages, otherwise pip will flatly refuse to install it system-wide.

I wonder if this is very suboptimal and we should, instead, install OBO dashboard as a pypi package and install it the usual way? Why is --break-system-packages not needed in docker/builder/Dockerfile?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What are the break system packages? I can update to the latest version, if possible.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if this is very suboptimal and we should, instead, install OBO dashboard as a pypi package and install it the usual way?

We could, but then we would need actual releases of the OBO Dashboard package whenever we want to update it in the ODK. For now, the OBO Dashboard is effectively working in a “rolling release” fashion: it doesn’t do releases and we always install it from the tip of the master branch. That would need to change.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What are the break system packages?

What do you mean? We are not breaking anything. --break-system-packages is merely the name of the option that forces pip to accept an instruction to install a package in the system-wide site-packages directory – without that option, it flatly refuses to do, because in Ubuntu 24.04 the system-wide directory is considered to be the private garden of the distribution’s own package system (APT).

The option is named like this as a rather condescending way of telling end-users that they should not try to install anything in the system-wide directory, because if they do they run the risk of introducing a conflict between packages that have been installed there by the distribution (packages installed with apt-get install python3-mypackage) and packages installed with pip.

But in a Docker image, we mostly don’t care about that risk, since when the image is built, the system-wide directory is in effect read-only. Once we are done installing whatever we need in it, nobody will ever modify that directory ever again.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That would need to change.

I can create a release right now.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is --break-system-packages not needed in docker/builder/Dockerfile?

Because in docker/builder/Dockerfile we are not installing Python packages to the system-wide directory: we are installing them to a staging area, from where we copy them to the ODKLite/ODKFull images.

echo " " >> Makefile && \
echo "build/robot.jar:" >> Makefile && \
echo " echo 'skipped ROBOT jar download.....' && touch \$@" >> Makefile && \
Expand Down
4 changes: 2 additions & 2 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@ docs:
@ODK_IMAGE=odklite ./odk.sh python ./odk/schema_documentation.py

# Building docker image
VERSION = "v1.5"
VERSION = "v1.6"
IM=obolibrary/odkfull
IMLITE=obolibrary/odklite
ROB=obolibrary/robot
Expand Down Expand Up @@ -168,7 +168,7 @@ publish-multiarch-dev:
.

constraints.txt: requirements.txt.full
docker run -v $$PWD:/work -w /work --rm -ti obolibrary/odkbuild:latest /work/update-constraints.sh
docker run -v $$PWD:/work -w /work --rm -ti obolibrary/odkbuild:latest /work/update-constraints.sh --install-virtualenv

clean-tests:
rm -rf target/*
Expand Down
26 changes: 13 additions & 13 deletions constraints.txt
Original file line number Diff line number Diff line change
Expand Up @@ -12,12 +12,12 @@ async-lru==2.0.4
attrs==23.2.0
Babel==2.15.0
babelon==0.2.9
backports.tarfile==1.1.1
backports.tarfile==1.2.0
bcp47==0.1.0
beautifulsoup4==4.12.3
bidict==0.23.1
bleach==6.1.0
bmt==1.4.1
bmt==1.4.2
cachetools==5.3.3
cattrs==23.2.3
certifi==2024.2.2
Expand Down Expand Up @@ -59,13 +59,13 @@ executing==2.0.1
fastjsonschema==2.19.1
fastobo==0.12.3
filelock==3.14.0
fonttools==4.52.1
fonttools==4.52.4
fqdn==1.5.1
funowl==0.2.3
ghp-import==2.1.0
google==3.0.0
google-api-core==2.19.0
google-api-python-client==2.130.0
google-api-python-client==2.131.0
google-auth==2.29.0
google-auth-httplib2==0.2.0
google-auth-oauthlib==1.2.0
Expand All @@ -88,7 +88,7 @@ inflection==0.5.1
iniconfig==2.0.0
ipykernel==6.29.4
ipython==8.24.0
ipywidgets==8.1.2
ipywidgets==8.1.3
isodate==0.6.1
isoduration==20.11.0
j2cli==0.3.10
Expand Down Expand Up @@ -125,7 +125,7 @@ jupyter_server_terminals==0.5.3
jupyterlab==4.2.1
jupyterlab_pygments==0.3.0
jupyterlab_server==2.27.2
jupyterlab_widgets==3.0.10
jupyterlab_widgets==3.0.11
keyring==25.2.1
kgcl-rdflib==0.5.0
kgcl_schema==0.6.8
Expand Down Expand Up @@ -177,8 +177,8 @@ ontobio==2.9.0
ontodev-cogs==0.3.3
ontodev-gizmos==0.3.2
ontoportal-client==0.0.4
openai==1.30.3
openpyxl==3.1.2
openai==1.30.4
openpyxl==3.1.3
ordered-set==4.1.0
overrides==7.7.0
packaging==24.0
Expand All @@ -200,7 +200,7 @@ prefixcommons==0.1.12
prefixmaps==0.2.4
prologterms==0.0.6
prometheus_client==0.20.0
prompt-toolkit==3.0.43
prompt_toolkit==3.0.45
pronto==2.5.7
proto-plus==1.23.0
protobuf==4.25.3
Expand All @@ -211,8 +211,8 @@ pyarrow==15.0.2
pyasn1==0.6.0
pyasn1_modules==0.4.0
pycparser==2.22
pydantic==2.7.1
pydantic_core==2.18.2
pydantic==2.7.2
pydantic_core==2.18.3
pydotplus==2.0.2
PyGithub==2.3.0
Pygments==2.18.0
Expand Down Expand Up @@ -248,7 +248,7 @@ readme_renderer==43.0
recommonmark==0.7.1
referencing==0.35.1
regex==2024.5.15
requests==2.32.2
requests==2.32.3
requests-cache==1.2.0
requests-oauthlib==2.0.0
requests-toolbelt==1.0.0
Expand Down Expand Up @@ -320,7 +320,7 @@ wcwidth==0.2.13
webcolors==1.13
webencodings==0.5.1
websocket-client==1.8.0
widgetsnbextension==4.0.10
widgetsnbextension==4.0.11
wrapt==1.16.0
xmltodict==0.13.0
yamldown==0.1.8
Expand Down
7 changes: 4 additions & 3 deletions docker/builder/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
# (typically because pre-compiled binaries don't exist for arm64) and
# we don't want to build directly on the final image (to avoid
# cluttering the image with build-time dependencies).
FROM ubuntu:22.04
FROM ubuntu:24.04
WORKDIR /build

# Software versions
Expand All @@ -27,8 +27,7 @@ RUN apt-get update && \
rustc \
cargo \
python3-dev \
python3-pip \
python3-virtualenv
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can no longer have virtualenv in the builder image, as it messes with the installation of our Python packages.

What about pipelines based on ODK that require the ability to install a virtual environment before running system? Seems quite fundamental a thing to remove from the image.. How does it "mess"?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are talking about the builder image here! This does not remove anything from the final ODKFull!

Having virtualenv installed in the builder image creates a mess because

the Ubuntu package for virtualenv automatically installs platformdirs version 2.5.1, which then prevents us from installing the platformdirs 4.x that we need as a dependency for some of our packages.

python3-pip

# Build the Python packages.
# On x86_64, most if not all of these packages should be available as
Expand All @@ -45,6 +44,7 @@ COPY constraints.txt /build/constraints.txt
RUN python3 -m pip install \
-r /build/requirements.txt.lite \
-c /build/constraints.txt \
--no-warn-script-location \
--root /staging/lite
# Then those needed by the odkfull image.
# After installing those packages, we forcibly remove from the odkfull
Expand All @@ -54,6 +54,7 @@ RUN python3 -m pip install \
RUN python3 -m pip install \
-c /build/constraints.txt \
-r /build/requirements.txt \
--no-warn-script-location \
--root /staging/full && \
cd /staging/lite && \
find . -type f | while read f ; do rm -f /staging/full/$f ; done && \
Expand Down
2 changes: 1 addition & 1 deletion docker/odklite/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
FROM ubuntu:22.04
FROM ubuntu:24.04
LABEL maintainer="[email protected]"

ENV ROBOT_VERSION=1.9.6
Expand Down
2 changes: 1 addition & 1 deletion docker/robot/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
FROM ubuntu:22.04
FROM ubuntu:24.04
LABEL maintainer="[email protected]"

WORKDIR /tools
Expand Down
6 changes: 3 additions & 3 deletions odk/odk.py
Original file line number Diff line number Diff line change
Expand Up @@ -778,7 +778,7 @@ class ExecutionContext(JsonSchemaMixin):
project : Optional[OntologyProject] = None
meta : str = ""


@dataclass
class Generator(object):
"""
Expand All @@ -787,7 +787,7 @@ class Generator(object):
"""

## TODO: consider merging Generator and ExecutionContext?
context : ExecutionContext = ExecutionContext()
context : ExecutionContext = field(default_factory=ExecutionContext)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this because of the newer dataclass version in the newer python?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes.

The other fix that the odk.py script needed was the removal of a bogus escape sequence in one of the messages it displays to the user (\@).


def generate(self, input : str) -> str:
"""
Expand Down Expand Up @@ -1083,7 +1083,7 @@ def seed(config, clean, outdir, templatedir, dependencies, title, user, source,
print(" 5. See the section under '…or push an existing repository from the command line'")
print(" E.g.:")
print("cd {}".format(outdir))
print("git remote add origin git\@github.com:{org}/{repo}.git".format(org=project.github_org, repo=project.repo))
print("git remote add origin [email protected]:{org}/{repo}.git".format(org=project.github_org, repo=project.repo))
print("git branch -M {branch}\n".format(branch=project.git_main_branch))
print("git push -u origin {branch}\n".format(branch=project.git_main_branch))
print("BE BOLD: you can always delete your repo and start again\n")
Expand Down
5 changes: 5 additions & 0 deletions update-constraints.sh
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,11 @@

set -e

if [ "x$1" = x--install-virtualenv ]; then
apt-get update
DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends python3-virtualenv
fi

virtualenv tmpdir
. tmpdir/bin/activate
python3 -m pip install -U pip
Expand Down
Loading