Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update Nextclade: Enable linux-aarch64 #46185

Merged
merged 5 commits into from
Mar 10, 2024
Merged

Conversation

corneliusroemer
Copy link
Member

@corneliusroemer corneliusroemer commented Mar 5, 2024

https://app.circleci.com/pipelines/github/bioconda/bioconda-recipes/109939/workflows/d6e1d687-5cdc-4c5d-865a-1ba0cb41b932/jobs/180788?utm_campaign=vcs-integration-link&utm_medium=referral&utm_source=github-checks-link&utm_content=summary


Please read the guidelines for Bioconda recipes before opening a pull request (PR).

General instructions

  • If this PR adds or updates a recipe, use "Add" or "Update" appropriately as the first word in its title.
  • New recipes not directly relevant to the biological sciences need to be submitted to the conda-forge channel instead of Bioconda.
  • PRs require reviews prior to being merged. Once your PR is passing tests and ready to be merged, please issue the @BiocondaBot please add label command.
  • Please post questions on Gitter or ping @bioconda/core in a comment.

Instructions for avoiding API, ABI, and CLI breakage issues

Conda is able to record and lock (a.k.a. pin) dependency versions used at build time of other recipes.
This way, one can avoid that expectations of a downstream recipe with regards to API, ABI, or CLI are violated by later changes in the recipe.
If not already present in the meta.yaml, make sure to specify run_exports (see here for the rationale and comprehensive explanation).
Add a run_exports section like this:

build:
  run_exports:
    - ...

with ... being one of:

Case run_exports statement
semantic versioning {{ pin_subpackage("myrecipe", max_pin="x") }}
semantic versioning (0.x.x) {{ pin_subpackage("myrecipe", max_pin="x.x") }}
known breakage in minor versions {{ pin_subpackage("myrecipe", max_pin="x.x") }} (in such a case, please add a note that shortly mentions your evidence for that)
known breakage in patch versions {{ pin_subpackage("myrecipe", max_pin="x.x.x") }} (in such a case, please add a note that shortly mentions your evidence for that)
calendar versioning {{ pin_subpackage("myrecipe", max_pin=None) }}

while replacing "myrecipe" with either name if a name|lower variable is defined in your recipe or with the lowercase name of the package in quotes.

Bot commands for PR management

Please use the following BiocondaBot commands:

Everyone has access to the following BiocondaBot commands, which can be given in a comment:

@BiocondaBot please update Merge the master branch into a PR.
@BiocondaBot please add label Add the please review & merge label.
@BiocondaBot please fetch artifacts Post links to CI-built packages/containers.
You can use this to test packages locally.

Note that the @BiocondaBot please merge command is now depreciated. Please just squash and merge instead.

Also, the bot watches for comments from non-members that include @bioconda/<team> and will automatically re-post them to notify the addressed <team>.

@corneliusroemer corneliusroemer enabled auto-merge (squash) March 5, 2024 15:14
@corneliusroemer
Copy link
Member Author

corneliusroemer commented Mar 5, 2024

Not sure why this happens, it's the same build script as for linux64 which passes:

15:17:25 BIOCONDA INFO (OUT) The following NEW packages will be INSTALLED:
15:17:25 BIOCONDA INFO (OUT) 
15:17:25 BIOCONDA INFO (OUT)     nextclade: 3.3.0-h23ed54d_1 local
15:17:25 BIOCONDA INFO (OUT) 
15:17:25 BIOCONDA INFO (OUT) Preparing transaction: ...working... done
15:17:27 BIOCONDA INFO (OUT) Verifying transaction: ...working... done
15:17:27 BIOCONDA INFO (OUT) Executing transaction: ...working... done
15:17:27 BIOCONDA INFO (OUT) export PREFIX=/opt/conda/conda-bld/nextclade_1709651729143/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol
15:17:27 BIOCONDA INFO (OUT) export SRC_DIR=/opt/conda/conda-bld/nextclade_1709651729143/test_tmp
15:17:28 BIOCONDA INFO (OUT) + set -e
15:17:28 BIOCONDA INFO (OUT) ++ echo 3.3.0
15:17:28 BIOCONDA INFO (OUT) ++ cut -d. -f1
15:17:28 BIOCONDA INFO (OUT) + MAJOR_VERSION=3
15:17:28 BIOCONDA INFO (OUT) + for BIN in nextclade 'nextclade$MAJOR_VERSION'
15:17:28 BIOCONDA INFO (OUT) + nextclade --version
15:17:28 BIOCONDA INFO (OUT) nextclade: relocation error: nextclade: symbol __cxa_thread_atexit_impl, version GLIBC_2.18 not defined in file libc.so.6 with link time reference
15:17:29 BIOCONDA INFO (OUT) Tests failed for nextclade-3.3.0-h23ed54d_1.tar.bz2 - moving package to /opt/conda/conda-bld/broken
15:17:29 BIOCONDA INFO (OUT) WARNING:conda_build.build:Tests failed for nextclade-3.3.0-h23ed54d_1.tar.bz2 - moving package to /opt/conda/conda-bld/broken
15:17:29 BIOCONDA INFO (OUT) TESTS FAILED: nextclade-3.3.0-h23ed54d_1.tar.bz2
15:17:33 BIOCONDA ERROR COMMAND FAILED (exited with 1): docker run -t --net host --rm -v /tmp/tmpifmzybis/build_script.bash:/opt/build_script.bash -v /opt/mambaforge/envs/bioconda/conda-bld/:/opt/host-conda-bld -v /home/circleci/project/recipes/nextclade:/opt/recipe -e LANG=C.UTF-8 -e HOST_USER_ID=1001 quay.io/bioconda/bioconda-utils-build-env-cos7-aarch64 /bin/bash /opt/build_script.bash

https://app.circleci.com/pipelines/github/bioconda/bioconda-recipes/109939/workflows/d6e1d687-5cdc-4c5d-865a-1ba0cb41b932/jobs/180788/parallel-runs/0/steps/0-107

Need to run from source? but why is the aarch glibc version lower than linux-64? see conda-forge/tensorflow-feedstock#6

@corneliusroemer
Copy link
Member Author

Skipping tests for aarch64 works - but isn't really super comforting - would be good to have a way of testing with current builder. See #46186

@rpetit3
Copy link
Member

rpetit3 commented Mar 5, 2024

@corneliusroemer is the issue here that the prebuilt binaries are built with the newer GLIBC?

@rpetit3 rpetit3 added the aarch64 Related to adding linux-aarch64 support label Mar 6, 2024
@corneliusroemer
Copy link
Member Author

Yes, Rust has raised minimum glibc version to something like 2.17 from 1.64 - and the builder has a very old glibc, see #46186

As the glibc used is the one at runtime not on the builder and all runtimes will have sufficiently recent glibc versions for nextclade to work I'd say we just bypass tests until there's a better way.

what's strange is that tests run on x86 but not on aarch64 - there might be a slight difference been builders, again see #46186

I actually have quite an urgent use case for aarch64 bioconda nextclade so it'd be great if you could approve

@corneliusroemer
Copy link
Member Author

@BiocondaBot please fetch artifacts

@BiocondaBot
Copy link
Collaborator

Azure

No artifacts found on the most recent Azure build. Either the build failed, the artifacts have been removed due to age, or the recipe was blacklisted/skipped.

@corneliusroemer
Copy link
Member Author

I've tried to work around the glibc issue but to no avail, see #46186 (comment)

Given the precedent quoted there for disabling tests when glibc causes issues (tensorflow etc.) I would like to ask for approval to skip tests just for the aarch version here.

@corneliusroemer
Copy link
Member Author

@BiocondaBot please fetch artifacts

@BiocondaBot
Copy link
Collaborator

Azure

Package(s) built on Azure are ready for inspection:

Arch Package Zip File
linux-64 nextclade-3.3.1-h9ee0642_1.tar.bz2 LinuxArtifacts

You may also use conda to install these after downloading and extracting the appropriate zip file. From the LinuxArtifacts or OSXArtifacts directories:

conda install -c ./packages <package name>

Docker image(s) built (images for Azure are in the LinuxArtifacts zip file above):

Package Tag Install with docker
nextclade 3.3.1--h9ee0642_1
showgzip -dc LinuxArtifacts/images/nextclade:3.3.1--h9ee0642_1.tar.gz | docker load

CircleCI

Package(s) built on CircleCI are ready for inspection:

Arch Package Repodata
linux-aarch64 nextclade-3.3.1-h23ed54d_1.tar.bz2 repodata.json

You may also use conda to install these:

conda install -c https://output.circle-artifacts.com/output/job/73835433-b7cc-40de-b7c0-dbe96c478fe2/artifacts/0/tmp/artifacts/packages <package name>

@corneliusroemer
Copy link
Member Author

I manually tested the aarch64 package in an aarch docker container (the one for create-env) and things work. So I think we're good to approve merge.

(base) root@e82e9469dc8e:/# file /opt/create-env/bin/nextclade
/opt/create-env/bin/nextclade: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for GNU/Linux 3.7.0, BuildID[sha1]=f774b66dc0f8781730fe9d8e02c2de25c4169bbc, stripped
(base) root@e82e9469dc8e:/# nextclade -h
Viral genome alignment, mutation calling, clade assignment, quality checks and phylogenetic placement.

Usage: nextclade [OPTIONS] <COMMAND>

Commands:
  completions      Generate shell completions
  run              Run sequence analysis: alignment, mutation calling, clade assignment, quality checks and phylogenetic placement
  dataset          List and download available Nextclade datasets (pathogens)
  sort             Sort sequences according to the inferred Nextclade dataset (pathogen)
  read-annotation  Read genome annotation and present it in Nextclade's internal formats. This is mostly only useful for Nextclade maintainers and the most curious users. Note that these internal formats have no stability guarantees and can be changed at any time without notice
  help-markdown    Print command-line reference documentation in Markdown format
  help             Print this message or the help of the given subcommand(s)

Options:
  -h, --help     Print help (see more with '--help')
  -V, --version  Print version

Verbosity:
      --verbosity <VERBOSITY>  Set verbosity level of console output [default: warn] [possible values: off, error, warn, info, debug, trace]
      --silent                 Disable all console output. Same as `--verbosity=off`
  -v, --verbose...             Make console output more verbose. Add multiple occurrences to increase verbosity further
  -q, --quiet...               Make console output more quiet. Add multiple occurrences to make output even more quiet

@corneliusroemer
Copy link
Member Author

@BiocondaBot please add label

@BiocondaBot BiocondaBot added the please review & merge set to ask for merge label Mar 9, 2024
@corneliusroemer corneliusroemer merged commit f88a164 into master Mar 10, 2024
6 checks passed
@corneliusroemer corneliusroemer deleted the nextclade-aarch branch March 10, 2024 06:13
corneliusroemer added a commit to nextstrain/nextclade that referenced this pull request Mar 11, 2024
Bioconda now publishes linux-aarch64 packages for nextclade as well, so those shas also should be updated.

bioconda/bioconda-recipes#46185
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
aarch64 Related to adding linux-aarch64 support please review & merge set to ask for merge
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants