Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Proposal] Support AL2023/NodeJS18 for OpenSearch/Dashboards Releases #1563

Closed
Tracked by #4058
peterzhuamazon opened this issue Jan 27, 2022 · 33 comments
Closed
Tracked by #4058
Assignees
Labels

Comments

@peterzhuamazon
Copy link
Member

peterzhuamazon commented Jan 27, 2022

With the announcement of Amazon Linux 2023 Preview available for testing, we want to setup some notes here for the support of AL2023 once it is fully released to public.
https://aws.amazon.com/about-aws/whats-new/2021/11/preview-amazon-linux-2022/
https://aws.amazon.com/about-aws/whats-new/2023/03/amazon-linux-2023/

  • Evaluate the compatibility of AL2023 when running OpenSearch/Dashboards
    • OpenSearch/Dashboards latest binary can run on AL2023 with the same expected behaviors on AL2
    • Working alongside plugin teams to ensure all the plugins are running as expected
    • Performance testing to ensure there is no performance regression
    • Evaluate the package/dependencies availability on AL2023 default repository
    • Evaluate the size of the image vs contents integrity
  • Evaluate AL2023 availability
    • AL2023 available on docker for both x64 and arm64
    • AL2023 available on EC2 for both x64 and arm64 for testing
  • Evaluate Build Process
    • What level of changes do we need to support AL2023 with current build process and scripts
    • Can we support both AL2 and AL2023 in the same system
    • Build tool related libs/packages/dependencies availability on AL2023
  • Evaluate deprecation plan for older OS
    • When would we deprecate AL2 while supporting AL2023
    • Will this introduce breaking changes to customer
    • Would we support older version of OpenSearch/Dashboards with new OS?

We welcome more discussion on this.

Thanks.

@peterzhuamazon peterzhuamazon added the proposal Proposal and RFC to the community label Jan 27, 2022
@bbarani bbarani added the docker label Feb 1, 2022
@bbarani bbarani changed the title [Proposal] Support AL2022 for OpenSearch/Dashboards Docker Release [Proposal] Support AL2023 for OpenSearch/Dashboards Docker Releas May 1, 2023
@bbarani bbarani added the v3.0.0 label May 1, 2023
@bbarani bbarani changed the title [Proposal] Support AL2023 for OpenSearch/Dashboards Docker Releas [Proposal] Support AL2023 for OpenSearch/Dashboards Docker Release May 1, 2023
@peterzhuamazon
Copy link
Member Author

Hi @nknize @dblock what do you think about this change?

From our teams perspective it makes sense to upgrade the main docker image from AL2 to AL2023 starting 3.0.0. It also make sense that we use this opportunities to upgrade other OS such as Ubuntu and Rockylinux to their latest versions.

Thanks.

@dblock
Copy link
Member

dblock commented May 2, 2023

I really don't know enough about this @peterzhuamazon. What are pros/cons?

@peterzhuamazon
Copy link
Member Author

peterzhuamazon commented May 16, 2023

Now, with this issue on the horizon, we need to consider how to proceed with the changes.

This implies:

  • We need to deprecate the ci image that utilizes centos7 or al2.
  • Our ci and release docker images need to transition from centos/al2 to al2023.
  • Migrate build repo code from using Python 3.7 to 3.9 version #3351
    • The Python upgrade from 3.7 to 3.9 need to happen.
    • AL2 has python 3.7 as the default, but AL2023 already includes 3.9. Attempting to compile Python 3.7 on AL2023 would consume significant time.
    • Python 3.7 support will reach its eol in 1 month.
  • If we continue using the same ci image for 1.x / 2.x / 3.x, KNN will crash for certain users with older operating systems (older glibc than the one on al2023), as centos7 represents the best compatibility for KNN, having the oldest glibc version on the OS, that does not support Node18.
    • Use containers for library artifact build opendistro-for-elasticsearch/k-NN#169
    • Unless we create a new jenkinsfile specifically for 1.x, we cannot extensively modify the existing dist-build-dashboards jenkinsfile, as it is already close to the Jenkinsfile max size limit.
    • Introducing a new jenkinsfile solely for 1.x will cause issues due to many hardcoded dir names as well as structures on S3.
    • One possible approach to avoid this issue is to use centos7 for building OS and use al2023 for building OSD, although this approach lacks standardization and is hard to maintain.
  • This change need to occur after the completion of the Jenkins Upgrade.
  • This is a significant breaking change, and it seems that we want to backport it to 2.x due to the deprecation of node14.

@bbarani
Copy link
Member

bbarani commented May 16, 2023

Tagging @vamshin @seanneumann @AMoo-Miki @ananzh @jmazanec15 for visibility.

@seanneumann
Copy link
Contributor

For the Node 18 upgrade, we're targeting 2.8. This is important as Node 14 no longer will receive security updates (as of April 2023). Is there an ask to delay to solve the above?

@peterzhuamazon
Copy link
Member Author

peterzhuamazon commented May 19, 2023

KNN support on the native code compilation:

node: /lib64/libm.so.6: version `GLIBC_2.27' not found (required by node)
node: /lib64/libc.so.6: version `GLIBC_2.25' not found (required by node)
node: /lib64/libc.so.6: version `GLIBC_2.28' not found (required by node)
node: /lib64/libstdc++.so.6: version `CXXABI_1.3.9' not found (required by node)
node: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by node)
node: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.21' not found (required by node)
# CentOS7
ldd (GNU libc) 2.17

# AL2
ldd (GNU libc) 2.26

# CentOS8/RockyLinux8
ldd (GNU libc) 2.28

# AL2023
ldd (GNU libc) 2.34

Seems like if we want to switch from CentOS7 to CentOS8/RockyLinux8, we will need to remove support of CentOS7 and AL2 from our support matrix on our website as well, since CentOS8/RockyLinux8 is the baseline of node18 with glibc 2.28.

And if we compile knn on 2.28 then the lib will crash on glibc that is lower than 2.28.

cc: @krisfreedain

@bbarani
Copy link
Member

bbarani commented May 19, 2023

@vamshin @jmazanec15 Can you please provide your inputs as well?

@krisfreedain
Copy link
Member

we will need to remove support of CentOS7 and AL2 from our support matrix on our website as well, since CentOS8/RockyLinux8 is the baseline of node18 with glibc 2.28

I would agree @peterzhuamazon - if we test on CentOS8/RockyLinux8 the compatibility matrix needs to be updated to accurately reflect that for the community

@ananzh
Copy link
Member

ananzh commented May 19, 2023

Hi @peterzhuamazon we will bump the node.js to version 18.16.0

@peterzhuamazon peterzhuamazon changed the title [Proposal] Support AL2023 for OpenSearch/Dashboards Docker Release [Proposal] Support AL2023/NodeJS18 for OpenSearch/Dashboards Releases May 19, 2023
@peterzhuamazon peterzhuamazon self-assigned this May 19, 2023
@peterzhuamazon peterzhuamazon pinned this issue May 22, 2023
@jmazanec15
Copy link
Member

@peterzhuamazon want to clarify, you are saying that node 18 requires 2.28, correct? Then doesn't this break compatibility with AL2 and centos 7? So, regardless of k-NN libs, upgrading to Node 18 would require us to remove centos 7 and AL2 from the compatibility matrix.

If this is not the case, we could also elect to compile k-NN libs in separate docker image to maintain compatibility.

@joshuali925
Copy link
Member

@peterzhuamazon not sure if you saw this opensearch-project/OpenSearch-Dashboards#3601 (comment)

unlike chromium, it is possible to do a custom build of nodejs on centos 7 to lower glibc dependencies (I also saw musl builds somewhere, which can be another possibility to workaround this issue). I've not looked into performance or other issues might be caused by downgrading glibc, but if OS upgrade is difficult maybe this would be easier?

@anirudha
Copy link
Contributor

Did anyone confirm node 18 with AL2? @seanneumann

We noticed some glibc related compatibility issues. Not sure if it's an isolated incident.

@peterzhuamazon
Copy link
Member Author

@peterzhuamazon not sure if you saw this opensearch-project/OpenSearch-Dashboards#3601 (comment)

unlike chromium, it is possible to do a custom build of nodejs on centos 7 to lower glibc dependencies (I also saw musl builds somewhere, which can be another possibility to workaround this issue). I've not looked into performance or other issues might be caused by downgrading glibc, but if OS upgrade is difficult maybe this would be easier?

That is likely not an option because CentOS7 is going to be out of support in 1 year, and we are also on track to move to al2023 anyway. Manually rebuild nodejs18 would add another dep on our side, which is not going to scale on the long run.

Thanks.

@peterzhuamazon
Copy link
Member Author

peterzhuamazon commented Jul 10, 2023

The cmake upgrade to 3.23.3 version in #3706 was initially setup as a follow up in the al2023 upgrade/nodejs upgrade, tho now it seems like a 2.9.0 dependencies thus it needs to be updated soon.

Thanks.

@peterzhuamazon
Copy link
Member Author

peterzhuamazon commented Aug 1, 2023

AL2023/NodeJS18 on 2.10.0 Release:

Required Steps

Optional Steps

  • Need to discuss after the 2.10.0 release whether or not 1.x line need to upgrade as well.
    • 1.x will keep using the old centos7 docker images as I have set measures to keep it that way
    • 20230912: We will keep 2.x on CentOS7 for OpenSearch on to b/c k-NN before fully deprecating old OSes with glibc<2.27/2.28 later
  • Have separate pipeline to test NodeJS14 bc abilities.
    • No need, one pipeline is enough for that.
  • Updating

@wbeckler
Copy link

What is the status of the separate pipeline to test node 14 backward compatibility? There are going to be users using old linux versions who can't go past node 14 yet, and it would be useful to know what fixes we can do so as to not unnecessarily break things for those users.

@bbarani
Copy link
Member

bbarani commented Aug 29, 2023

@wbeckler We do not have plans to create a separate pipeline in public Jenkins infra for EOL versions of Node. If needed, this needs to be tackled at repo level.

@peterzhuamazon
Copy link
Member Author

Have quick talk with OSD @AMoo-Miki about core, and we are planning to switch 2.x to 18.16.0 next week to start testing on the 2.10.0 build with newer nodejs.

Thanks.

kavilla pushed a commit to opensearch-project/OpenSearch-Dashboards that referenced this issue Sep 6, 2023
….16.0 version and update changelog of 2.9.0/Unreleased Sections (#4948)

* Bump 2.10.0 to use nodejs 18.16.0 version
* Update 2.9.0 and 2.x changelog for unreleased 2.10.0 version

Issues resolved:
opensearch-project/opensearch-build#1563
opensearch-project/opensearch-build#3743

Signed-off-by: Peter Zhu <[email protected]>
@peterzhuamazon
Copy link
Member Author

The majority of the tasks in this issue is resolved and implemented.
We will issue new tickets whenever new tasks coming up.

Thanks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
Development

No branches or pull requests

12 participants