[CI] macOS Integ Test Flakiness

[dbwiddis]

Is your feature request related to a problem?

Integ tests are flaky on macOS. Given that failures have primarily been resource-constrained, and that macOS runners are really VMs and tend to ve more limiting, this is not a surprise.

What solution would you like?

Pick one:

• Include longer timeouts that reduce/eliminate issues with macOS tests. We're already wating 4-6 minutes for tests to complete, adding an extra minute to avoid a "retry" that adds another 4-6 minutes is a win.
• Use JNA Platform class to test whether we're on macOS and add longer time for that platform
• Use a bash script in the github action to automatically retry N times. See this example from a project I maintain. This (with 2 tries) is consistent with OpenSearch Jenkins retries.

What alternatives have you considered?

Investigating whether macOS is officially supported and if not, using GitHub's allow_failure: true on macOS tests. I don't like this option but it's the easiest.

Do you have any additional context?

macOS is a fantastic operating system but it is specifically designed to run on Apple hardware and not in a VM.

[owaiskazi19]

How about doing something similar to core opensearch-project/OpenSearch#2547? We can use the https://blog.gradle.org/gradle-flaky-test-retry-plugin plugin to perform the retry.

[kgantchev]

Investigating whether macOS is officially supported and if not, using GitHub's allow_failure: true on macOS tests. I don't like this option but it's the easiest.

I apologize if I'm intruding here, but if you're considering MacOS runners, may I suggest giving FlyCI a try? We offer MacOS M1 and M2 runners that are faster than GitHub's. For public repos, we offer 500 mins/month of free M1 usage (4 vCPUs, 7 GB RAM, 28 GB storage).

The setup is super easy:

1. Install the FlyCI GitHub app.
2. Give the FlyCI app permissions to this repo.
3. Change your runs-on flag whenever you implement the ARM64 MacOS workflow:

jobs:
ci:
-    runs-on: macos-latest
+    runs-on: flyci-macos-large-latest-m1
      steps:
      - name: 👀 Checkout repo
         uses: actions/checkout@v4

Do you think this might be a good option for flow-framework?

Web: flyci.net | Twitter: flyciapp | LinkedIn: FlyCI | Discord: FlyCI

[dbwiddis]

Hey @kgantchev thanks for the idea. I've actually taken advantage of these runners on other projects I maintain.

All OpenSearch project repo settings are centrally managed so getting this app approved has to go through these organization admins. We'd need to come up with a justification for the need to do this testing to propose adding it, but I am not sure it's really necessary here (and given the scale of this organization's projects, I doubt we'd fit under your free offering.)

our primary user base uses cloud servers on hardware optimized for that purpose (flexible storage and memory choices, no need for displays), while Apple Silicon is primarily optimized for the end user market (integrated on-chip memory and GPU). So our investments in testing hardware/software need to focus on the overwhelming majority of users.

[kgantchev]

I've actually taken advantage of these runners on other projects I maintain.

Awesome! I'm glad it has come in handy.

...
our primary user base uses cloud servers on hardware optimized for that purpose (flexible storage and memory choices, no need for displays), while Apple Silicon is primarily optimized for the end user market (integrated on-chip memory and GPU). So our investments in testing hardware/software need to focus on the overwhelming majority of users.

Got it. Thanks for the feedback. I guess I missed the full context here. :) I appreciate it.

[dbwiddis]

Reopening this issue as the recent PRs did not fix the node drop issue (but did give much less confusing logs)

[dbwiddis]

Closing this as it's no longer macOS central, opening a new issue for multi-node.