[MNT] MPS backend test failures on MacOS

[fkiraly]

The CI fails with MPS backend failures on a number of tests:

RuntimeError: MPS backend out of memory (MPS allocated: 0 bytes, other allocations: 0 bytes, max allowed: 7.93 GB). Tried to allocate 256 bytes on shared pool. Use PYTORCH_MPS_HIGH_WATERMARK_RATIO=0.0 to disable upper limit for memory allocations (may cause system failure).
[W 2024-08-22 20:58:07,168] Trial 0 failed with value None.

[fkiraly]

Update: these seem to happen only on Mac

[fkiraly]

happens on macos-latest but not on macos-13 - a temporary fix that @XinyuWuu discovered is to downgrade mac runners to macos-13

[fkiraly]

The failure on macos-latest is here: #1633

[fnhirwa]

This is an issue related to device handling on GPU for macos. Will open a PR fixing this generally.

[benHeid]

MPS errors can only happen to macOS since they are the shaders of the m Chips as @fnhirwa said.

This is probably caused by large neural networks that might perhaps run in parallel on the shaders. Fix could be to set the device to cpu for all tests (bottleneck now will become the normal ram) or alternatively reduce the ram.

A more complicated solution could also be to check if we can control the parallel execution of tests so that no neural networks are run in parallel to others but only in parallel with simpler models.

[fnhirwa]

Given that this is a resource issue PyTorch helps to set an environment variable that falls back to CPU via PYTORCH_ENABLE_MPS_FALLBACK=1 when there is overutilization of the mps device. We can use monkeypatch fixture to set this variable in the tests.

I am adding the changes to #1648 to see if it works.

[XinyuWuu]

A more complicated solution could also be to check if we can control the parallel execution of tests so that no neural networks are run in parallel to others but only in parallel with simpler models.

We can do it by using a filelock as a fixture. I have tried it in sktime/sktime#6774.

[XinyuWuu]

It's caused by lack of nested-virtualization support:
https://docs.github.com/en/actions/using-github-hosted-runners/using-github-hosted-runners/about-github-hosted-runners#limitations-for-arm64-macos-runners
actions/runner-images#9254 (comment)
actions/runner-images#9918

My tests:
https://github.com/jdb78/pytorch-forecasting/actions/runs/10714861256/job/29709292560?pr=1654
https://github.com/jdb78/pytorch-forecasting/actions/runs/10714737635/job/29708945795?pr=1654

torch.backends.mps.is_available() returns true in macos-latest but it should return false.

@fnhirwa I am afraid PYTORCH_ENABLE_MPS_FALLBACK won't help. It enables fallback for some operators such as aten::_slow_conv2d_forward but in our case MPS is totally unusable.

We need to find a way to make torch.backends.mps.is_available() return false in macos-latest.

Unfortunately, we do not have something like CUDA_VISIBLE_DEVICES for MPS.