CI #3307

Sign in to view logs

Triggered via schedule September 8, 2024 09:30

yhtang

main

Status Failure

Total duration 1d 3h 56m 31s

Artifacts 55

ci.yaml

on: schedule

Matrix: amd64 / test-distribution

Matrix: arm64 / test-distribution

amd64 / ... / build-base

arm64 / ... / build-base

amd64 / ... / build-jax

arm64 / ... / build-jax

amd64 / ... / build-upstream-pax

Matrix: amd64 / test-jax / run-unit-test

Matrix: amd64 / test-nsys-jax / run-unit-test

Matrix: amd64 / test-pallas / run-unit-test

amd64 / ... / build-upstream-t5x

amd64 / ... / launch-slurm-runner

amd64 / ... / launch-slurm-runner

amd64 / ... / launch-slurm-runner

amd64 / ... / build-gemma

amd64 / ... / build-levanter

amd64 / ... / build-maxtext

amd64 / ... / build-triton

amd64 / ... / build-equinox

arm64 / ... / build-upstream-pax

Matrix: arm64 / test-jax / run-unit-test

Waiting for pending jobs

Matrix: arm64 / test-nsys-jax / run-unit-test

Waiting for pending jobs

Matrix: arm64 / test-pallas / run-unit-test

Waiting for pending jobs

arm64 / ... / build-upstream-t5x

arm64 / ... / launch-slurm-runner

arm64 / ... / launch-slurm-runner

arm64 / ... / launch-slurm-runner

arm64 / ... / build

arm64 / ... / build-levanter

arm64 / ... / build-maxtext

arm64 / ... / build

arm64 / ... / build-equinox

amd64 / ... / build-rosetta

Matrix: amd64 / test-te / run-unit-test

amd64 / ... / launch-slurm-runner

Matrix: amd64 / test-upstream-pax / pax-multi-node

Matrix: amd64 / test-upstream-pax / single-process-evaluation

Matrix: amd64 / test-upstream-pax / single-process-multi-device

Matrix: amd64 / test-te-multigpu / te-multi-gpu

amd64 / ... / build-rosetta

Matrix: amd64 / test-upstream-t5x / t5x-multi-gpu

Matrix: amd64 / test-upstream-t5x / t5x-multi-node

Matrix: amd64 / test-gemma / run-unit-test

amd64 / ... / launch-slurm-runner

Matrix: amd64 / test-levanter / run-unit-test

amd64 / ... / launch-slurm-runner

Matrix: amd64 / test-maxtext / maxtext-multinode

Matrix: amd64 / test-maxtext / single-process-multi-device

Matrix: amd64 / test-triton / run-unit-test

amd64 / ... / launch-slurm-runner

arm64 / ... / build-rosetta

Matrix: arm64 / test-te / run-unit-test

Waiting for pending jobs

arm64 / ... / launch-slurm-runner

Matrix: arm64 / test-upstream-pax / pax-multi-node

Waiting for pending jobs

Matrix: arm64 / test-upstream-pax / single-process-evaluation

Waiting for pending jobs

Matrix: arm64 / test-upstream-pax / single-process-multi-device

Waiting for pending jobs

Matrix: arm64 / test-te-multigpu / te-multi-gpu

Waiting for pending jobs

arm64 / ... / build-rosetta

Matrix: arm64 / test-upstream-t5x / t5x-multi-gpu

Waiting for pending jobs

Matrix: arm64 / test-upstream-t5x / t5x-multi-node

Waiting for pending jobs

Matrix: arm64 / test-gemma / run-unit-test

Waiting for pending jobs

arm64 / ... / launch-slurm-runner

Matrix: arm64 / test-levanter / run-unit-test

Waiting for pending jobs

arm64 / ... / launch-slurm-runner

Matrix: arm64 / test-maxtext / maxtext-multinode

Waiting for pending jobs

Matrix: arm64 / test-maxtext / single-process-multi-device

Waiting for pending jobs

Matrix: arm64 / test-triton / run-unit-test

Waiting for pending jobs

arm64 / ... / launch-slurm-runner

Matrix: amd64 / test-rosetta-pax / rosetta-pax-multi-node-te

Matrix: amd64 / test-rosetta-pax / rosetta-pax-multi-node

Matrix: amd64 / test-rosetta-pax / rosetta-pax-single-node-dropout-te

Matrix: amd64 / test-rosetta-pax / single-process-evaluation-te

Matrix: amd64 / test-rosetta-pax / single-process-multi-device-te

amd64 / ... / test-upstream-pax-summary

amd64 / ... / test-upstream-pax-metrics

amd64 / ... / sitrep

amd64 / collect-docker-tags

Matrix: amd64 / test-rosetta-t5x / multi-gpu-multi-node

Matrix: amd64 / test-rosetta-t5x / single-process-multi-device

Matrix: amd64 / test-rosetta-t5x / vit-multi-gpu-multi-node

Matrix: amd64 / test-rosetta-t5x / vit-single-process-multi-device

amd64 / ... / test-upstream-t5x-summary

amd64 / ... / test-upstream-t5x-metrics

amd64 / ... / test-maxtext-summary

amd64 / ... / test-maxtext-metrics

Matrix: arm64 / test-rosetta-pax / rosetta-pax-multi-node-te

Waiting for pending jobs

Matrix: arm64 / test-rosetta-pax / rosetta-pax-multi-node

Waiting for pending jobs

Matrix: arm64 / test-rosetta-pax / rosetta-pax-single-node-dropout-te

Waiting for pending jobs

Matrix: arm64 / test-rosetta-pax / single-process-evaluation-te

Waiting for pending jobs

Matrix: arm64 / test-rosetta-pax / single-process-multi-device-te

Waiting for pending jobs

arm64 / ... / test-upstream-pax-summary

arm64 / ... / test-upstream-pax-metrics

arm64 / ... / sitrep

arm64 / collect-docker-tags

Matrix: arm64 / test-rosetta-t5x / multi-gpu-multi-node

Waiting for pending jobs

Matrix: arm64 / test-rosetta-t5x / single-process-multi-device

Waiting for pending jobs

Matrix: arm64 / test-rosetta-t5x / vit-multi-gpu-multi-node

Waiting for pending jobs

Matrix: arm64 / test-rosetta-t5x / vit-single-process-multi-device

Waiting for pending jobs

arm64 / ... / test-upstream-t5x-summary

arm64 / ... / test-upstream-t5x-metrics

arm64 / ... / test-maxtext-summary

arm64 / ... / test-maxtext-metrics

amd64 / ... / test-pax-rosetta-summary

amd64 / ... / test-pax-rosetta-metrics

amd64 / ... / sitrep

amd64 / ... / test-t5x-rosetta-summary

amd64 / ... / test-t5x-rosetta-metrics

amd64 / ... / sitrep

amd64 / ... / sitrep

arm64 / ... / test-pax-rosetta-summary

arm64 / ... / test-pax-rosetta-metrics

arm64 / ... / sitrep

arm64 / ... / test-t5x-rosetta-summary

arm64 / ... / test-t5x-rosetta-metrics

arm64 / ... / sitrep

arm64 / ... / sitrep

amd64 / ... / sitrep

amd64 / ... / test-upstream-pax-outcome

amd64 / ... / sitrep

amd64 / ... / test-upstream-t5x-outcome

amd64 / ... / test-maxtext-outcome

arm64 / ... / sitrep

arm64 / ... / test-upstream-pax-outcome

arm64 / ... / sitrep

arm64 / ... / test-upstream-t5x-outcome

arm64 / ... / test-maxtext-outcome

amd64 / ... / test-pax-rosetta-outcome

amd64 / ... / test-t5x-rosetta-outcome

arm64 / ... / test-pax-rosetta-outcome

arm64 / ... / test-t5x-rosetta-outcome

make-publish-configs

merge-new-manifest

Matrix: publish-containers

finalize / workflow-badge

finalize / report

finalize / upload-badge

finalize / publish-badge

Annotations

102 errors and 2 warnings

amd64 / test-nsys-jax / runner / launch-slurm-runner

Process completed with exit code 255.

amd64 / test-jax / runner / launch-slurm-runner

Process completed with exit code 255.

amd64 / test-pallas / runner / launch-slurm-runner

Process completed with exit code 255.

amd64 / test-levanter / runner / launch-slurm-runner

Process completed with exit code 255.

amd64 / test-gemma / runner / launch-slurm-runner

Process completed with exit code 255.

amd64 / test-maxtext / single-process-multi-device (1, 1, 2, 4)

Process completed with exit code 255.

amd64 / test-maxtext / maxtext-multinode (1, 1, 4, 2)

Process completed with exit code 255.

amd64 / test-maxtext / maxtext-multinode (1, 4, 2, 2)

Process completed with exit code 255.

amd64 / test-maxtext / maxtext-multinode (1, 1, 8, 1)

Process completed with exit code 255.

amd64 / test-maxtext / maxtext-multinode (1, 2, 2, 2)

Process completed with exit code 255.

amd64 / test-maxtext / maxtext-multinode (1, 1, 1, 8)

Process completed with exit code 255.

amd64 / test-maxtext / maxtext-multinode (1, 1, 1, 1)

Process completed with exit code 255.

amd64 / test-upstream-pax / pax-multi-node (1DP8FSDP1TP1PP, 1, 1, 8, 1, 4)

Process completed with exit code 255.

amd64 / test-upstream-pax / single-process-multi-device (1, 8, 1, 1)

Process completed with exit code 255.

amd64 / test-upstream-pax / pax-multi-node (16DP1FSDP1TP1PP, 1, 16, 1, 1, 4)

Process completed with exit code 255.

amd64 / test-upstream-pax / pax-multi-node (2DP1FSDP2TP4PP, 4, 2, 1, 2, 4)

Process completed with exit code 255.

amd64 / test-upstream-pax / pax-multi-node (2DP1FSDP1TP4PP, 4, 2, 1, 1, 4)

Process completed with exit code 255.

amd64 / test-upstream-pax / pax-multi-node (4DP1FSDP2TP1PP, 1, 4, 1, 2, 4)

Process completed with exit code 255.

amd64 / test-upstream-pax / pax-multi-node (8DP1FSDP1TP1PP, 1, 8, 1, 1, 4)

Process completed with exit code 255.

amd64 / test-upstream-pax / single-process-evaluation (1, 8, 1, 1)

Process completed with exit code 255.

amd64 / test-upstream-pax / pax-multi-node (1DP1FSDP1TP1PP, 1, 1, 1, 1, 4)

Process completed with exit code 255.

amd64 / test-te / runner / launch-slurm-runner

Process completed with exit code 255.

amd64 / test-te-multigpu / te-multi-gpu (4) / te-4GPU

Process completed with exit code 255.

amd64 / test-te-multigpu / te-multi-gpu (4) / te-4GPU

Process completed with exit code 255.

amd64 / test-upstream-pax / single-process-multi-device (1, 1, 2, 4)

Process completed with exit code 255.

amd64 / test-upstream-pax / pax-multi-node (LLaMA_eval, 1, 1, 8, 1, 4, true, --model-type LLaMA70BProxy --evaluate)

Process completed with exit code 255.

amd64 / test-upstream-t5x / t5x-multi-node (8G2N, 8, 2)

Process completed with exit code 255.

amd64 / test-upstream-t5x / t5x-multi-node (8G2N_fmha, 8, 2, --enable-fmha 1)

Process completed with exit code 255.

amd64 / test-te-multigpu / te-multi-gpu (2) / te-2GPU

Process completed with exit code 255.

amd64 / test-te-multigpu / te-multi-gpu (2) / te-2GPU

Process completed with exit code 255.

amd64 / test-upstream-t5x / t5x-multi-gpu (1P8G, 8)

Process completed with exit code 255.

amd64 / test-te-multigpu / te-multi-gpu (8) / te-8GPU

Process completed with exit code 255.

amd64 / test-te-multigpu / te-multi-gpu (8) / te-8GPU

Process completed with exit code 255.

amd64 / test-maxtext / test-maxtext-sitrep / sitrep

Process completed with exit code 2.

amd64 / test-upstream-t5x / t5x-multi-gpu (1P2G_fmha, 2, --enable-fmha 1)

Process completed with exit code 255.

amd64 / test-upstream-pax / test-upstream-pax-sitrep / sitrep

Process completed with exit code 2.

amd64 / test-maxtext / test-maxtext-outcome

Process completed with exit code 2.

amd64 / test-upstream-t5x / test-upstream-t5x-sitrep / sitrep

Process completed with exit code 2.

amd64 / test-upstream-pax / test-upstream-pax-outcome

Process completed with exit code 2.

amd64 / test-upstream-t5x / test-upstream-t5x-outcome

Process completed with exit code 2.

amd64 / test-triton / runner / launch-slurm-runner

Process completed with exit code 255.

amd64 / test-rosetta-pax / rosetta-pax-multi-node (4, 2, 1, 2)

Process completed with exit code 255.

amd64 / test-rosetta-pax / rosetta-pax-multi-node (4, 2, 1, 2)

Process completed with exit code 1.

amd64 / test-rosetta-pax / rosetta-pax-multi-node (1, 4, 1, 2)

Process completed with exit code 255.

amd64 / test-rosetta-pax / rosetta-pax-multi-node (1, 4, 1, 2)

Process completed with exit code 1.

amd64 / test-rosetta-pax / rosetta-pax-multi-node-te (LLaMA_eval_TE, 1, 1, 8, 1, 4, true, --model-type LLaMA70BProxy --evalu...

Process completed with exit code 1.

amd64 / test-rosetta-pax / rosetta-pax-multi-node-te (LLaMA_eval_TE, 1, 1, 8, 1, 4, true, --model-type LLaMA70BProxy --evalu...

Process completed with exit code 255.

amd64 / test-rosetta-pax / rosetta-pax-multi-node-te (1DP1FSDP1TP1PP_TE, 1, 1, 1, 1, 4)

Process completed with exit code 255.

amd64 / test-rosetta-pax / rosetta-pax-multi-node-te (1DP1FSDP1TP1PP_TE, 1, 1, 1, 1, 4)

Process completed with exit code 1.

amd64 / test-rosetta-pax / single-process-multi-device-te (1, 1, 2, 4)

Process completed with exit code 1.

amd64 / test-rosetta-pax / single-process-multi-device-te (1, 1, 2, 4)

Process completed with exit code 255.

amd64 / test-rosetta-pax / rosetta-pax-multi-node-te (4DP1FSDP2TP1PP_TE, 1, 4, 1, 2, 4)

Process completed with exit code 255.

amd64 / test-rosetta-pax / rosetta-pax-multi-node-te (4DP1FSDP2TP1PP_TE, 1, 4, 1, 2, 4)

Process completed with exit code 1.

amd64 / test-rosetta-pax / single-process-evaluation-te (1, 8, 1, 1)

Process completed with exit code 1.

amd64 / test-rosetta-pax / single-process-evaluation-te (1, 8, 1, 1)

Process completed with exit code 255.

amd64 / test-rosetta-pax / rosetta-pax-multi-node-te (5B_fused_attn_0, 1, 1, 8, 1, 2, --model-type 5B --disable-fused-attn)

Process completed with exit code 255.

amd64 / test-rosetta-pax / rosetta-pax-multi-node-te (5B_fused_attn_0, 1, 1, 8, 1, 2, --model-type 5B --disable-fused-attn)

Process completed with exit code 1.

amd64 / test-rosetta-pax / rosetta-pax-multi-node-te (1DP8FSDP1TP1PP_TE, 1, 1, 8, 1, 4)

Process completed with exit code 255.

amd64 / test-rosetta-pax / rosetta-pax-multi-node-te (1DP8FSDP1TP1PP_TE, 1, 1, 8, 1, 4)

Process completed with exit code 1.

amd64 / test-rosetta-pax / rosetta-pax-multi-node-te (16DP1FSDP1TP1PP_TE, 1, 16, 1, 1, 4)

Process completed with exit code 1.

amd64 / test-rosetta-pax / rosetta-pax-multi-node-te (16DP1FSDP1TP1PP_TE, 1, 16, 1, 1, 4)

Process completed with exit code 255.

amd64 / test-rosetta-pax / rosetta-pax-multi-node (4, 2, 1, 1)

Process completed with exit code 255.

amd64 / test-rosetta-pax / rosetta-pax-multi-node (4, 2, 1, 1)

Process completed with exit code 1.

amd64 / test-rosetta-pax / rosetta-pax-multi-node (1, 8, 1, 1)

Process completed with exit code 1.

amd64 / test-rosetta-pax / rosetta-pax-multi-node (1, 8, 1, 1)

Process completed with exit code 255.

amd64 / test-rosetta-pax / rosetta-pax-multi-node-te (8DP1FSDP1TP1PP_TE, 1, 8, 1, 1, 4)

Process completed with exit code 255.

amd64 / test-rosetta-pax / rosetta-pax-multi-node-te (8DP1FSDP1TP1PP_TE, 1, 8, 1, 1, 4)

Process completed with exit code 1.

amd64 / test-rosetta-pax / single-process-multi-device-te (1, 8, 1, 1)

Process completed with exit code 255.

amd64 / test-rosetta-pax / single-process-multi-device-te (1, 8, 1, 1)

Process completed with exit code 1.

amd64 / test-rosetta-pax / rosetta-pax-single-node-dropout-te (1, 8, 1, 1)

Process completed with exit code 1.

amd64 / test-rosetta-pax / rosetta-pax-single-node-dropout-te (1, 8, 1, 1)

Process completed with exit code 255.

amd64 / test-rosetta-pax / rosetta-pax-multi-node-te (5B_fused_attn_1, 1, 1, 8, 1, 2, --model-type 5B)

Process completed with exit code 1.

amd64 / test-rosetta-pax / rosetta-pax-multi-node-te (5B_fused_attn_1, 1, 1, 8, 1, 2, --model-type 5B)

Process completed with exit code 255.

amd64 / test-rosetta-pax / test-pax-rosetta-sitrep / sitrep

Process completed with exit code 2.

amd64 / test-rosetta-pax / test-pax-rosetta-outcome

Process completed with exit code 2.

amd64 / test-rosetta-t5x / vit-single-process-multi-device (8)

Process completed with exit code 1.

amd64 / test-rosetta-t5x / vit-single-process-multi-device (8)

Process completed with exit code 255.

amd64 / test-rosetta-t5x / multi-gpu-multi-node (2N2G_te-0, 2, 2, --enable-te 0)

Process completed with exit code 255.

amd64 / test-rosetta-t5x / multi-gpu-multi-node (2N2G_te-0, 2, 2, --enable-te 0)

Process completed with exit code 1.

amd64 / test-rosetta-t5x / single-process-multi-device (1P1G_te-0, 1, --enable-te 0)

Process completed with exit code 255.

amd64 / test-rosetta-t5x / single-process-multi-device (1P1G_te-0, 1, --enable-te 0)

Process completed with exit code 1.

amd64 / test-rosetta-t5x / vit-multi-gpu-multi-node (8, 2)

Process completed with exit code 255.

amd64 / test-rosetta-t5x / vit-multi-gpu-multi-node (1, 2)

Process completed with exit code 1.

amd64 / test-rosetta-t5x / vit-multi-gpu-multi-node (8, 2)

Process completed with exit code 1.

amd64 / test-rosetta-t5x / vit-multi-gpu-multi-node (1, 2)

Process completed with exit code 255.

amd64 / test-rosetta-t5x / single-process-multi-device (1P8G_te-1, 8, --gin.train/utils.DatasetConfig.pack=False --gin.train...

Process completed with exit code 1.

amd64 / test-rosetta-t5x / single-process-multi-device (1P8G_te-1, 8, --gin.train/utils.DatasetConfig.pack=False --gin.train...

Process completed with exit code 255.

amd64 / test-rosetta-t5x / multi-gpu-multi-node (2N8G-te-1, 8, 2, --gin.train/utils.DatasetConfig.pack=False --gin.train_eva...

Process completed with exit code 255.

amd64 / test-rosetta-t5x / multi-gpu-multi-node (2N8G-te-1, 8, 2, --gin.train/utils.DatasetConfig.pack=False --gin.train_eva...

Process completed with exit code 1.

amd64 / test-rosetta-t5x / vit-multi-gpu-multi-node (8, 1)

Process completed with exit code 255.

amd64 / test-rosetta-t5x / vit-multi-gpu-multi-node (8, 1)

Process completed with exit code 1.

amd64 / test-rosetta-t5x / vit-multi-gpu-multi-node (1, 1)

Process completed with exit code 1.

amd64 / test-rosetta-t5x / vit-multi-gpu-multi-node (1, 1)

Process completed with exit code 255.

amd64 / test-rosetta-t5x / test-t5x-rosetta-sitrep / sitrep

Process completed with exit code 2.

amd64 / test-rosetta-t5x / test-t5x-rosetta-outcome

Process completed with exit code 2.

amd64 / test-jax / jax-A100-unit-test

This request was automatically failed because there were no enabled runners online to process the request for more than 1 days.

amd64 / test-pallas / pallas-A100-unit-test

This request was automatically failed because there were no enabled runners online to process the request for more than 1 days.

amd64 / test-levanter / levanter-A100-unit-test

This request was automatically failed because there were no enabled runners online to process the request for more than 1 days.

amd64 / test-gemma / gemma-A100-unit-test

This request was automatically failed because there were no enabled runners online to process the request for more than 1 days.

amd64 / test-nsys-jax / nsys-jax-A100-unit-test

This request was automatically failed because there were no enabled runners online to process the request for more than 1 days.

amd64 / test-te / te-A100-unit-test

This request was automatically failed because there were no enabled runners online to process the request for more than 1 days.

amd64 / test-triton / triton-A100-unit-test

This request was automatically failed because there were no enabled runners online to process the request for more than 1 days.

merge-new-manifest

Unexpected input(s) 'owner_and_repo', valid inputs are ['route', 'mediaType']

merge-new-manifest

Unexpected input(s) 'owner_and_repo', 'head', 'base', 'body', 'title', 'draft', valid inputs are ['route', 'mediaType']

Artifacts

Produced during runtime

Name	Size
artifact-base-build-amd64	567 Bytes
artifact-base-build-arm64	566 Bytes
artifact-equinox-build-amd64	569 Bytes
artifact-equinox-build-arm64	567 Bytes
artifact-final-report	3.09 KB
artifact-gemma-build-amd64	559 Bytes
artifact-jax-build-amd64	553 Bytes
artifact-jax-build-arm64	554 Bytes
artifact-levanter-build-amd64	572 Bytes
artifact-levanter-build-arm64	571 Bytes
artifact-maxtext-build-amd64	568 Bytes
artifact-maxtext-build-arm64	569 Bytes
artifact-pax-build-amd64	567 Bytes
artifact-pax-build-arm64	567 Bytes
artifact-rosetta-build-pax-amd64	584 Bytes
artifact-rosetta-build-pax-arm64	583 Bytes
artifact-rosetta-build-t5x-amd64	585 Bytes
artifact-rosetta-build-t5x-arm64	584 Bytes
artifact-t5x-build-amd64	571 Bytes
artifact-t5x-build-arm64	567 Bytes
artifact-triton-build-amd64	564 Bytes
artifact-workflow-metadata	277 Bytes
bumped-manifest	71.2 KB
final-base	249 Bytes
final-equinox	258 Bytes
final-gemma	252 Bytes
final-jax	246 Bytes
final-levanter	261 Bytes
final-maxtext	258 Bytes
final-pax	246 Bytes
final-t5x	246 Bytes
final-triton	255 Bytes
final-upstream-pax	273 Bytes
final-upstream-t5x	273 Bytes
gemma-unit-test-V100	2.91 KB
jax-unit-test-V100	17.7 KB
levanter-unit-test-V100	13.9 KB
mealkit-equinox	269 Bytes
mealkit-gemma	263 Bytes
mealkit-jax	256 Bytes
mealkit-levanter	272 Bytes
mealkit-maxtext	268 Bytes
mealkit-pax	257 Bytes
mealkit-t5x	258 Bytes
mealkit-triton	266 Bytes
mealkit-upstream-pax	282 Bytes
mealkit-upstream-t5x	282 Bytes
nsys-jax-unit-test-V100	16.3 MB
pallas-unit-test-V100	7.35 KB
te-2GPU	199 Bytes
te-4GPU	199 Bytes
te-8GPU	199 Bytes
te-multigpu-test	519 Bytes
te-unit-test-V100	375 KB
triton-unit-test-V100	4.95 KB