Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Flake] Sanity tests #49

Open
rzetelskik opened this issue May 9, 2024 · 16 comments
Open

[Flake] Sanity tests #49

rzetelskik opened this issue May 9, 2024 · 16 comments
Assignees
@zimnx
Labels
kind/flake Categorizes issue or PR as related to a flaky test. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release.

Comments

@rzetelskik
Copy link
Member

rzetelskik commented May 9, 2024 •
edited
Loading

Sanity tests are flaky on current master (ba394ac).

To reproduce:

$  go run ./vendor/github.com/onsi/ginkgo/v2/ginkgo --until-it-fails -race ./test/sanity/...
...
Running Suite: Sanity Suite - /home/rzetelskik/github.com/scylladb/k8s-local-volume-provisioner/test/sanity/set/localdriver
===========================================================================================================================
Random Seed: 1715243119

Will run 83 of 84 specs
••SS
------------------------------
• [FAILED] [60.080 seconds]
Local CSI Driver CSI sanity Controller Service [Controller Server] [BeforeEach] ListVolumes check the presence of new volumes and absence of deleted ones in the volume list
  [BeforeEach] /home/rzetelskik/github.com/scylladb/k8s-local-volume-provisioner/vendor/github.com/kubernetes-csi/csi-test/v5/pkg/sanity/tests.go:46
  [It] /home/rzetelskik/github.com/scylladb/k8s-local-volume-provisioner/vendor/github.com/kubernetes-csi/csi-test/v5/pkg/sanity/controller.go:209

  Timeline >>
  STEP: connecting to CSI driver @ 05/09/24 10:25:19.402
  [FAILED] in [BeforeEach] - /home/rzetelskik/github.com/scylladb/k8s-local-volume-provisioner/vendor/github.com/kubernetes-csi/csi-test/v5/pkg/sanity/sanity.go:265 @ 05/09/24 10:26:19.458
  << Timeline

  [FAILED] Unexpected error:
      <*errors.errorString | 0xc0004ded00>:
      Connection timed out
      {
          s: "Connection timed out",
      }
  occurred
  In [BeforeEach] at: /home/rzetelskik/github.com/scylladb/k8s-local-volume-provisioner/vendor/github.com/kubernetes-csi/csi-test/v5/pkg/sanity/sanity.go:265 @ 05/09/24 10:26:19.458
------------------------------
P [PENDING]
Local CSI Driver CSI sanity Controller Service [Controller Server] ListVolumes pagination should detect volumes added between pages and accept tokens when the last volume from a page is deleted
/home/rzetelskik/github.com/scylladb/k8s-local-volume-provisioner/vendor/github.com/kubernetes-csi/csi-test/v5/pkg/sanity/controller.go:268
------------------------------
•••••••SSSS•••••••SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS•••••••••••SSSSSSSSSSSSS••

Summarizing 1 Failure:
  [FAIL] Local CSI Driver CSI sanity Controller Service [Controller Server] [BeforeEach] ListVolumes check the presence of new volumes and absence of deleted ones in the volume list
  /home/rzetelskik/github.com/scylladb/k8s-local-volume-provisioner/vendor/github.com/kubernetes-csi/csi-test/v5/pkg/sanity/sanity.go:265

Ran 30 of 84 Specs in 61.337 seconds
FAIL! -- 29 Passed | 1 Failed | 1 Pending | 53 Skipped
--- FAIL: TestSanity (61.37s)
FAIL

Tests failed on attempt #3


Ginkgo ran 1 suite in 1m7.147438575s

Test Suite Failed
exit status 1

From what I've seen different tests are failing on connection timeout - it's not a specific test.

/kind flake
@scylla-operator-bot scylla-operator-bot bot added kind/flake Categorizes issue or PR as related to a flaky test. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels May 9, 2024
@rzetelskik rzetelskik mentioned this issue May 9, 2024
Merged
@tnozicka tnozicka added the priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete. label May 9, 2024
@scylla-operator-bot scylla-operator-bot bot removed the needs-priority Indicates a PR lacks a `priority/foo` label and requires one. label May 9, 2024
@zimnx
Copy link
Collaborator

zimnx commented May 14, 2024

https://prow.scylla-operator.scylladb.com/view/gs/scylla-operator-prow/pr-logs/pull/scylladb_k8s-local-volume-provisioner/50/pull-k8s-local-volume-provisioner-master-sanity/1790354916707930112

@zimnx zimnx mentioned this issue May 14, 2024
Merged
@scylla-operator-bot scylla-operator-bot
Copy link
Contributor

The Scylla Operator project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

  • After 30d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue as fresh with /remove-lifecycle stale
  • Close this issue with /close
  • Offer to help out

/lifecycle stale

@scylla-operator-bot scylla-operator-bot bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jul 9, 2024
@tnozicka
Copy link
Member

https://prow.scylla-operator.scylladb.com/view/gs/scylla-operator-prow/pr-logs/pull/scylladb_local-csi-driver/56/pull-local-csi-driver-master-sanity/1812867410189881344

@tnozicka tnozicka mentioned this issue Jul 16, 2024
Merged
@tnozicka
Copy link
Member

/remove-lifecycle stale

@scylla-operator-bot scylla-operator-bot bot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jul 16, 2024
@tnozicka
Copy link
Member

https://prow.scylla-operator.scylladb.com/view/gs/scylla-operator-prow/pr-logs/pull/scylladb_local-csi-driver/58/pull-local-csi-driver-v0.4-sanity/1813520994044743680#1:build-log.txt%3A23

@tnozicka tnozicka mentioned this issue Jul 17, 2024
Merged
1 task
@tnozicka
Copy link
Member

https://prow.scylla-operator.scylladb.com/view/gs/scylla-operator-prow/pr-logs/pull/scylladb_local-csi-driver/57/pull-local-csi-driver-master-sanity/1813521778492837888#1:build-log.txt%3A23

@tnozicka tnozicka mentioned this issue Jul 17, 2024
Merged
@tnozicka tnozicka added priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. and removed priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete. labels Jul 17, 2024
@tnozicka
Copy link
Member

raising the priority, this seems to be flaking in more than 50% of cases

@tnozicka
Copy link
Member

https://prow.scylla-operator.scylladb.com/view/gs/scylla-operator-prow/pr-logs/pull/scylladb_local-csi-driver/59/pull-local-csi-driver-v0.4-sanity/1813843822224871424#1:build-log.txt%3A23

@tnozicka
Copy link
Member

https://prow.scylla-operator.scylladb.com/view/gs/scylla-operator-prow/pr-logs/pull/scylladb_local-csi-driver/59/pull-local-csi-driver-v0.4-sanity/1813846717901049856#1:build-log.txt%3A22

@scylla-operator-bot scylla-operator-bot
Copy link
Contributor

The Scylla Operator project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

  • After 30d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue as fresh with /remove-lifecycle stale
  • Close this issue with /close
  • Offer to help out

/lifecycle stale

@scylla-operator-bot scylla-operator-bot bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Aug 17, 2024
@scylla-operator-bot scylla-operator-bot
Copy link
Contributor

The Scylla Operator project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

  • After 30d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue as fresh with /remove-lifecycle rotten
  • Close this issue with /close
  • Offer to help out

/lifecycle rotten

@scylla-operator-bot scylla-operator-bot bot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Sep 17, 2024
@tnozicka tnozicka removed the lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. label Sep 17, 2024
@scylla-operator-bot scylla-operator-bot
Copy link
Contributor

The Scylla Operator project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

  • After 30d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue as fresh with /remove-lifecycle stale
  • Close this issue with /close
  • Offer to help out

/lifecycle stale

@scylla-operator-bot scylla-operator-bot bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Oct 18, 2024
@tnozicka
Copy link
Member

/remove-lifecycle stale

@scylla-operator-bot scylla-operator-bot bot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Oct 18, 2024
@tnozicka
Copy link
Member

https://prow.scylla-operator.scylladb.com/view/gs/scylla-operator-prow/pr-logs/pull/scylladb_local-csi-driver/64/pull-local-csi-driver-master-sanity/1851957381261430784#1:build-log.txt%3A30

@tnozicka tnozicka mentioned this issue Oct 31, 2024
Merged
@tnozicka
Copy link
Member

https://prow.scylla-operator.scylladb.com/view/gs/scylla-operator-prow/pr-logs/pull/scylladb_local-csi-driver/64/pull-local-csi-driver-master-sanity/1851958342826594304#1:build-log.txt%3A26

@tnozicka
Copy link
Member

tnozicka commented Nov 1, 2024

https://prow.scylla-operator.scylladb.com/view/gs/scylla-operator-prow/pr-logs/pull/scylladb_local-csi-driver/65/pull-local-csi-driver-master-sanity/1852002790944018432#1:build-log.txt%3A29

@tnozicka tnozicka mentioned this issue Nov 1, 2024
Merged
@zimnx zimnx self-assigned this Nov 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@zimnx zimnx

Labels
kind/flake Categorizes issue or PR as related to a flaky test. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@zimnx @tnozicka @rzetelskik