-
Notifications
You must be signed in to change notification settings - Fork 64
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Get error when creating volume snapshot #239
Comments
After fixing issue #273, that was causing different errors due to ArgoCD auto-sync policies, I am able to reproduce the problem. How to reproduce the error
VolumeSnapshot error messages are displayed at the end of the logs:
|
After upgrading to latest longhorn version (1.6), which requires a newer version of csi-external-snapshotter (6.3.2), and latest velero version (1.13.0), the error seems not to be solved
Backup process seems not to end. Its status remains "WaitingForPluginOperationsPartiallyFailed" velero backup get fullagain
NAME STATUS ERRORS WARNINGS CREATED EXPIRES STORAGE LOCATION SELECTOR
fullagain WaitingForPluginOperationsPartiallyFailed 5 5 2024-02-03 13:02:27 +0100 CET 29d default <none>
velero backup describe fullagain
Name: fullagain
Namespace: velero
Labels: velero.io/storage-location=default
Annotations: velero.io/resource-timeout=10m0s
velero.io/source-cluster-k8s-gitversion=v1.28.2+k3s1
velero.io/source-cluster-k8s-major-version=1
velero.io/source-cluster-k8s-minor-version=28
Phase: WaitingForPluginOperationsPartiallyFailed
Warnings:
Velero: message: /VolumeSnapshot has a temporary error Failed to create snapshot: failed to remove VolumeSnapshotBeingCreated annotation on the content snapcontent-2dc16475-3006-4eca-9fa6-4ca809b12492: "snapshot controller failed to update snapcontent-2dc16475-3006-4eca-9fa6-4ca809b12492 on API server: Operation cannot be fulfilled on volumesnapshotcontents.snapshot.storage.k8s.io \"snapcontent-2dc16475-3006-4eca-9fa6-4ca809b12492\": the object has been modified; please apply your changes to the latest version and try again". Snapshot controller will retry later.
message: /VolumeSnapshot has a temporary error Failed to create snapshot: failed to remove VolumeSnapshotBeingCreated annotation on the content snapcontent-f30bae8c-790d-461a-9be6-a53b63c974b5: "snapshot controller failed to update snapcontent-f30bae8c-790d-461a-9be6-a53b63c974b5 on API server: Operation cannot be fulfilled on volumesnapshotcontents.snapshot.storage.k8s.io \"snapcontent-f30bae8c-790d-461a-9be6-a53b63c974b5\": the object has been modified; please apply your changes to the latest version and try again". Snapshot controller will retry later.
message: /VolumeSnapshot has a temporary error Failed to check and update snapshot content: failed to remove VolumeSnapshotBeingCreated annotation on the content snapcontent-17cded38-345b-4a7e-abbf-d1d657cb31ca: "snapshot controller failed to update snapcontent-17cded38-345b-4a7e-abbf-d1d657cb31ca on API server: Operation cannot be fulfilled on volumesnapshotcontents.snapshot.storage.k8s.io \"snapcontent-17cded38-345b-4a7e-abbf-d1d657cb31ca\": the object has been modified; please apply your changes to the latest version and try again". Snapshot controller will retry later.
message: /VolumeSnapshot has a temporary error Failed to check and update snapshot content: failed to remove VolumeSnapshotBeingCreated annotation on the content snapcontent-3717b965-1ca1-4853-ba75-1f82d8eaf362: "snapshot controller failed to update snapcontent-3717b965-1ca1-4853-ba75-1f82d8eaf362 on API server: Operation cannot be fulfilled on volumesnapshotcontents.snapshot.storage.k8s.io \"snapcontent-3717b965-1ca1-4853-ba75-1f82d8eaf362\": the object has been modified; please apply your changes to the latest version and try again". Snapshot controller will retry later.
message: /VolumeSnapshot has a temporary error Failed to create snapshot: failed to remove VolumeSnapshotBeingCreated annotation on the content snapcontent-754910e0-a98d-4e56-a82c-f5b7b3b6da18: "snapshot controller failed to update snapcontent-754910e0-a98d-4e56-a82c-f5b7b3b6da18 on API server: Operation cannot be fulfilled on volumesnapshotcontents.snapshot.storage.k8s.io \"snapcontent-754910e0-a98d-4e56-a82c-f5b7b3b6da18\": the object has been modified; please apply your changes to the latest version and try again". Snapshot controller will retry later.
Cluster: <none>
Namespaces: <none>
Errors:
Velero: message: /0
message: /1
message: /2
message: /3
message: /4
Cluster: <none>
Namespaces: <none>
Namespaces:
Included: *
Excluded: <none>
Resources:
Included: *
Excluded: <none>
Cluster-scoped: auto
Label selector: <none>
Storage Location: default
Velero-Native Snapshot PVs: auto
Snapshot Move Data: false
Data Mover: velero
TTL: 720h0m0s
CSISnapshotTimeout: 10m0s
ItemOperationTimeout: 4h0m0s
Hooks: <none>
Backup Format Version: 1.1.0
Started: 2024-02-03 13:02:27 +0100 CET
Completed: <n/a>
Expiration: 2024-03-04 13:02:27 +0100 CET
Total items to be backed up: 11882
Items backed up: 11882
Backup Item Operations: 28 of 38 completed successfully, 5 failed (specify --details for more information)
Velero-Native Snapshots: <none included>
time="2024-02-03T12:07:12Z" level=warning msg="VolumeSnapshot has a temporary error Failed to create snapshot: failed to remove VolumeSnapshotBeingCreated annotation on the content snapcontent-2dc16475-3006-4eca-9fa6-4ca809b12492: \"snapshot controller failed to update snapcontent-2dc16475-3006-4eca-9fa6-4ca809b12492 on API server: Operation cannot be fulfilled on volumesnapshotcontents.snapshot.storage.k8s.io \\\"snapcontent-2dc16475-3006-4eca-9fa6-4ca809b12492\\\": the object has been modified; please apply your changes to the latest version and try again\". Snapshot controller will retry later." backup=velero/fullagain cmd=/plugins/velero-plugin-for-csi logSource="/go/src/velero-plugin-for-csi/internal/backup/volumesnapshot_action.go:250" pluginName=velero-plugin-for-csi
time="2024-02-03T12:07:12Z" level=warning msg="VolumeSnapshot has a temporary error Failed to create snapshot: failed to remove VolumeSnapshotBeingCreated annotation on the content snapcontent-f30bae8c-790d-461a-9be6-a53b63c974b5: \"snapshot controller failed to update snapcontent-f30bae8c-790d-461a-9be6-a53b63c974b5 on API server: Operation cannot be fulfilled on volumesnapshotcontents.snapshot.storage.k8s.io \\\"snapcontent-f30bae8c-790d-461a-9be6-a53b63c974b5\\\": the object has been modified; please apply your changes to the latest version and try again\". Snapshot controller will retry later." backup=velero/fullagain cmd=/plugins/velero-plugin-for-csi logSource="/go/src/velero-plugin-for-csi/internal/backup/volumesnapshot_action.go:250" pluginName=velero-plugin-for-csi
time="2024-02-03T12:07:12Z" level=warning msg="VolumeSnapshot has a temporary error Failed to check and update snapshot content: failed to remove VolumeSnapshotBeingCreated annotation on the content snapcontent-17cded38-345b-4a7e-abbf-d1d657cb31ca: \"snapshot controller failed to update snapcontent-17cded38-345b-4a7e-abbf-d1d657cb31ca on API server: Operation cannot be fulfilled on volumesnapshotcontents.snapshot.storage.k8s.io \\\"snapcontent-17cded38-345b-4a7e-abbf-d1d657cb31ca\\\": the object has been modified; please apply your changes to the latest version and try again\". Snapshot controller will retry later." backup=velero/fullagain cmd=/plugins/velero-plugin-for-csi logSource="/go/src/velero-plugin-for-csi/internal/backup/volumesnapshot_action.go:250" pluginName=velero-plugin-for-csi
time="2024-02-03T12:07:13Z" level=warning msg="VolumeSnapshot has a temporary error Failed to check and update snapshot content: failed to remove VolumeSnapshotBeingCreated annotation on the content snapcontent-3717b965-1ca1-4853-ba75-1f82d8eaf362: \"snapshot controller failed to update snapcontent-3717b965-1ca1-4853-ba75-1f82d8eaf362 on API server: Operation cannot be fulfilled on volumesnapshotcontents.snapshot.storage.k8s.io \\\"snapcontent-3717b965-1ca1-4853-ba75-1f82d8eaf362\\\": the object has been modified; please apply your changes to the latest version and try again\". Snapshot controller will retry later." backup=velero/fullagain cmd=/plugins/velero-plugin-for-csi logSource="/go/src/velero-plugin-for-csi/internal/backup/volumesnapshot_action.go:250" pluginName=velero-plugin-for-csi
time="2024-02-03T12:07:13Z" level=warning msg="VolumeSnapshot has a temporary error Failed to create snapshot: failed to remove VolumeSnapshotBeingCreated annotation on the content snapcontent-754910e0-a98d-4e56-a82c-f5b7b3b6da18: \"snapshot controller failed to update snapcontent-754910e0-a98d-4e56-a82c-f5b7b3b6da18 on API server: Operation cannot be fulfilled on volumesnapshotcontents.snapshot.storage.k8s.io \\\"snapcontent-754910e0-a98d-4e56-a82c-f5b7b3b6da18\\\": the object has been modified; please apply your changes to the latest version and try again\". Snapshot controller will retry later." backup=velero/fullagain cmd=/plugins/velero-plugin-for-csi logSource="/go/src/velero-plugin-for-csi/internal/backup/volumesnapshot_action.go:250" pluginName=velero-plugin-for-csi
time="2024-02-03T12:07:13Z" level=error msg=0 backup=velero/fullagain logSource="pkg/controller/backup_controller.go:678"
time="2024-02-03T12:07:13Z" level=error msg=1 backup=velero/fullagain logSource="pkg/controller/backup_controller.go:678"
time="2024-02-03T12:07:13Z" level=error msg=2 backup=velero/fullagain logSource="pkg/controller/backup_controller.go:678"
time="2024-02-03T12:07:13Z" level=error msg=3 backup=velero/fullagain logSource="pkg/controller/backup_controller.go:678"
time="2024-02-03T12:07:13Z" level=error msg=4 backup=velero/fullagain logSource="pkg/controller/backup_controller.go:678" and velero logs shows that it keep on tryng the failed operation: time="2024-02-03T13:43:31Z" level=info msg="BackupStorageLocations is valid, marking as available" backup-storage-location=velero/default controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:126"
time="2024-02-03T13:43:31Z" level=warning msg="VolumeSnapshot has a temporary error Failed to create snapshot: failed to remove VolumeSnapshotBeingCreated annotation on the content snapcontent-f30bae8c-790d-461a-9be6-a53b63c974b5: \"snapshot controller failed to update snapcontent-f30bae8c-790d-461a-9be6-a53b63c974b5 on API server: Operation cannot be fulfilled on volumesnapshotcontents.snapshot.storage.k8s.io \\\"snapcontent-f30bae8c-790d-461a-9be6-a53b63c974b5\\\": the object has been modified; please apply your changes to the latest version and try again\". Snapshot controller will retry later." cmd=/plugins/velero-plugin-for-csi logSource="/go/src/velero-plugin-for-csi/internal/backup/volumesnapshot_action.go:250" pluginName=velero-plugin-for-csi
time="2024-02-03T13:43:31Z" level=warning msg="VolumeSnapshot has a temporary error Failed to check and update snapshot content: failed to remove VolumeSnapshotBeingCreated annotation on the content snapcontent-17cded38-345b-4a7e-abbf-d1d657cb31ca: \"snapshot controller failed to update snapcontent-17cded38-345b-4a7e-abbf-d1d657cb31ca on API server: Operation cannot be fulfilled on volumesnapshotcontents.snapshot.storage.k8s.io \\\"snapcontent-17cded38-345b-4a7e-abbf-d1d657cb31ca\\\": the object has been modified; please apply your changes to the latest version and try again\". Snapshot controller will retry later." cmd=/plugins/velero-plugin-for-csi logSource="/go/src/velero-plugin-for-csi/internal/backup/volumesnapshot_action.go:250" pluginName=velero-plugin-for-csi
time="2024-02-03T13:43:31Z" level=warning msg="VolumeSnapshot has a temporary error Failed to check and update snapshot content: failed to remove VolumeSnapshotBeingCreated annotation on the content snapcontent-3717b965-1ca1-4853-ba75-1f82d8eaf362: \"snapshot controller failed to update snapcontent-3717b965-1ca1-4853-ba75-1f82d8eaf362 on API server: Operation cannot be fulfilled on volumesnapshotcontents.snapshot.storage.k8s.io \\\"snapcontent-3717b965-1ca1-4853-ba75-1f82d8eaf362\\\": the object has been modified; please apply your changes to the latest version and try again\". Snapshot controller will retry later." cmd=/plugins/velero-plugin-for-csi logSource="/go/src/velero-plugin-for-csi/internal/backup/volumesnapshot_action.go:250" pluginName=velero-plugin-for-csi
time="2024-02-03T13:43:31Z" level=warning msg="VolumeSnapshot has a temporary error Failed to create snapshot: failed to remove VolumeSnapshotBeingCreated annotation on the content snapcontent-754910e0-a98d-4e56-a82c-f5b7b3b6da18: \"snapshot controller failed to update snapcontent-754910e0-a98d-4e56-a82c-f5b7b3b6da18 on API server: Operation cannot be fulfilled on volumesnapshotcontents.snapshot.storage.k8s.io \\\"snapcontent-754910e0-a98d-4e56-a82c-f5b7b3b6da18\\\": the object has been modified; please apply your changes to the latest version and try again\". Snapshot controller will retry later." cmd=/plugins/velero-plugin-for-csi logSource="/go/src/velero-plugin-for-csi/internal/backup/volumesnapshot_action.go:250" pluginName=velero-plugin-for-csi
|
It seems there is a external-snapshotter's PR kubernetes-csi/external-snapshotter#876 that might solve this issue. This PR is merged into main branch but it is not going to be merged in 6.3 release branch (kubernetes-csi/external-snapshotter#876 (comment)) Another PR has been created that merges the changes into 6.3.3 branch: kubernetes-csi/external-snapshotter#974. Latest version of Longhorn (1.6) does support csi-external-snapshotter 6.3.2. 6.3.3 version is not supported yet. |
It's better explained here
kubernetes-csi/external-snapshotter#300
I hope longhorn will support latest version soon:
time="2023-12-04T14:31:24Z" level=warning msg="VolumeSnapshot has a temporary error Failed to check and update snapshot content: failed to remove VolumeSnapshotBeingCreated annotation on the content snapcontent-0e1ad33d-708e-42d6-b091-95250cac6dc2: \"snapshot controller failed to update snapcontent-0e1ad33d-708e-42d6-b091-95250cac6dc2 on API server: Operation cannot be fulfilled on volumesnapshotcontents.snapshot.storage.k8s.io \\\"snapcontent-0e1ad33d-708e-42d6-b091-95250cac6dc2\\\": the object has been modified; please apply your changes to the latest version and try again\". Snapshot controller will retry later." cmd=/plugins/velero-plugin-for-csi logSource="/go/src/velero-plugin-for-csi/internal/backup/volumesnapshot_action.go:250" pluginName=velero-plugin-for-csi
The text was updated successfully, but these errors were encountered: