Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Get error when creating volume snapshot #239

Open
i5Js opened this issue Dec 4, 2023 · 3 comments
Open

Get error when creating volume snapshot #239

i5Js opened this issue Dec 4, 2023 · 3 comments
Labels
bug Something isn't working

Comments

@i5Js
Copy link

i5Js commented Dec 4, 2023

It's better explained here

kubernetes-csi/external-snapshotter#300

I hope longhorn will support latest version soon:

time="2023-12-04T14:31:24Z" level=warning msg="VolumeSnapshot has a temporary error Failed to check and update snapshot content: failed to remove VolumeSnapshotBeingCreated annotation on the content snapcontent-0e1ad33d-708e-42d6-b091-95250cac6dc2: \"snapshot controller failed to update snapcontent-0e1ad33d-708e-42d6-b091-95250cac6dc2 on API server: Operation cannot be fulfilled on volumesnapshotcontents.snapshot.storage.k8s.io \\\"snapcontent-0e1ad33d-708e-42d6-b091-95250cac6dc2\\\": the object has been modified; please apply your changes to the latest version and try again\". Snapshot controller will retry later." cmd=/plugins/velero-plugin-for-csi logSource="/go/src/velero-plugin-for-csi/internal/backup/volumesnapshot_action.go:250" pluginName=velero-plugin-for-csi

@ricsanfre
Copy link
Owner

ricsanfre commented Feb 3, 2024

After fixing issue #273, that was causing different errors due to ArgoCD auto-sync policies, I am able to reproduce the problem.

How to reproduce the error

  • Execute velero full backup

    velero backup create fullbackup --include-namespaces '*' --wait
  • Backup ends with "Backup completed with status: PartiallyFailed" message

  • Check logs

    velero backup logs fullbackup

VolumeSnapshot error messages are displayed at the end of the logs:

time="2024-02-03T10:22:07Z" level=warning msg="VolumeSnapshot has a temporary error Failed to create snapshot: failed to remove VolumeSnapshotBeingCreated annotation on the content snapcontent-2a626df8-48a5-48e1-810c-22f489ab0a7f: \"snapshot controller failed to update snapcontent-2a626df8-48a5-48e1-810c-22f489ab0a7f on API server: Operation cannot be fulfilled on volumesnapshotcontents.snapshot.storage.k8s.io \\\"snapcontent-2a626df8-48a5-48e1-810c-22f489ab0a7f\\\": the object has been modified; please apply your changes to the latest version and try again\". Snapshot controller will retry later." backup=velero/full03022024 cmd=/plugins/velero-plugin-for-csi logSource="/go/src/velero-plugin-for-csi/internal/backup/volumesnapshot_action.go:250" pluginName=velero-plugin-for-csi
time="2024-02-03T10:22:08Z" level=warning msg="VolumeSnapshot has a temporary error Failed to check and update snapshot content: failed to remove VolumeSnapshotBeingCreated annotation on the content snapcontent-a339b4f4-dcac-49a3-8cb8-1a7ffbaefd5c: \"snapshot controller failed to update snapcontent-a339b4f4-dcac-49a3-8cb8-1a7ffbaefd5c on API server: Operation cannot be fulfilled on volumesnapshotcontents.snapshot.storage.k8s.io \\\"snapcontent-a339b4f4-dcac-49a3-8cb8-1a7ffbaefd5c\\\": the object has been modified; please apply your changes to the latest version and try again\". Snapshot controller will retry later." backup=velero/full03022024 cmd=/plugins/velero-plugin-for-csi logSource="/go/src/velero-plugin-for-csi/internal/backup/volumesnapshot_action.go:250" pluginName=velero-plugin-for-csi
time="2024-02-03T10:22:08Z" level=warning msg="VolumeSnapshot has a temporary error Failed to check and update snapshot content: failed to remove VolumeSnapshotBeingCreated annotation on the content snapcontent-98181105-aa85-4e05-8e46-c0670a1508ac: \"snapshot controller failed to update snapcontent-98181105-aa85-4e05-8e46-c0670a1508ac on API server: Operation cannot be fulfilled on volumesnapshotcontents.snapshot.storage.k8s.io \\\"snapcontent-98181105-aa85-4e05-8e46-c0670a1508ac\\\": the object has been modified; please apply your changes to the latest version and try again\". Snapshot controller will retry later." backup=velero/full03022024 cmd=/plugins/velero-plugin-for-csi logSource="/go/src/velero-plugin-for-csi/internal/backup/volumesnapshot_action.go:250" pluginName=velero-plugin-for-csi
time="2024-02-03T10:22:08Z" level=error msg=0 backup=velero/full03022024 logSource="pkg/controller/backup_controller.go:675"
time="2024-02-03T10:22:08Z" level=error msg=1 backup=velero/full03022024 logSource="pkg/controller/backup_controller.go:675"
time="2024-02-03T10:22:08Z" level=error msg=2 backup=velero/full03022024 logSource="pkg/controller/backup_controller.go:675"

  • Same error messages appears in longhorn's csi-snapshotter pod logs:
     I0203 10:20:51.894692       1 event.go:285] Event(v1.ObjectReference{Kind:"VolumeSnapshotContent", Namespace:"", Name:"snapcontent-2a626df8-48a5-48e1-810c-22f489ab0a7f", UID:"1342dbbf-1767-4676-907b-2a85eee25a5d", APIVersion:"snapshot.storage.k8s.io/v1", ResourceVersion:"272872", FieldPath:""}): type: 'Warning' reason: 'SnapshotCreationFailed' Failed to create snapshot: failed to remove VolumeSnapshotBeingCreated annotation on the content snapcontent-2a626df8-48a5-48e1-810c-22f489ab0a7f: "snapshot controller failed to update snapcontent-2a626df8-48a5-48e1-810c-22f489ab0a7f on API server: Operation cannot be fulfilled on volumesnapshotcontents.snapshot.storage.k8s.io \"snapcontent-2a626df8-48a5-48e1-810c-22f489ab0a7f\": the object has been modified; please apply your changes to the latest version and try again
    

@ricsanfre ricsanfre added the bug Something isn't working label Feb 3, 2024
@ricsanfre
Copy link
Owner

After upgrading to latest longhorn version (1.6), which requires a newer version of csi-external-snapshotter (6.3.2), and latest velero version (1.13.0), the error seems not to be solved

  • velero backup get command

Backup process seems not to end. Its status remains "WaitingForPluginOperationsPartiallyFailed"

velero backup get fullagain
NAME        STATUS                                      ERRORS   WARNINGS   CREATED                         EXPIRES   STORAGE LOCATION   SELECTOR
fullagain   WaitingForPluginOperationsPartiallyFailed   5        5          2024-02-03 13:02:27 +0100 CET   29d       default            <none>
  • velero backup describe command
velero backup describe fullagain
Name:         fullagain
Namespace:    velero
Labels:       velero.io/storage-location=default
Annotations:  velero.io/resource-timeout=10m0s
              velero.io/source-cluster-k8s-gitversion=v1.28.2+k3s1
              velero.io/source-cluster-k8s-major-version=1
              velero.io/source-cluster-k8s-minor-version=28

Phase:  WaitingForPluginOperationsPartiallyFailed


Warnings:
  Velero:    message: /VolumeSnapshot has a temporary error Failed to create snapshot: failed to remove VolumeSnapshotBeingCreated annotation on the content snapcontent-2dc16475-3006-4eca-9fa6-4ca809b12492: "snapshot controller failed to update snapcontent-2dc16475-3006-4eca-9fa6-4ca809b12492 on API server: Operation cannot be fulfilled on volumesnapshotcontents.snapshot.storage.k8s.io \"snapcontent-2dc16475-3006-4eca-9fa6-4ca809b12492\": the object has been modified; please apply your changes to the latest version and try again". Snapshot controller will retry later.
             message: /VolumeSnapshot has a temporary error Failed to create snapshot: failed to remove VolumeSnapshotBeingCreated annotation on the content snapcontent-f30bae8c-790d-461a-9be6-a53b63c974b5: "snapshot controller failed to update snapcontent-f30bae8c-790d-461a-9be6-a53b63c974b5 on API server: Operation cannot be fulfilled on volumesnapshotcontents.snapshot.storage.k8s.io \"snapcontent-f30bae8c-790d-461a-9be6-a53b63c974b5\": the object has been modified; please apply your changes to the latest version and try again". Snapshot controller will retry later.
             message: /VolumeSnapshot has a temporary error Failed to check and update snapshot content: failed to remove VolumeSnapshotBeingCreated annotation on the content snapcontent-17cded38-345b-4a7e-abbf-d1d657cb31ca: "snapshot controller failed to update snapcontent-17cded38-345b-4a7e-abbf-d1d657cb31ca on API server: Operation cannot be fulfilled on volumesnapshotcontents.snapshot.storage.k8s.io \"snapcontent-17cded38-345b-4a7e-abbf-d1d657cb31ca\": the object has been modified; please apply your changes to the latest version and try again". Snapshot controller will retry later.
             message: /VolumeSnapshot has a temporary error Failed to check and update snapshot content: failed to remove VolumeSnapshotBeingCreated annotation on the content snapcontent-3717b965-1ca1-4853-ba75-1f82d8eaf362: "snapshot controller failed to update snapcontent-3717b965-1ca1-4853-ba75-1f82d8eaf362 on API server: Operation cannot be fulfilled on volumesnapshotcontents.snapshot.storage.k8s.io \"snapcontent-3717b965-1ca1-4853-ba75-1f82d8eaf362\": the object has been modified; please apply your changes to the latest version and try again". Snapshot controller will retry later.
             message: /VolumeSnapshot has a temporary error Failed to create snapshot: failed to remove VolumeSnapshotBeingCreated annotation on the content snapcontent-754910e0-a98d-4e56-a82c-f5b7b3b6da18: "snapshot controller failed to update snapcontent-754910e0-a98d-4e56-a82c-f5b7b3b6da18 on API server: Operation cannot be fulfilled on volumesnapshotcontents.snapshot.storage.k8s.io \"snapcontent-754910e0-a98d-4e56-a82c-f5b7b3b6da18\": the object has been modified; please apply your changes to the latest version and try again". Snapshot controller will retry later.
  Cluster:    <none>
  Namespaces: <none>

Errors:
  Velero:    message: /0
             message: /1
             message: /2
             message: /3
             message: /4
  Cluster:    <none>
  Namespaces: <none>

Namespaces:
  Included:  *
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        <none>
  Cluster-scoped:  auto

Label selector:  <none>

Storage Location:  default

Velero-Native Snapshot PVs:  auto
Snapshot Move Data:          false
Data Mover:                  velero

TTL:  720h0m0s

CSISnapshotTimeout:    10m0s
ItemOperationTimeout:  4h0m0s

Hooks:  <none>

Backup Format Version:  1.1.0

Started:    2024-02-03 13:02:27 +0100 CET
Completed:  <n/a>

Expiration:  2024-03-04 13:02:27 +0100 CET

Total items to be backed up:  11882
Items backed up:              11882

Backup Item Operations:  28 of 38 completed successfully, 5 failed (specify --details for more information)
Velero-Native Snapshots: <none included>
  • velero backup log command
time="2024-02-03T12:07:12Z" level=warning msg="VolumeSnapshot has a temporary error Failed to create snapshot: failed to remove VolumeSnapshotBeingCreated annotation on the content snapcontent-2dc16475-3006-4eca-9fa6-4ca809b12492: \"snapshot controller failed to update snapcontent-2dc16475-3006-4eca-9fa6-4ca809b12492 on API server: Operation cannot be fulfilled on volumesnapshotcontents.snapshot.storage.k8s.io \\\"snapcontent-2dc16475-3006-4eca-9fa6-4ca809b12492\\\": the object has been modified; please apply your changes to the latest version and try again\". Snapshot controller will retry later." backup=velero/fullagain cmd=/plugins/velero-plugin-for-csi logSource="/go/src/velero-plugin-for-csi/internal/backup/volumesnapshot_action.go:250" pluginName=velero-plugin-for-csi
time="2024-02-03T12:07:12Z" level=warning msg="VolumeSnapshot has a temporary error Failed to create snapshot: failed to remove VolumeSnapshotBeingCreated annotation on the content snapcontent-f30bae8c-790d-461a-9be6-a53b63c974b5: \"snapshot controller failed to update snapcontent-f30bae8c-790d-461a-9be6-a53b63c974b5 on API server: Operation cannot be fulfilled on volumesnapshotcontents.snapshot.storage.k8s.io \\\"snapcontent-f30bae8c-790d-461a-9be6-a53b63c974b5\\\": the object has been modified; please apply your changes to the latest version and try again\". Snapshot controller will retry later." backup=velero/fullagain cmd=/plugins/velero-plugin-for-csi logSource="/go/src/velero-plugin-for-csi/internal/backup/volumesnapshot_action.go:250" pluginName=velero-plugin-for-csi
time="2024-02-03T12:07:12Z" level=warning msg="VolumeSnapshot has a temporary error Failed to check and update snapshot content: failed to remove VolumeSnapshotBeingCreated annotation on the content snapcontent-17cded38-345b-4a7e-abbf-d1d657cb31ca: \"snapshot controller failed to update snapcontent-17cded38-345b-4a7e-abbf-d1d657cb31ca on API server: Operation cannot be fulfilled on volumesnapshotcontents.snapshot.storage.k8s.io \\\"snapcontent-17cded38-345b-4a7e-abbf-d1d657cb31ca\\\": the object has been modified; please apply your changes to the latest version and try again\". Snapshot controller will retry later." backup=velero/fullagain cmd=/plugins/velero-plugin-for-csi logSource="/go/src/velero-plugin-for-csi/internal/backup/volumesnapshot_action.go:250" pluginName=velero-plugin-for-csi
time="2024-02-03T12:07:13Z" level=warning msg="VolumeSnapshot has a temporary error Failed to check and update snapshot content: failed to remove VolumeSnapshotBeingCreated annotation on the content snapcontent-3717b965-1ca1-4853-ba75-1f82d8eaf362: \"snapshot controller failed to update snapcontent-3717b965-1ca1-4853-ba75-1f82d8eaf362 on API server: Operation cannot be fulfilled on volumesnapshotcontents.snapshot.storage.k8s.io \\\"snapcontent-3717b965-1ca1-4853-ba75-1f82d8eaf362\\\": the object has been modified; please apply your changes to the latest version and try again\". Snapshot controller will retry later." backup=velero/fullagain cmd=/plugins/velero-plugin-for-csi logSource="/go/src/velero-plugin-for-csi/internal/backup/volumesnapshot_action.go:250" pluginName=velero-plugin-for-csi
time="2024-02-03T12:07:13Z" level=warning msg="VolumeSnapshot has a temporary error Failed to create snapshot: failed to remove VolumeSnapshotBeingCreated annotation on the content snapcontent-754910e0-a98d-4e56-a82c-f5b7b3b6da18: \"snapshot controller failed to update snapcontent-754910e0-a98d-4e56-a82c-f5b7b3b6da18 on API server: Operation cannot be fulfilled on volumesnapshotcontents.snapshot.storage.k8s.io \\\"snapcontent-754910e0-a98d-4e56-a82c-f5b7b3b6da18\\\": the object has been modified; please apply your changes to the latest version and try again\". Snapshot controller will retry later." backup=velero/fullagain cmd=/plugins/velero-plugin-for-csi logSource="/go/src/velero-plugin-for-csi/internal/backup/volumesnapshot_action.go:250" pluginName=velero-plugin-for-csi
time="2024-02-03T12:07:13Z" level=error msg=0 backup=velero/fullagain logSource="pkg/controller/backup_controller.go:678"
time="2024-02-03T12:07:13Z" level=error msg=1 backup=velero/fullagain logSource="pkg/controller/backup_controller.go:678"
time="2024-02-03T12:07:13Z" level=error msg=2 backup=velero/fullagain logSource="pkg/controller/backup_controller.go:678"
time="2024-02-03T12:07:13Z" level=error msg=3 backup=velero/fullagain logSource="pkg/controller/backup_controller.go:678"
time="2024-02-03T12:07:13Z" level=error msg=4 backup=velero/fullagain logSource="pkg/controller/backup_controller.go:678"

and velero logs shows that it keep on tryng the failed operation:

time="2024-02-03T13:43:31Z" level=info msg="BackupStorageLocations is valid, marking as available" backup-storage-location=velero/default controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:126"
time="2024-02-03T13:43:31Z" level=warning msg="VolumeSnapshot has a temporary error Failed to create snapshot: failed to remove VolumeSnapshotBeingCreated annotation on the content snapcontent-f30bae8c-790d-461a-9be6-a53b63c974b5: \"snapshot controller failed to update snapcontent-f30bae8c-790d-461a-9be6-a53b63c974b5 on API server: Operation cannot be fulfilled on volumesnapshotcontents.snapshot.storage.k8s.io \\\"snapcontent-f30bae8c-790d-461a-9be6-a53b63c974b5\\\": the object has been modified; please apply your changes to the latest version and try again\". Snapshot controller will retry later." cmd=/plugins/velero-plugin-for-csi logSource="/go/src/velero-plugin-for-csi/internal/backup/volumesnapshot_action.go:250" pluginName=velero-plugin-for-csi
time="2024-02-03T13:43:31Z" level=warning msg="VolumeSnapshot has a temporary error Failed to check and update snapshot content: failed to remove VolumeSnapshotBeingCreated annotation on the content snapcontent-17cded38-345b-4a7e-abbf-d1d657cb31ca: \"snapshot controller failed to update snapcontent-17cded38-345b-4a7e-abbf-d1d657cb31ca on API server: Operation cannot be fulfilled on volumesnapshotcontents.snapshot.storage.k8s.io \\\"snapcontent-17cded38-345b-4a7e-abbf-d1d657cb31ca\\\": the object has been modified; please apply your changes to the latest version and try again\". Snapshot controller will retry later." cmd=/plugins/velero-plugin-for-csi logSource="/go/src/velero-plugin-for-csi/internal/backup/volumesnapshot_action.go:250" pluginName=velero-plugin-for-csi
time="2024-02-03T13:43:31Z" level=warning msg="VolumeSnapshot has a temporary error Failed to check and update snapshot content: failed to remove VolumeSnapshotBeingCreated annotation on the content snapcontent-3717b965-1ca1-4853-ba75-1f82d8eaf362: \"snapshot controller failed to update snapcontent-3717b965-1ca1-4853-ba75-1f82d8eaf362 on API server: Operation cannot be fulfilled on volumesnapshotcontents.snapshot.storage.k8s.io \\\"snapcontent-3717b965-1ca1-4853-ba75-1f82d8eaf362\\\": the object has been modified; please apply your changes to the latest version and try again\". Snapshot controller will retry later." cmd=/plugins/velero-plugin-for-csi logSource="/go/src/velero-plugin-for-csi/internal/backup/volumesnapshot_action.go:250" pluginName=velero-plugin-for-csi
time="2024-02-03T13:43:31Z" level=warning msg="VolumeSnapshot has a temporary error Failed to create snapshot: failed to remove VolumeSnapshotBeingCreated annotation on the content snapcontent-754910e0-a98d-4e56-a82c-f5b7b3b6da18: \"snapshot controller failed to update snapcontent-754910e0-a98d-4e56-a82c-f5b7b3b6da18 on API server: Operation cannot be fulfilled on volumesnapshotcontents.snapshot.storage.k8s.io \\\"snapcontent-754910e0-a98d-4e56-a82c-f5b7b3b6da18\\\": the object has been modified; please apply your changes to the latest version and try again\". Snapshot controller will retry later." cmd=/plugins/velero-plugin-for-csi logSource="/go/src/velero-plugin-for-csi/internal/backup/volumesnapshot_action.go:250" pluginName=velero-plugin-for-csi

@ricsanfre
Copy link
Owner

It seems there is a external-snapshotter's PR kubernetes-csi/external-snapshotter#876 that might solve this issue. This PR is merged into main branch but it is not going to be merged in 6.3 release branch (kubernetes-csi/external-snapshotter#876 (comment))

Another PR has been created that merges the changes into 6.3.3 branch: kubernetes-csi/external-snapshotter#974.

Latest version of Longhorn (1.6) does support csi-external-snapshotter 6.3.2. 6.3.3 version is not supported yet.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants