[Bug]: Content Discovery pagination support

[uvegla]

zot version

v2.1.1

Describe the bug

When setting up onDemand: false sync extension rules, the prefix matching is done only the 1st page result of the content discovery made towards the remote registry. With large registries this results in some images never getting synced by Zot.

The pagination rules are defined by the specification here: https://github.com/opencontainers/distribution-spec/blob/v1.1.0/spec.md#content-discovery

The code that does the discovery is rooted here: https://github.com/project-zot/zot/blob/v2.1.1/pkg/extensions/sync/service.go#L221-L232

It makes a call to <remote>/v2/_catalog and only makes it once. Going further down the path, the Link response header is not checked and simply the response body is returned, thus not getting a full picture of the registry.

This could potentially be an issue for listing tags as well - as according to the docs, that supports pagination as well - but I did not check that use case.

To reproduce

Configuration

You have to try this with a large registry with more than a 100 repositories. Relevant part of the Zot JSON configuration:

"registries": [
  {
    "urls": [
      "https://example.org/"
    ],
    "onDemand": false,
    "pollInterval": "12h",
    "tlsVerify": true,
    "maxRetries": 3,
    "retryDelay": "30m",
    "onlySigned": false,
    "content": [
      {
        "prefix": "my-org/aaa-operator",
        "tags": {
          "regex": "^v?\\d+\\.\\d+\\.\\d+$"
        }
      },
      {
        "prefix": "my-org/zzz-operator",
        "tags": {
          "regex": "^v?\\d+\\.\\d+\\.\\d+$"
        }
      }
    ]
  }
]

Then simply run:

zot serve config.json

Assuming there are more than 100 images in the repository and my-org/zzz-operator is alphabetically beyond 100, the syncing will never occur for it, practically it is ignored.

Manual reproduction

# 1st 100 repositories
curl -H "Authorization: Bearer $(curl -s https://<REMOTE>/oauth2/token\?scope\=registry%3Acatalog%3A%2A\&service\=<REMOTE> | jq -r .access_token)" https://<REMOTE>/v2/_catalog

# Take the contents on the Link response header and make follow up requests, for example
curl -H "Authorization: Bearer $(curl -s https://<REMOTE>/oauth2/token\?scope\=registry%3Acatalog%3A%2A\&service\=<REMOTE> | jq -r .access_token)" 'https://<REMOTE>/v2/_catalog?last=my-org%2Fxxx-operator&n=100&orderby=>'

# Repeat until Link response header is empty

Expected behavior

In the GetRepositories logic the Link response header should be respected and followed until it is empty. Potentially this might affect listing tags for repositories as well, according to the specification.

Screenshots

No response

Additional context

No response

[rchincha]

@eusebiu-constantin-petu-dbk can you take a look pls?