Paging is broken

[aliasmrchips]

Paging seems to be broken when using the default stac-server instance at https://earth-search.aws.element84.com/v0

The root cause seems to be the fact that the "next" links from stac-server are malformed.

For example, for a search with a bbox parameter:

search = Search(bbox=[-110, 39.5, -105, 40.5],
               collections=['sentinel-s2-l2a']
)

The initial POST returns otherwise correctly but with

"links": [
        {
            "rel": "next",
            "title": "Next page of results",
            "href": "https://earth-search.aws.element84.com/v0/search?intersects=%5Bobject%20Object%5D&collections[]=sentinel-s2-l2a-cogs&page=2&limit=500"
        }
    ]

which results in the client repeatedly requesting https://earth-search.aws.element84.com/v0/search?intersects=%5Bobject%20Object%5D&collections[]=sentinel-s2-l2a-cogs&page=2&limit=500 until it reaches the 10000 item limit.

This issue could be mitigated on the client, but sat-search is maybe doing the right thing here.

I have submitted a PR to stac-server stac-utils/stac-server#56 that seems to fix the issue.

[pablotcarreira]

Same here, search.found() returns 2493, but len(search.items()) returns 10000.

[gjoseph92]

Sounds like #107 and #109? (Howdy @aliasmrchips @acannistra #worldscollide)

[matthewhanson]

@aliasmrchips thanks for the report, I've not looked into this yet, but wanted you to know the issue has been seen here and at stac-server. Will spend some time today.

[smantovani]

@matthewhanson any update on this issue?

[matthewhanson]

@smantovani @aliasmrchips

sat-search is trying to do the right thing, and it works with other APIs that do pagination correctly, including the develop version of stac-server. That has been fixed on the develop branch of stac-server, as well as the earth-search deployed branch. Have to dig into the logs and redeploy.

[RichardScottOZ]

Think I am seeing this, too

e.g. query={'eo:cloud_cover': {'eq': 0}, 'platform':{'eq': 'sentinel-2a'}, 'sentinel:utm_zone':{'eq': 22} },

returns me 283

query={'eo:cloud_cover': {'eq': 0}, 'platform':{'eq': 'sentinel-2a'}, 'sentinel:utm_zone':{'gt': 20}, 'sentinel:utm_zone':{'lt': 24} },

returns 100000

or if I take the utmzone out part I get 10000

which is certainly not going to be the case for zero cloud numbers

bbox = [-58.9030617619999362,-10.2614079999999994, -41.7954588009999384,2.5911015890000608]

stac_items = satsearch.Search(
url="https://earth-search.aws.element84.com/v0",
bbox=bbox,
collections=["sentinel-s2-l2a-cogs"],
datetime="2020-03-01/2020-08-01",
#query={'eo:cloud_cover': {'eq': 0}, 'platform':{'eq': 'sentinel-2a'}, 'sentinel:utm_zone':{'gt': 20}, 'sentinel:utm_zone':{'lt': 24} },
#query={'eo:cloud_cover': {'eq': 0}, 'platform':{'eq': 'sentinel-2a'}, 'sentinel:utm_zone':{'gt': 20} },
query={'eo:cloud_cover': {'eq': 0}, 'platform':{'eq': 'sentinel-2a'}, 'sentinel:utm_zone':{'eq': 22} },
).items()

[RichardScottOZ]

same with query={'eo:cloud_cover': {'lt': 5}, 'platform':{'eq': 'sentinel-2a'}, 'sentinel:utm_zone':{'eq': 22} },

[RichardScottOZ]

or
stac_found = satsearch.Search(
url="https://earth-search.aws.element84.com/v0",
bbox=bbox,
collections=["sentinel-s2-l2a-cogs"],
query={'eo:cloud_cover': {'eq': 0}, 'platform':{'eq': 'sentinel-2a'}},
).found()

stac_items = satsearch.Search(
url="https://earth-search.aws.element84.com/v0",
bbox=bbox,
collections=["sentinel-s2-l2a-cogs"],
datetime="2017-01-01/2021-03-25",
query={'eo:cloud_cover': {'eq': 0}, 'platform':{'eq': 'sentinel-2a'}},
).items()

5287
searching time: 0:00:58.037354
Num of Items 10000