Error reading Landsat 9 files #101
Replies: 7 comments 16 replies
-
|
Thanks for reporting this. It seems that the USGS server returned a bad result, and our data pipeline didn't handle it properly. The actual contents of that file are HTML Simplifying things a bit: import planetary_computer
import requests
from IPython.display import HTML
url = "https://landsateuwest.blob.core.windows.net/landsat-c2/level-2/standard/oli-tirs/2022/174/072/LC09_L2SP_174072_20220705_20220803_02_T1/LC09_L2SP_174072_20220705_20220803_02_T1_SR_B7.TIF"
url = planetary_computer.sign(url)
HTML(requests.get(url).text)
We'll look into it (along with searching for other places this might have happened) |
Beta Was this translation helpful? Give feedback.
-
|
Found another one at |
Beta Was this translation helpful? Give feedback.
-
|
It looks like the problem may apply to Landsat 8 imagery in the new collection as well. I ran into the problem with this file |
Beta Was this translation helpful? Give feedback.
-
|
Any movement on reprocessing these files? |
Beta Was this translation helpful? Give feedback.
-
|
Some. The problem we've run into is that USGS has apparently removed some of original scenes, some of which have been replaced by new scenes (reprocessed) but others haven't. We're trying to track those down before figuring out alternatives. |
Beta Was this translation helpful? Give feedback.
-
|
Quick status update here: we've got a list of scenes with the bad TIFF files. USGS has reprocessed many of those scenes (and deleted the original data). I'm able to get a list of the matching scenes and will begin to copy the data over once USGS's distribution service has stabilized a bit. It will take some time for all of the bytes to be copied. The STAC item IDs will not change (we don't include the processing timestamp in our item IDs). The assets for that scene will be all new, and the original assets will be deleted. |
Beta Was this translation helpful? Give feedback.
-
|
It's probably on your list but here is another file with this issue: https://landsateuwest.blob.core.windows.net/landsat-c2/level-2/standard/oli-tirs/2022/101/055/LC08_L2SP_101055_20220612_20220617_02_T2/LC08_L2SP_101055_20220612_20220617_02_T2_SR_B3.TIF Trying to get stackstac or ods.stac to ignore files like this has proven difficult (using e.g. 'errors_as_nodata' for stacstack.stack doesn't seem to working in this case). Short of trying to open every asset and catching errors before calling stack, I've resorted to maintaining a list of bad item ids to filter the itemcollection by prior to reading. Anyone have alternative ideas? |
Beta Was this translation helpful? Give feedback.
-
|
That scene is indeed on the list. We're currently paused since USGS is offline for maintenance. It'll pick back up again tonight or tomorrow. That said, we are seeing a number of "fixed" scenes missing from USGS's download endpoint. I'm waiting to hear back about those. |
Beta Was this translation helpful? Give feedback.
-
|
We've resorted to establishing our own list of bad ids and filtering them out of any item collections. Would it be possible to get access the full list that you created? |
Beta Was this translation helpful? Give feedback.
-
|
I suspect that list won't be too useful now, since it's now a mixture of items that were fixed with this reingestion (which just completed) and items that failed to reingest. I'll post an update once I hear back from USGS, and we'll work towards getting every single item fixed. |
Beta Was this translation helpful? Give feedback.
-
|
Just in case it's not on the list, wanted to flag https://landsateuwest.blob.core.windows.net/landsat-c2/level-2/standard/oli-tirs/2022/014/031/LC08_L2SP_014031_20220626_20220706_02_T1/LC08_L2SP_014031_20220626_20220706_02_T1_SR_B3.TIF as the same issue 😄 Looking forward to an update when this is fixed! |
Beta Was this translation helpful? Give feedback.
-
|
That is on the list, and was unfortunately one of the ones that failed when we tried to reingest it :/ |
Beta Was this translation helpful? Give feedback.
-
|
Is there any way of catching the error message to feed a list of bad links and therefore, run the search again with a period ending right before the bad link date? Otherwise, one would have to manually list these scenes, which is not a good solution since the bad dates change from path/row to path/row. If this process of catching error, listing bad links and running the search again is possible, one can retrieve a whole time-series by concatenating the xarrays skipping the bad links. That is my goal! p.s.: I've been working in many parts of Brazil and for what I noticed was that the missing scenes occur in 2022. |
Beta Was this translation helpful? Give feedback.
-
It depends a bit on how exactly you're loading the data. Wit something like stackstac or odc.stac, I don't think it would be too easy. The STAC metadata (which is all that those tools use, until you actual read the data) doesn't know that the assets are invalid. A way to check this is to manually iterate over the assets you're using, do a HEAD request on each one, and check the size. If it's abnormally small (like the error message) then you can assume that asset is bad and skip the item. Not a great workaround though :/ |
Beta Was this translation helpful? Give feedback.
-
|
We've updated the STAC items for all of the scenes reported in this discussion. Note that for some of the scenes, the actual URLs in blob storage have changed because USGS reprocessed that scene since we originally pulled it and USGS embeds the processing date in the file paths. For example, the STAC item https://planetarycomputer.microsoft.com/api/stac/v1/collections/landsat-c2-l2/items/LC09_L2SP_174072_20220705_02_T1 previously linked to assets under the prefix Now, it links to assets under the prefix (note the We'll eventually delete the original assets, with the problematic files. I don't have an ETA yet, but we'll be auditing our landsat archive to find all the scenes with at least one buggy asset, and will re-download each of those scenes. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks Tom, this is fantastic! |
Beta Was this translation helpful? Give feedback.
-
|
Great news! Is this sufficient to mark this as answered or would it be better to keep the thread unanswered until the audit is complete? |
Beta Was this translation helpful? Give feedback.
-
|
Up to you :) Either way, I'll post here once everything is fixed for good. |
Beta Was this translation helpful? Give feedback.
-
|
I ran into this issue today with the following asset; one more for the list! |
Beta Was this translation helpful? Give feedback.
-
|
Hello! I've been running into this issue lately. So far I've been listing the offending files and ignoring them. Is it useful for me to post them here, or should I just stay put? Did you get around to auditing the landsat archive? Anything we can do to help? |
Beta Was this translation helpful? Give feedback.
-
|
Still on our TODO list, sorry. If you want to post a couple Item IDs / bad files feel free to post them and we could explicitly double check those when we do the audit / fix. But no deal if it's a hassle. |
Beta Was this translation helpful? Give feedback.
-
|
No worries at all, I can imagine how gargantuan a task this is with such a big catalogue! Below is a list with a few. To filter them out I ended up using the ['LC08_L2SP_126051_20221102_02_T1',
'LC08_L2SP_200037_20221108_02_T1',
'LC08_L2SP_171063_20221012_02_T1',
'LC09_L2SP_197055_20220401_02_T1',
'LC08_L2SP_198055_20221025_02_T1',
'LC09_L2SP_171063_20220630_02_T1',
'LC09_L2SP_171064_20220630_02_T1',
'LC08_L2SP_172075_20220512_02_T1',
'LC09_L2SP_172075_20220402_02_T1',
'LC08_L2SP_172063_20220731_02_T1',
'LC08_L2SP_172064_20220629_02_T1',
'LC08_L2SP_172062_20220731_02_T1',
'LC08_L2SP_186054_20221106_02_T1',
'LC08_L2SP_189052_20130611_02_T1'] |
Beta Was this translation helpful? Give feedback.
-
I am running through a loop which pulls the least cloudy Landsat image from the collection in a given time range. Some of these time ranges return Landsat 9 imagery. I am running code in this notebook. Here is the error I get for the least cloudy image of July 2022 which covers
array([ 24.784878, -16.664232]):Error
Any idea what is causing this? This is the second time it has come up and both times it was for Landsat 9 imagery
Beta Was this translation helpful? Give feedback.
All reactions