What format should I use for the last-modified property in list API response when using scheme = azblob?

[BaurzhanSakhariev]

Hi, thanks for providing a convenient way to access data!

Have a question about specific API.

I have working implementation for read/write from Azure Blob Storage using java_binding.

I tried to create a lightweight test by using stub http server.

Read/write work fine, but when I try to use list API, i get an error parse datetime from rfc2822 failed

After checking code, I think I'm hitting last modified date extraction
https://github.com/apache/opendal/blob/main/core/src/services/azblob/lister.rs#L99

I tried to add <Last-Modified>some date in rfc2822 format</LastModified> to the List Blobs part of the stub server and I can confirm that it does react on changes in that property but I never managed to get it working.

Experiments so far:

• input contains invalid characters error if I provide dummy string (just to test that list API is sensitive to this property)
• Connection reset by peer (os error 54) if I provide some valid date, say, Mon, 12 Aug 2024 11:16:46 UT
• OpenDAL passes x-ms-date header, I thought that maybe list expects modified date in the response to be the same with passed header? I tried using exchange.getRequestHeaders().get("X-ms-date").get(0) as LastModified but still getting os error 54.

It's probably similar to #3478.

What format should I use for Last-Modified property? Is there any specific requirement for the date other than being rfc2822 complaint?

Thanks!

[Xuanwo]

Hi, we are using chrono to parse DateTime by DateTime::parse_from_rfc2822. I believe Mon, 12 Aug 2024 11:16:46 UT should be a valid date time for us.

The error Connection reset by peer (os error 54) suggests there might be other network-related issues, such as the server closing the connection too early.

---

I'm not an expert in Java, so I just skimmed the code at AzureHttpHandler.java to give you some inspiration.

For example, the content length of the response could be incorrect if we have multi-byte characters like Chinese in the response:

https://github.com/crate/crate/blob/9ecf9b81515cb336683f62fc22b44d6876456e03/plugins/es-repository-azure/src/test/java/org/elasticsearch/repositories/azure/AzureHttpHandler.java#L189-L192

byte[] response = list.toString().getBytes(StandardCharsets.UTF_8);
exchange.getResponseHeaders().add("Content-Type", "application/xml");
exchange.sendResponseHeaders(RestStatus.OK.getStatus(), response.length);
exchange.getResponseBody().write(response);

[BaurzhanSakhariev]

Thanks for confirming date format and providing an idea!

I was thinking that OpenDAL does some extra operations with the date and wants smth on top of rfc.

I will check other directions

[Xuanwo]

Oh, sorry. The code I mentioned here appears to be correct.

[BaurzhanSakhariev]

In case its useful for someone else: in addition to <Last-Modified>some_date</Last-Modified>,
I had to add <NextMarker/> right after </Blobs> to avoid timeouts.