Uploaded files truncated to 16 megabytes

[jannefoo]

original file on phone: 17 797 262 bytes
file downloaded from aws s3 bucket: 16 777 216 bytes

I'm running 1.1.1 from f-droid on pixel 8a.

from exfilac's logs, its seems the file is uploaded as two 8388608 byte chunks, but three would've been needed.

a bug or have I misconfigured something?

[jannefoo]

btw, exfilac's log also say that hashes match between local and s3 files, but how that can be when file sizes mismatch? 🤔

sub-chunk sized files are uploaded intact.

[io7m]

Very strange.

What's the S3 server provider in this case?

As you may know, we go through the Amazon AWS SDK. For files as large as you mention, this will be the multi-part upload code path. Essentially, this method:

exfilac/com.io7m.exfilac.s3_uploader.amazon/src/main/java/com/io7m/exfilac/s3_uploader/amazon/EFS3AMZUpload.kt

Line 276 in 17b07f4

private fun executeUploadMultiPart(

You can see that we set the expected hash of the file as part of the metadata for the upload. This should mean (assuming that the S3 provider adheres to the spec) that you can't observe a file that exposes the hash value that exfilac reads back until that multi-part upload has been fully completed and the content verified on the server side.

I'm also seeing occasional odd 403 errors on a locally hosted MinIO server these past couple of days related to multi-part uploads, so I'm suspecting that something isn't quite right somewhere with respect to the underlying S3 client.

[jannefoo]

My host is Amazon S3.

[io7m]

Tracking this here: #29

[io7m]

Thanks for reporting this. This turned out to be a pretty serious issue with multi-part size calculation that was somehow left out of testing. The latest commit should fix this, but I'm running it through a period of testing locally before a new release goes out.