Persistent EOF errors when trying to sync directory

[isabanin]

Hey,

We are using s3sync extensively to sync thousands of Subversion and Git repositories to S3. Most of the time it works just great, but recently I had this very strange case when s3sync would fail to sync one specific Subversion repository. I ran s3sync with debugging enabled and noticed this:

s3TreeRecurse xxx xxx /db/revprops/102/
Trying command list_bucket xxx prefix xxx/xxx/db/revprops/102/ max-keys 200 delimiter / with 100 retries left
EOF error: end of file reached
No result available
99 retries left, sleeping for 30 seconds

It would go through all 100 retires, throw the same EOF error every time and then exit. Every time I run it it fails on a different directory inside "db/revprops". This directory contains around 12,000 files.

Any ideas?

[isabanin]

Here's a better excerpt from a verbose output from s3sync with debug enabled:

local item /storage/performances.git/objects/d9/384ec76a47eebeade1369aa93e97ecb8cb904f
local node object init. Name:objects/d9/384ec76a47eebeade1369aa93e97ecb8cb904f Path:/storage/performances.git/objects/d9/384ec76a47eebeade1369aa93e97ecb8cb904f Size:1983 Tag:79f8564e7b07180644dbd2431d83fded Date:Tue Dec 06 17:02:07 UTC 2011
source: objects/d9/384ec76a47eebeade1369aa93e97ecb8cb904f
s3 node object init. Name:objects/d9/384ec76a47eebeade1369aa93e97ecb8cb904f Path:performances.git/objects/d9/384ec76a47eebeade1369aa93e97ecb8cb904f Size: Tag: Date:
Create node objects/d9/384ec76a47eebeade1369aa93e97ecb8cb904f
performances.git/objects/d9/384ec76a47eebeade1369aa93e97ecb8cb904f
File extension: git/objects/d9/384ec76a47eebeade1369aa93e97ecb8cb904f
Trying command put backups performances.git/objects/d9/384ec76a47eebeade1369aa93e97ecb8cb904f # Content-Length 1983 with 99 retries left
Connection timed out: Connection timed out - connect(2)
warning: peer certificate won't be verified in this SSL session
No result available
98 retries left, sleeping for 30 seconds
Trying command put backups performances.git/objects/d9/384ec76a47eebeade1369aa93e97ecb8cb904f # Content-Length 1983 with 98 retries left
EOF error: end of file reached
warning: peer certificate won't be verified in this SSL session
No result available
97 retries left, sleeping for 30 seconds

So far it looks like s3sync can not recover from a connection time out properly.

[tomchiverton]

As a work around, you can use find to try each top level directory in turn.