Unable to upload files to bucket

[Angelo-gh3990]

Deployed the motion S3 docker containers. Works well.
Created a bucket : aws --endpoint-url http://localhost:8000 s3 mb s3://mybucket

Looking up the bucket and showing what's in there seems to work:
aws --endpoint-url http://localhost:8000 s3 ls s3://mybucket

Now when I try to upload a file I get: aws --endpoint-url http://149.5.22.34:8000 s3 cp README.md s3://mybucket

I'm getting

upload failed: ./README.md to s3://mybucket/README.md An error occurred (ServiceUnavailable) when calling the PutObject operation (reached max retries: 4): The request has failed due to a temporary failure of the server.

[masih]

Make sure you have specified a wallet key for motion

please also provide the output of docker ps. My suspicion is that Motion failed due to wallet key not being specified.

[Angelo-gh3990]

Wallet key is in there (in the .enc file) - same as in the setup I use in Motion and that works

CONTAINER ID   IMAGE                                                   COMMAND                  CREATED          STATUS                    PORTS                                                                                                                                                                                                                                                 NAMES
c9596434d6aa   ghcr.io/filecoin-project/motion-cloudserver:0.1.1       "/usr/src/app/docker…"   16 minutes ago   Up 16 minutes             0.0.0.0:8000->8000/tcp, :::8000->8000/tcp, 8002/tcp                                                                                                                                                                                                   motion-s3-connector-cloudserver-1
9b1f08bd11ad   ghcr.io/data-preservation-programs/singularity:v0.5.9   "/app/singularity ru…"   16 minutes ago   Up 16 minutes             0.0.0.0:7778->7778/tcp, :::7778->7778/tcp                                                                                                                                                                                                             motion-s3-connector-singularity_content_provider-1
7bec967d7fa3   postgres:15                                             "docker-entrypoint.s…"   16 minutes ago   Up 16 minutes (healthy)   0.0.0.0:5432->5432/tcp, :::5432->5432/tcp                                                                                                                                                                                                             motion-s3-connector-db-1

[Angelo-gh3990]

I assume Motion is not booting or something, this is the error in the logs:

motion-s3-connector-cloudserver-1                   | {"name":"S3","bucketName":"mybucket","objectKey":"README.md","bytesReceived":3703,"bodyLength":3703,"service":"s3","action":"PutObject","accountDisplay
Name":"Bart","accountName":"Bart","implName":"multipleBackends","keyContext":{"bucketName":"mybucket","owner":"79a59df900b949e55d96a1e698fbacedfd6e09d98eacf8f8d5218e7cd47ef2be","namespace":"default","objec
tKey":"README.md","metaHeaders":{},"isDeleteMarker":false},"method":"put","time":1699879266572,"req_id":"06334d64444da16c414f","level":"debug","message":"sending put to datastore","hostname":"c9596434d6aa"
,"pid":66}
motion-s3-connector-cloudserver-1                   | {
motion-s3-connector-cloudserver-1                   |   hostname: 'motion',
motion-s3-connector-cloudserver-1                   |   port: 40080,
motion-s3-connector-cloudserver-1                   |   path: '/v0/blob',
motion-s3-connector-cloudserver-1                   |   method: 'POST',
motion-s3-connector-cloudserver-1                   |   headers: { 'Content-Type': 'application/octet-stream' }
motion-s3-connector-cloudserver-1                   | }
motion-s3-connector-cloudserver-1                   | error on request to motion Error: getaddrinfo EAI_AGAIN motion
motion-s3-connector-cloudserver-1                   |     at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:109:26) {
motion-s3-connector-cloudserver-1                   |   errno: -3001,
motion-s3-connector-cloudserver-1                   |   code: 'EAI_AGAIN',
motion-s3-connector-cloudserver-1                   |   syscall: 'getaddrinfo',
motion-s3-connector-cloudserver-1                   |   hostname: 'motion'
motion-s3-connector-cloudserver-1                   | }

[masih]

Thanks. Looks like only 3 containers out of 8 are running.

I recommend looking at logs from those containers, e.g. docker compose logs | grep motion-s3-connector-singularity_api and so on.

[Angelo-gh3990]

I assumed everything was in the other containers ... I'll see if I can find something. Thanks

[Angelo-gh3990]

@masih any thoughts on:

motion-s3-connector-singularity_deal_pusher-1       | 2023-11-13T13:03:03.033Z  ERROR   database        database/util.go:80     SELECT count(*) FROM "workers" WHERE type = 'deal_pusher' AND last_heartbeat
> '2023-11-13 12:58:03.031'     {"rowsAffected": 0, "elapsed": 0.001014885, "err": "ERROR: relation \"workers\" does not exist (SQLSTATE 42P01)"}
motion-s3-connector-singularity_deal_pusher-1       | 2023-11-13T13:03:03.033Z  ERROR   dealpusher      service/service.go:65   failed to start service {"name": "DealPusher", "error": "failed to register w
orker: failed to count active workers: ERROR: relation \"workers\" does not exist (SQLSTATE 42P01)", "errorVerbose": "failed to register worker: failed to count active workers: ERROR: relation \"workers\"
does not exist (SQLSTATE 42P01)\n(1) attached stack trace\n  -- stack trace:\n  | github.com/data-preservation-programs/singularity/service/dealpusher.(*DealPusher).Start\n  | \t/app/service/dealpusher/dea
lpusher.go:529\n  | github.com/data-preservation-programs/singularity/service.StartServers\n  | \t/app/service/service.go:62\n  | github.com/data-preservation-programs/singularity/cmd/run.glob..func3\n  |
\t/app/cmd/run/dealpusher.go:46\n  | github.com/urfave/cli/v2.(*Command).Run\n  | \t/go/pkg/mod/github.com/urfave/cli/[email protected]/command.go:274\n  | github.com/urfave/cli/v2.(*Command).Run\n  | \t/go/pkg/m
od/github.com/urfave/cli/[email protected]/command.go:267\n  | github.com/urfave/cli/v2.(*Command).Run\n  | \t/go/pkg/mod/github.com/urfave/cli/[email protected]/command.go:267\n  | github.com/urfave/cli/v2.(*App).RunCo
ntext\n  | \t/go/pkg/mod/github.com/urfave/cli/[email protected]/app.go:332\n  | main.main\n  | \t/app/singularity.go:54\n  | runtime.main\n  | \t/usr/local/go/src/runtime/proc.go:250\nWraps: (2) failed to regist
er worker\nWraps: (3) attached stack trace\n  -- stack trace:\n  | github.com/data-preservation-programs/singularity/service/healthcheck.Register\n  | \t/app/service/healthcheck/healthcheck.go:140\n  | [..
.repeated from below...]\nWraps: (4) attached stack trace\n  -- stack trace:\n  | github.com/data-preservation-programs/singularity/service/healthcheck.Register.func1\n  | \t/app/service/healthcheck/health
check.go:129\n  | github.com/avast/retry-go.Do\n  | \t/go/pkg/mod/github.com/avast/[email protected]+incompatible/retry.go:127\n  | github.com/data-preservation-programs/singularity/database.DoRetry\n  | \t/
app/database/util.go:33\n  | github.com/data-preservation-programs/singularity/service/healthcheck.Register\n  | \t/app/service/healthcheck/healthcheck.go:123\n  | github.com/data-preservation-programs/sin
gularity/service/dealpusher.(*DealPusher).Start\n  | \t/app/service/dealpusher/dealpusher.go:527\n  | github.com/data-preservation-programs/singularity/service.StartServers\n  | \t/app/service/service.go:62\n  | github.com/data-preservation-programs/singularity/cmd/run.glob..func3\n  | \t/app/cmd/run/dealpusher.go:46\n  | github.com/urfave/cli/v2.(*Command).Run\n  | \t/go/pkg/mod/github.com/urfave/cli/[email protected]/command.go:274\n  | github.com/urfave/cli/v2.(*Command).Run\n  | \t/go/pkg/mod/github.com/urfave/cli/[email protected]/command.go:267\n  | github.com/urfave/cli/v2.(*Command).Run\n  | \t/go/pkg/mod/github.com/urfave/cli/[email protected]/command.go:267\n  | github.com/urfave/cli/v2.(*App).RunContext\n  | \t/go/pkg/mod/github.com/urfave/cli/[email protected]/app.go:332\n  | main.main\n  | \t/app/singularity.go:54\n  | runtime.main\n  | \t/usr/local/go/src/runtime/proc.go:250\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_amd64.s:1598\nWraps: (5) failed to count active workers\nWraps: (6) ERROR: relation \"workers\" does not exist (SQLSTATE 42P01)\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *withstack.withStack (5) *errutil.withPrefix (6) *pgconn.PgError"}

That's the first error I get when I do a docker compose up

[masih]

Looks like a missing singularity db initialisation issue.

[Angelo-gh3990]

I think I found it, I have another database running on the same port.

[elijaharita]

@Angelo-gh3990 Were you able to fix it? If not, try this branch #15