Concurrent creation of two users fails, while FLOW_WALLET_ADMIN_PROPOSAL_KEY_COUNT is set

[uhbif19]

If more than one user is created, all but one transaction fails.

[uhbif19]

Admin account actually has 30 keys:

https://flow-view-source.com/testnet/account/0xeac6eb45fad808df

But wallet api produces error:

[JOBS] 2021/12/14 18:24:10 workerpool.go:239: WARNING: Job(id: "968d5bb6-df22-4444-b8d0-1a2fdea8453a", type: "account_create") execution resulted with error: [Error Code: 1007] invalid proposal key: public key 0 on account eac6eb45fad808df ...

This bug is on both mainnet and testnet.

[latenssi]

Thank you @uhbif19 for posting this! We will look into it.

[uhbif19]

UPD It works okay on mainnet with 0.7 version, but gives error on testnet 0.8 version.

[latenssi]

@uhbif19 I am unable reproduce this. Do you have FLOW_WALLET_ADMIN_PROPOSAL_KEY_COUNT set to 30? (it is not well documented yet so I understand if you don't have it set)

Here is a PR testing this: #236

[uhbif19]

@latenssi Yes it is set to 30.

[latenssi]

@uhbif19 Ok, thanks! Are you running multiple instances of the wallet service with the same settings?

I assume you are not using the sync way of calling the API when creating the accounts (as the error log had a job id in it)?

[seitau]

I've encountered the same error with v0.8.0 but not in the latest main branch.

[uhbif19]

@latenssi

We using one instance per one depolyment and not using sync.

[latenssi]

@uhbif19 Couple of important questions I should have asked first: is this happening every time you try to create multiple accounts and how are you creating multiple accounts concurrently?

[uhbif19]

@uhbif19 Looks like it happens every time. We POSTing /accounts to create account.

[latenssi]

@uhbif19 Do the accounts eventually get created? The wallet should retry failed attempts (max 10 retries).

[uhbif19]

@latenssi How does this work? We do not cover retrying in our code.

[uhbif19]

@latenssi Is there any way to disable retrying.

[latenssi]

How does this work (retrying)? We do not cover retrying in our code.

The service will retry failed jobs (transactions) by itself (you don't have to do anything). You can set a webhook to receive updates on jobs so you don't have to poll for the results: https://github.com/flow-hydraulics/flow-wallet-api#updates-on-async-requests-webhook

Is there any way to disable retrying.

No, this is a necessary part of the functionality of the service. I am curious tho', why would you want to disable this?

Regarding the initial issue: I have discovered a bug regarding concurrent database calls and the way workers are started.

Temporary fix: Set FLOW_WALLET_WORKER_COUNT to 1 (defaults to 100). This will slow down processing somewhat depending on you workload (it might even make it faster as there are fewer errors).

[sheerryy]

@latenssi I encountered this error while performing a batch (more than 100) of transactions (not account creation). Any insight on why this error is produced?

{
        "jobId": "bc6c4485-6e82-4100-bbee-6db07e187259",
        "type": "transaction",
        "state": "ERROR",
        "error": "client: rpc error: code = InvalidArgument desc = invalid transaction: transaction is expired: ref_height=32554913 final_height=32557064",
        "errors": [
            "[Error Code: 1006] invalid proposal key: public key 1 on account 69fa357a1fe69201 does not have a valid signature: [Error Code: 1008] invalid payload signature: public key 1 on account 69fa357a1fe69201 does not have a valid signature: signature is not valid",
            "client: rpc error: code = InvalidArgument desc = invalid transaction: transaction is expired: ref_height=32554913 final_height=32555707",
            "client: rpc error: code = InvalidArgument desc = invalid transaction: transaction is expired: ref_height=32554913 final_height=32555907",
            "client: rpc error: code = InvalidArgument desc = invalid transaction: transaction is expired: ref_height=32554913 final_height=32556031",
            "client: rpc error: code = InvalidArgument desc = invalid transaction: transaction is expired: ref_height=32554913 final_height=32556222",
            "client: rpc error: code = InvalidArgument desc = invalid transaction: transaction is expired: ref_height=32554913 final_height=32556403",
            "client: rpc error: code = InvalidArgument desc = invalid transaction: transaction is expired: ref_height=32554913 final_height=32556680",
            "client: rpc error: code = InvalidArgument desc = invalid transaction: transaction is expired: ref_height=32554913 final_height=32556762",
            "client: rpc error: code = InvalidArgument desc = invalid transaction: transaction is expired: ref_height=32554913 final_height=32556940",
            "client: rpc error: code = InvalidArgument desc = invalid transaction: transaction is expired: ref_height=32554913 final_height=32557064"
        ],
        "result": "",
        "transactionId": "907437d08a34dc880beec705a636b9702cd7d642d74f5e5818639ea47ef4ed14",
        "createdAt": "2022-07-01T22:57:02.399377Z",
        "updatedAt": "2022-07-01T23:42:56.657504Z"
    }

[sheerryy]

@latenssi I reduced the batch size to 50 but still facing the above issue. Can you please help here?

[nvdtf]

Is this still happening?
I had a PoC on testnet that I could create lots of accounts without errors:

#!/bin/bash

for ((i=1; i<=100; i++)); do
curl -X POST "http://localhost:3000/v1/accounts" -H "Idempotency-Key: $i"; done