Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

history reports duplicate version numbers for Parameter Store key #491

Open
james-valente-simplisafe opened this issue May 16, 2024 · 3 comments
Open

history reports duplicate version numbers for Parameter Store key #491

james-valente-simplisafe opened this issue May 16, 2024 · 3 comments

Comments

@james-valente-simplisafe
Copy link

james-valente-simplisafe commented May 16, 2024 •
edited
Loading

The output of history reports duplicate version numbers for the same Parameter Store key.

Initially, I created 3 versions of the same key.

$ AWS_REGION=us-east-1 CHAMBER_KMS_KEY_ALIAS=xx/foo/kms-key ./chamber-v2.14.1-linux-amd64 \
    write --verbose foo/foo88 my-key-82 -- my-value-82

$ AWS_REGION=us-east-1 CHAMBER_KMS_KEY_ALIAS=xx/foo/kms-key ./chamber-v2.14.1-linux-amd64 \
    write --verbose foo/foo88 my-key-82 -- my-value-AA

$ AWS_REGION=us-east-1 CHAMBER_KMS_KEY_ALIAS=xx/foo/kms-key ./chamber-v2.14.1-linux-amd64 \
    write --verbose foo/foo88 my-key-82 -- my-value-82

List the history of the versions. Note how the two most recent versions are both reported as version 2.

$ AWS_REGION=us-east-1 ./chamber-v2.14.1-linux-amd64 history foo/foo88 my-key-82

Event		Version		Date			User
Created		1		2024-05-16 18:30:11	arn:aws:sts::123456789012:assumed-role/other-admin/[email protected]
Updated		2		2024-05-16 18:37:00	arn:aws:sts::123456789012:assumed-role/other-admin/[email protected]
Updated		2		2024-05-16 18:37:02	arn:aws:sts::123456789012:assumed-role/other-admin/[email protected]

Version 1 and the initial version 2 can be referenced directly, by number.

$ AWS_REGION=us-east-1 ./chamber-v2.14.1-linux-amd64 read foo/foo88 my-key-82 --version 1

Key		Value		Version		LastModified		User
my-key-82	my-value-82	1		2024-05-16 18:30:11	arn:aws:sts::123456789012:assumed-role/other-admin/[email protected]

$ AWS_REGION=us-east-1 ./chamber-v2.14.1-linux-amd64 read foo/foo88 my-key-82 --version 2

Key		Value		Version		LastModified		User
my-key-82	my-value-AA	2		2024-05-16 18:37:00	arn:aws:sts::123456789012:assumed-role/other-admin/[email protected]

Unfortunately, the third version cannot be referenced by number. However, it can be referenced by relative number (-1 is latest).

$ AWS_REGION=us-east-1 ./chamber-v2.14.1-linux-amd64 read foo/foo88 my-key-82 --version 3

Error: Failed to read: secret not found

$ AWS_REGION=us-east-1 ./chamber-v2.14.1-linux-amd64 read foo/foo88 my-key-82 --version -1

Key		Value		Version		LastModified		User
my-key-82	my-value-82	2		2024-05-16 18:37:02	arn:aws:sts::123456789012:assumed-role/other-admin/[email protected]

Now, create a fourth version of this key.

$ AWS_REGION=us-east-1 CHAMBER_KMS_KEY_ALIAS=xx/foo/kms-key ./chamber-v2.14.1-linux-amd64 \
    write --verbose foo/foo88 my-key-82 -- my-value-BB

The history still reports the two items as having version 2, and the most recent (fourth) version is listed as version 3.

$ AWS_REGION=us-east-1 ./chamber-v2.14.1-linux-amd64 history foo/foo88 my-key-82

Event		Version		Date			User
Created		1		2024-05-16 18:30:11	arn:aws:sts::123456789012:assumed-role/other-admin/[email protected]
Updated		2		2024-05-16 18:37:00	arn:aws:sts::123456789012:assumed-role/other-admin/[email protected]
Updated		2		2024-05-16 18:37:02	arn:aws:sts::123456789012:assumed-role/other-admin/[email protected]
Updated		3		2024-05-16 18:49:58	arn:aws:sts::123456789012:assumed-role/other-admin/[email protected]

However, a screenshot of the AWS console reports distinct versions 1, 2, 3, and 4. Note how the Last modified date column in the AWS console reports the same timestamps as found in the output of the chamber history command above.

image

Note also how the following command fetches version 3 (according to Chamber), but in fact, the timestamp corresponds to Version 4 within the AWS console. As further confirmation, the value corresponds to the fourth version of the key, as modified by the chamber write command earlier in this issue description.

$ AWS_REGION=us-east-1 ./chamber-v2.14.1-linux-amd64 read foo/foo88 my-key-82 --version 3

Key		Value		Version		LastModified		User
my-key-82	my-value-BB	3		2024-05-16 18:49:58	arn:aws:sts::123456789012:assumed-role/other-admin/[email protected]

Therefore, it would seem that Chamber may have a bug in how it tracks/reports version numbers of a Parameter Store key's history.

Although I have not been able to reproduce this, I searched previous Issues in this repo and found issue #308, which seems to report a similar bug.
@james-valente-simplisafe james-valente-simplisafe changed the title "history" reports duplicate version numbers for Parameter Store key history reports duplicate version numbers for Parameter Store key May 16, 2024
@james-valente-simplisafe
Copy link
Author

james-valente-simplisafe commented May 16, 2024 •
edited
Loading

Actually, I am able to reproduce this bug, if I create and update the key/values in quick succession. Note the Date column timestamps relative to the duplicate version 2 reported in history outputs below.

Here, the delay between updates is short (2 or 3 seconds).

$ AWS_REGION=us-east-1 CHAMBER_KMS_KEY_ALIAS=xx/foo/kms-key ./chamber-v2.14.1-linux-amd64 \
    write --verbose foo/foo77 my-key-71 -- my-value-71

$ AWS_REGION=us-east-1 CHAMBER_KMS_KEY_ALIAS=xx/foo/kms-key ./chamber-v2.14.1-linux-amd64 \
    write --verbose foo/foo77 my-key-71 -- my-value-AA

$ AWS_REGION=us-east-1 CHAMBER_KMS_KEY_ALIAS=xx/foo/kms-key ./chamber-v2.14.1-linux-amd64 \
    write --verbose foo/foo77 my-key-71 -- my-value-BB

$ AWS_REGION=us-east-1 CHAMBER_KMS_KEY_ALIAS=xx/foo/kms-key ./chamber-v2.14.1-linux-amd64 \
    write --verbose foo/foo77 my-key-71 -- my-value-CC

$  AWS_REGION=us-east-1 ./chamber-v2.14.1-linux-amd64 history foo/foo77 my-key-71

Event		Version		Date			User
Created		1		2024-05-16 20:24:59	arn:aws:sts::123456789012:assumed-role/other-admin/[email protected]
Updated		2		2024-05-16 20:25:01	arn:aws:sts::123456789012:assumed-role/other-admin/[email protected]
Updated		2		2024-05-16 20:25:04	arn:aws:sts::123456789012:assumed-role/other-admin/[email protected]
Updated		3		2024-05-16 20:25:06	arn:aws:sts::123456789012:assumed-role/other-admin/[email protected]

Another example with short delay between updates (2 seconds). Note how 3 of the 4 versions have the same version 2.

$ AWS_REGION=us-east-1 CHAMBER_KMS_KEY_ALIAS=xx/foo/kms-key ./chamber-v2.14.1-linux-amd64 \
    write --verbose foo/foo77 my-key-74 -- aaaaaaaa-01

$ AWS_REGION=us-east-1 CHAMBER_KMS_KEY_ALIAS=xx/foo/kms-key ./chamber-v2.14.1-linux-amd64 \
    write --verbose foo/foo77 my-key-74 -- aaaaaaaa-02

$ AWS_REGION=us-east-1 CHAMBER_KMS_KEY_ALIAS=xx/foo/kms-key ./chamber-v2.14.1-linux-amd64 \
    write --verbose foo/foo77 my-key-74 -- aaaaaaaa-03

$ AWS_REGION=us-east-1 CHAMBER_KMS_KEY_ALIAS=xx/foo/kms-key ./chamber-v2.14.1-linux-amd64 \
    write --verbose foo/foo77 my-key-74 -- aaaaaaaa-04

$ AWS_REGION=us-east-1 ./chamber-v2.14.1-linux-amd64 history foo/foo77 my-key-74

Event		Version		Date			User
Created		1		2024-05-16 20:42:29	arn:aws:sts::123456789012:assumed-role/other-admin/[email protected]
Updated		2		2024-05-16 20:42:31	arn:aws:sts::123456789012:assumed-role/other-admin/[email protected]
Updated		2		2024-05-16 20:42:33	arn:aws:sts::123456789012:assumed-role/other-admin/[email protected]
Updated		2		2024-05-16 20:42:35	arn:aws:sts::123456789012:assumed-role/other-admin/[email protected]

If the delay between updates is slightly longer (4+ seconds), Chamber reports unique version numbers for each history item.

$ AWS_REGION=us-east-1 CHAMBER_KMS_KEY_ALIAS=xx/foo/kms-key ./chamber-v2.14.1-linux-amd64 \
    write --verbose foo/foo77 my-key-72 -- alpha

$ AWS_REGION=us-east-1 CHAMBER_KMS_KEY_ALIAS=xx/foo/kms-key ./chamber-v2.14.1-linux-amd64 \
    write --verbose foo/foo77 my-key-72 -- bravo

$ AWS_REGION=us-east-1 CHAMBER_KMS_KEY_ALIAS=xx/foo/kms-key ./chamber-v2.14.1-linux-amd64 \
    write --verbose foo/foo77 my-key-72 -- charlie

$ AWS_REGION=us-east-1 CHAMBER_KMS_KEY_ALIAS=xx/foo/kms-key ./chamber-v2.14.1-linux-amd64 \
    write --verbose foo/foo77 my-key-72 -- delta

$ AWS_REGION=us-east-1 ./chamber-v2.14.1-linux-amd64 history foo/foo77 my-key-72

Event		Version		Date			User
Created		1		2024-05-16 20:26:48	arn:aws:sts::123456789012:assumed-role/other-admin/[email protected]
Updated		2		2024-05-16 20:26:52	arn:aws:sts::123456789012:assumed-role/other-admin/[email protected]
Updated		3		2024-05-16 20:26:58	arn:aws:sts::123456789012:assumed-role/other-admin/[email protected]
Updated		4		2024-05-16 20:27:02	arn:aws:sts::123456789012:assumed-role/other-admin/[email protected]

@james-valente-simplisafe
Copy link
Author

Further investigation reveals that Chamber appears to be incorrectly reporting the Description and not the Version of the each Parameter Store key's history.

The history of my-key-71 gets reported as versions 1, 2, 2, 3 but should be 1, 2, 3, 4.

$ aws --region us-east-1 ssm get-parameter-history --with-decryption --name /foo/foo77/my-key-71

{
    "Parameters": [
        {
            "KeyId": "alias/xx/foo/kms-key",
            "Name": "/foo/foo77/my-key-71",
            "DataType": "text",
            "LastModifiedDate": 1715891099.458,
            "Labels": [],
            "Value": "my-value-71",
            "Version": 1,
            "LastModifiedUser": "arn:aws:sts::123456789012:assumed-role/other-admin/[email protected]",
            "Policies": [],
            "Tier": "Standard",
            "Type": "SecureString",
            "Description": "1"
        },
        {
            "KeyId": "alias/xx/foo/kms-key",
            "Name": "/foo/foo77/my-key-71",
            "DataType": "text",
            "LastModifiedDate": 1715891101.833,
            "Labels": [],
            "Value": "my-value-AA",
            "Version": 2,
            "LastModifiedUser": "arn:aws:sts::123456789012:assumed-role/other-admin/[email protected]",
            "Policies": [],
            "Tier": "Standard",
            "Type": "SecureString",
            "Description": "2"
        },
        {
            "KeyId": "alias/xx/foo/kms-key",
            "Name": "/foo/foo77/my-key-71",
            "DataType": "text",
            "LastModifiedDate": 1715891104.183,
            "Labels": [],
            "Value": "my-value-BB",
            "Version": 3,
            "LastModifiedUser": "arn:aws:sts::123456789012:assumed-role/other-admin/[email protected]",
            "Policies": [],
            "Tier": "Standard",
            "Type": "SecureString",
            "Description": "2"
        },
        {
            "KeyId": "alias/xx/foo/kms-key",
            "Name": "/foo/foo77/my-key-71",
            "DataType": "text",
            "LastModifiedDate": 1715891106.422,
            "Labels": [],
            "Value": "my-value-CC",
            "Version": 4,
            "LastModifiedUser": "arn:aws:sts::123456789012:assumed-role/other-admin/[email protected]",
            "Policies": [],
            "Tier": "Standard",
            "Type": "SecureString",
            "Description": "3"
        }
    ]
}

The history of my-key-74 gets reported as versions 1, 2, 2, 2 but should be 1, 2, 3, 4.

$ aws --region us-east-1 ssm get-parameter-history --with-decryption --name /foo/foo77/my-key-74

{
    "Parameters": [
        {
            "KeyId": "alias/xx/foo/kms-key",
            "Name": "/foo/foo77/my-key-74",
            "DataType": "text",
            "LastModifiedDate": 1715892149.853,
            "Labels": [],
            "Value": "aaaaaaaa-01",
            "Version": 1,
            "LastModifiedUser": "arn:aws:sts::123456789012:assumed-role/other-admin/[email protected]",
            "Policies": [],
            "Tier": "Standard",
            "Type": "SecureString",
            "Description": "1"
        },
        {
            "KeyId": "alias/xx/foo/kms-key",
            "Name": "/foo/foo77/my-key-74",
            "DataType": "text",
            "LastModifiedDate": 1715892151.883,
            "Labels": [],
            "Value": "aaaaaaaa-02",
            "Version": 2,
            "LastModifiedUser": "arn:aws:sts::123456789012:assumed-role/other-admin/[email protected]",
            "Policies": [],
            "Tier": "Standard",
            "Type": "SecureString",
            "Description": "2"
        },
        {
            "KeyId": "alias/xx/foo/kms-key",
            "Name": "/foo/foo77/my-key-74",
            "DataType": "text",
            "LastModifiedDate": 1715892153.716,
            "Labels": [],
            "Value": "aaaaaaaa-03",
            "Version": 3,
            "LastModifiedUser": "arn:aws:sts::123456789012:assumed-role/other-admin/[email protected]",
            "Policies": [],
            "Tier": "Standard",
            "Type": "SecureString",
            "Description": "2"
        },
        {
            "KeyId": "alias/xx/foo/kms-key",
            "Name": "/foo/foo77/my-key-74",
            "DataType": "text",
            "LastModifiedDate": 1715892155.541,
            "Labels": [],
            "Value": "aaaaaaaa-04",
            "Version": 4,
            "LastModifiedUser": "arn:aws:sts::123456789012:assumed-role/other-admin/[email protected]",
            "Policies": [],
            "Tier": "Standard",
            "Type": "SecureString",
            "Description": "2"
        }
    ]
}

@james-valente-simplisafe
Copy link
Author

Looking at the source code, I see the chamber write command intentionally stores Chamber's notion of "version" number in the secret's Description field (link).

Additionally, I see that source code was committed on June 12, 2017, which predates the AWS Blog post (link) on October 26, 2017, announcing that Parameter Store now provides native support for unique version numbers in secrets history. Thus, Chamber's use of theDescription field to store Chamber's notion of "version" understandably predates the native support for versioning within AWS Parameter Store.

As for why Chamber is unable to increment the version number in the Description field correctly, I imagine there may be some kind of eventual consistency delay within AWS when a Parameter Store secret is overwritten. So, it may be possible that these lines in source code (link) fetch the initial secret (with Description = 1, such that multiple subsequent overwrites (within a short period of time) are resulting in Description = 2.

That said, the source code (link) in the readVersion function can probably be updated to make use of the AWS native versions, assuming a history.Version field can be accessed instead of the history.Description field. According to the AWS API documentation (link), it should be possible.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@james-valente-simplisafe