Allow reading from and writing to plain text using secretsmanager backend

[nitrocode]

My AWS Secrets Manager secret contains only plain text information so no key-value

key-value works

$ chamber -b secretsmanager list devops/key-value
Key			Version		LastModified		User
<redacted>		3		2022-09-01 12:37:33	arn:<redacted>
<redacted>	        1		2022-09-01 12:38:21	arn:<redacted>

but plain text does not

$ chamber -b secretsmanager list devops/plain-text
Error: Failed to list store contents: invalid character '-' in numeric literal
$ chamber -b secretsmanager read devops/plain-text
<shows help>

Here's the creation using the awscli of both of the types above

https://stackoverflow.com/a/58943565

---

Workaround for the list (or list-services) is to go back to awscli

aws secretsmanager list-secrets --query 'SecretList[].Name'

---

Here is the list function

chamber/store/secretsmanagerstore.go

Lines 365 to 378 in 0fda232

	func (s *SecretsManagerStore) List(serviceName string, includeValues bool) ([]Secret, error) {
	secrets := map[string]Secret{}
	latest, err := s.readLatest(serviceName)
	if err != nil {
	return nil, err
	}
	metadata, err := getHydratedMetadata(&latest)
	if err != nil {
	return nil, err
	}
	for key, value := range latest {

which calls readLatest

chamber/store/secretsmanagerstore.go

Lines 334 to 354 in 0fda232

	func (s *SecretsManagerStore) readLatest(service string) (secretValueObject, error) {
	getSecretValueInput := &secretsmanager.GetSecretValueInput{
	SecretId: aws.String(service),
	}
	resp, err := s.svc.GetSecretValue(getSecretValueInput)
	if err != nil {
	return secretValueObject{}, err
	}
	if len(*resp.SecretString) == 0 {
	return secretValueObject{}, ErrSecretNotFound
	}
	var obj secretValueObject
	if obj, err = jsonToSecretValueObject(*resp.SecretString); err != nil {
	return secretValueObject{}, err
	}
	return obj, nil

which calls jsonToSecretValueObject

chamber/store/secretsmanagerstore.go

Lines 556 to 562 in 0fda232

	func jsonToSecretValueObject(s string) (secretValueObject, error) {
	var obj secretValueObject
	err := json.Unmarshal([]byte(s), &obj)
	if err != nil {
	return secretValueObject{}, err
	}
	return obj, nil

which tries to do a json.Unmarshal and fails

---

Here is the read function

chamber/store/secretsmanagerstore.go

Lines 229 to 239 in 0fda232

	func (s *SecretsManagerStore) Read(id SecretId, version int) (Secret, error) {
	if version == -1 {
	latest, err := s.readLatest(id.Service)
	if err != nil {
	return Secret{}, err
	}
	value, ok := latest[id.Key]
	if !ok {
	return Secret{}, ErrSecretNotFound
	}

[bhavanki]

I haven't been involved in this project previously, but would like to respond. (I'm a Segment employee.)

It does look like chamber requires secret values in AWS Secrets Manager to be a specific JSON structure, containing keys and values and other expected metadata. So, a secret value that's plain text - say, something like "foo" or even "foo=bar" - just isn't supported. I don't see a way to support plain text in the secret value and retain parity between Secrets Manager and other backends.

If the problem is writing a plain text value into a secret that can be read by chamber, then the chamber write command should do the trick, as opposed to AWS CLI usage. You will still need to provide a service name and key name, but the value can be just about any string. (The service name becomes the name / ID of the secret in Secrets Manager, but that's a detail of the current implementation.)

chamber -b secretsmanager write my-service key1 plaintext

If I'm missing the point of the problem, please respond with more details. Thanks!

[nitrocode]

Thanks for commenting.

I suppose writing plain text works (I haven't tried and I will) but reading plain text doesn't work and reading isn't possible for this backend and have it work with other backends. Thanks for confirming.

If we can leave this open for now it would be nice if there are other ideas. There must be a way to do it

[bhavanki]

If we can leave this open for now it would be nice if there are other ideas. There must be a way to do it

Sure, let's leave this open until we work it out! Digging deeper:

In your original example, there are two service arguments used: devops/key-value and devops/plain-text. I assume that these are intended to be two different services with those exact names, and not just one "devops" service.

I think the problem may be down to how chamber expects to unmarshal the secret string value as JSON. (In secretsmanagerstore.go, it's the readLatest and jsonToSecretValueObject functions.) If the string is just ordinary text, the unmarshaling fails. I tried this myself with a secret value of "this-is-just-plain-text":

$ chamber -b secretsmanager list plaintext
readLatest: invalid character 'h' in literal true (expecting 'r')
Error: Failed to list store contents: invalid character 'h' in literal true (expecting 'r')

When I change the raw string to {"value":"this-is-just-plain-text"}, chamber is happy.

$ ./chamber -b secretsmanager list -e plaintext
Key	Version		LastModified		User	Value
value	0		0000-12-31 19:03:58		this-is-just-plain-text

So, at the moment, by design, chamber requires a JSON structure. It can deal with a flat map structure that wasn't originally written by chamber, in order to consume values written by the AWS console. Even with that extended support, though, there's no chamber metadata available, so stuff like history isn't available.

With some code changes, I guess chamber could be able to read an arbitrary string in as just a single, unnamed secret. However, one problem with that is that chamber wouldn't be able to tell if it's reading an arbitrary string or malformed JSON. Maybe a command line option could give it a hint? Regardless, it seems safer to update whatever writes the secret either use chamber write or write a basic JSON map that gives the secret some arbitrary key name.

Thoughts?

[nitrocode]

Thanks for diving in again.

The issue only comes up when reading plain-text secrets that weren't originally saved by chamber. It's true, it's possible to overwrite the secret with a json value, however that would also require updating service code or infrastructure to ensure it understands the secret and its new structure.

If you have 1000s of plain-text secrets, now you're talking about a very long migration plan and for no business value so in the end, the awscli would then be more preferable to use. 😭