Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement Secret storage backend as gRPC server #644

Merged
merged 10 commits into from
Feb 28, 2022

Conversation

pkosiec
Copy link
Member

@pkosiec pkosiec commented Feb 24, 2022

Description

Changes proposed in this pull request:

  • Add Protocol Buffers schema
  • Generate gRPC client + server code from Protocol Buffers
  • Implement gRPC server for secret storage backend
    • Support initially just AWS Secrets Manager. Tested also dotenv provider. The vault provider needs to be tested
    • Add tests for the gRPC server with fake secret provider

TODO

Notes

Lint failing because of timeout. Don't worry about it - The job will be green after review 🙂

Testing

To regenerate the gRPC code, run:

SKIP_DEPS_INSTALLATION=false make gen-grpc-resources

To run example calls against the gRPC service:

  1. Create AWS security credentials with SecretsManagerReadWrite policy.

  2. Export environment variables:

     export AWS_ACCESS_KEY_ID="{accessKey}"
     export AWS_SECRET_ACCESS_KEY="{secretKey}"
  3. Run the server:

    APP_SUPPORTED_PROVIDERS=dotenv,aws_secretsmanager APP_LOGGER_DEV_MODE=true go run ./cmd/secret-storage-backend/main.go
  4. Now let's use Go generated gRPC client to to see how the server behaves. Run the example pkg/hub/api/grpc/storage_backend/example_test.go.

    To run it, you can e.g. change the func ExampleNewStorageBackendClient() { to func TestE2EScenario(t *testing.T) { and run it as an usual test.

  5. Change the line 12 of this test and run it again:

    provider := "aws_secretsmanager"

Related issue(s)

Resolves #637

@pkosiec pkosiec added enhancement New feature or request WIP Work in progress area/hub Relates to Hub labels Feb 24, 2022
@pkosiec pkosiec changed the title Implement Secret storage backend Implement Secret storage backend as gRPC server Feb 24, 2022
@pkosiec pkosiec marked this pull request as ready for review February 24, 2022 13:41
go.sum Show resolved Hide resolved
go.sum Show resolved Hide resolved
@pkosiec pkosiec removed the WIP Work in progress label Feb 24, 2022
@pkosiec pkosiec added the WIP Work in progress label Feb 24, 2022
Copy link
Member

@mszostok mszostok left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I really enjoyed reviewing your PR, really nice work 👍

cmd/secret-storage-backend/README.md Outdated Show resolved Hide resolved
cmd/secret-storage-backend/README.md Show resolved Hide resolved
cmd/secret-storage-backend/README.md Outdated Show resolved Hide resolved
cmd/secret-storage-backend/README.md Show resolved Hide resolved
cmd/secret-storage-backend/main.go Outdated Show resolved Hide resolved
internal/secret-storage-backend/server_test.go Outdated Show resolved Hide resolved
internal/secret-storage-backend/server_test.go Outdated Show resolved Hide resolved
pkg/hub/api/grpc/storage_backend.proto Outdated Show resolved Hide resolved
pkg/hub/api/grpc/storage_backend/example_test.go Outdated Show resolved Hide resolved
internal/secret-storage-backend/server.go Outdated Show resolved Hide resolved
Copy link
Member

@mszostok mszostok left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tutti bueno 😄 🚀

italian-hand-fancy

pkg/hub/api/grpc/storage_backend/client_test.go Outdated Show resolved Hide resolved
@@ -0,0 +1,315 @@
package secretstoragebackend
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably even earlier, as using the dotenvwill be useful in e2e tests, so we will need to add some padding to other task to figure this out or you can create a follow-up task to enable the Secret Store storage "consumption" by Local Hub.

pkg/hub/api/grpc/storage_backend/client_test.go Outdated Show resolved Hide resolved
@pkosiec
Copy link
Member Author

pkosiec commented Feb 28, 2022

Integration tests failed during Capact iunstallation, but the previous run was successful (https://github.com/capactio/capact/actions/runs/1900095690) - and also this PR doesn't change anything related to integration tests, so I'm merging this without retry.

@pkosiec pkosiec merged commit f51af88 into capactio:main Feb 28, 2022
@pkosiec pkosiec deleted the secret-storage-backend branch February 28, 2022 08:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/hub Relates to Hub enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Implement AWS Secrets Manager storage backend logic
2 participants