Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Vault Agent Sidecar With Spring Cloud Vault Not Working #664

Open
ChrisJBurns opened this issue Nov 18, 2022 · 3 comments
Open

Vault Agent Sidecar With Spring Cloud Vault Not Working #664

ChrisJBurns opened this issue Nov 18, 2022 · 3 comments

Comments

@ChrisJBurns
Copy link

ChrisJBurns commented Nov 18, 2022

Describe the bug
Using version 2.2.7.RELEASE of spring-cloud-starter-vault-config and I have got the following bootstrap.yaml

spring:
  cloud:
    vault:
      kv:
        enabled: true
      authentication: NONE

When I deploy it into Kubernetes, with the following annotations on the Deployment

      annotations:
        vault.hashicorp.com/agent-inject: 'true'
        vault.hashicorp.com/role: 'bot-sa'
        vault.hashicorp.com/namespace: "development"
        vault.hashicorp.com/agent-init-first: "true"

I can see that the vault-init starts, then the vault-agent indeed starts and runs. When the Spring App runs, I see the following

{"@timestamp":"2022-11-18T19:17:57.910Z","@version":"1","message":"[RequestedSecret [path='secrets/dev-team/bot', mode=RENEW]] Lease [leaseId='null', leaseDuration=PT0S, renewable=false] I/O error on GET request for \"https://localhost:8200/v1/secrets/dev-team/bot\": Connect to localhost:8200 [localhost/127.0.0

Ignoring the issue that the KV Engine is Version 2 and for some reason isn't recognising it (have raised this here spring-projects/spring-vault#738), but it doesn't seem to be connecting to the Vault Agent. Tbh, I'm not entirely sure how the whole NONE authentication stuff works, because there isn't any actual code examples of this on any docs - apart from just saying to use NONE. Am I misunderstanding the entire thing, or is the above a genuine issue?

Previous to this, we were using Kubernetes authentication to retrieve the secrets, but we want to cut down the code by just using the Vault Agent - but are having the above problems.

To note: There is nothing wrong with the injection of secrets into the main container, so we know that in terms of injecting secrets using the Vault Secret Injection Template works. We just can't figure out why Spring itself can't pull the secrets through the Vault Agent.

@ChrisJBurns
Copy link
Author

Ok, so, it seems to be somewhat working now after spamming as many combinations of the Vault Agent Annotations as possible.

For the record and also other's who are reading this thread, to enable the above to work I had to do 2 things.

The addition of the agent-cache-enable annotation to true. I did try a bunch of others:

vault.hashicorp.com/agent-inject-token: "true"
vault.hashicorp.com/agent-cache-listener-port: "8200"
vault.hashicorp.com/agent-cache-use-auto-auth-token: "force"

But the agent-cache-enable was the only annotation that made any effect.

Secondly, I also noticed that there were SSL problems when connecting to Agent because it was doing a secure request via https://localhost:8200. To stop this, I just added scheme in the bootstrap config, and it ended up looking like:

spring:
  cloud:
    vault:
      kv:
        enabled: true
      scheme: http
      authentication: NONE

Realistically, I should probably set the uri instead to http://localhost:8200 just so it's completely explicit - but am happy that it works for now.

Lastly the annotations used where the following:

kind: Deployment
apiVersion: apps/v1
metadata:
  name: bot-test
  namespace: development
  labels:
    app: bot-test
spec:
  replicas: 1
  selector:
    matchLabels:
      app: bot-test
  template:
    metadata:
      labels:
        app: bot-test
      annotations:
        vault.hashicorp.com/agent-inject: 'true'
        vault.hashicorp.com/role: 'gitbot-sa'
        vault.hashicorp.com/namespace: "development"
        vault.hashicorp.com/agent-init-first: "true"
        vault.hashicorp.com/agent-cache-enable: "true"

I think it's best that we at least add some documentation around this? As I remember trying to do this on other projects and got no luck and have to give up, I had a bit more time this time around to get it working. I'd love to help others reduce time spent finding the above out the hard way if possible?

@ChrisJBurns
Copy link
Author

One last comment. The logs indicate that the KV2 /data/ path isn't being used, but it is quite strange, because it's definitely pulling in secrets from a KV2 engine - not entirely sure what's going on there. Logs for fullness

 {"@timestamp":"2022-11-18T19:59:34.714Z","@version":"1","message":"Located property source: [BootstrapPropertySource {name='bootstrapProperties-Context backend: secrets/dev-team/bot'}]","logger_name":"org.springframework.cloud.bootstrap.config.PropertySourceBootstrapConfiguration","thread_name":"main","level":"I

@patpatpat123
Copy link

Same issue here. Any update?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants