Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

It doesn't work in EKS #4

Open
dominicicerse opened this issue Jun 6, 2019 · 4 comments
Open

It doesn't work in EKS #4

dominicicerse opened this issue Jun 6, 2019 · 4 comments

Comments

@dominicicerse
Copy link

the 'custom-metrics-apiserver' cannot run in EKS:
$ kubectl get po -n monitoring
NAME READY STATUS RESTARTS AGE
custom-metrics-apiserver-55c757b95d-x5zdg 0/1 CrashLoopBackOff 6 9m51s
prometheus-8669f7f7d8-slmqq 1/1 Running 0 11m

I checked the log. It has the following error error:
Error: cluster doesn't provide client-ca-file

Just past all below:
$ kubectl logs custom-metrics-apiserver-55c757b95d-x5zdg -n monitoring
I0606 10:15:51.477908 1 round_trippers.go:417] curl -k -v -XGET -H "Accept: application/json, /" -H "User-Agent: adapter/v0.0.0 (linux/amd64) kubernetes/$Format" -H "Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJtb25pdG9yaW5nIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImN1c3RvbS1tZXRyaWNzLWFwaXNlcnZlci10b2tlbi1oNXZjYiIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJjdXN0b20tbWV0cmljcy1hcGlzZXJ2ZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiJhNWU0YjQxZC04ODQzLTExZTktODVmMS0wMjdhMzdkNDczMWUiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6bW9uaXRvcmluZzpjdXN0b20tbWV0cmljcy1hcGlzZXJ2ZXIifQ.qzcldE0DaAoaROLHg-mfWY5BegiN9YQwH4qcJGj3-wjWCt9NN3QECLRfdizbFCnglBmc5OmZw8bPc3n7mODBKLRIzqfwryh9RXfVD-heNuB1SnFYp3aTOyDM81k02m2eMcMxULtsfPi1OzyIx_TXqBdxvNiAWgSE9zdtKT9j_DQq2GHAqPLRiSDFI4HepDleyO0CXBGI0xvFiCVuy8J5o7D1aSHlQ232YE4w9ZaB4NgERIRwrQzHZbvGrk6pQn-PEvKFqdHICtR1f8Fgwi3LJtD3NGiichR4zXJNNCtdqxZRM8gJNG7aT-gF_sKH9iYL36EmVef0BRLu4FqSlrBQGw" https://172.20.0.1:443/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication
I0606 10:15:51.494442 1 round_trippers.go:436] GET https://172.20.0.1:443/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication 200 OK in 16 milliseconds
I0606 10:15:51.494464 1 round_trippers.go:442] Response Headers:
I0606 10:15:51.494469 1 round_trippers.go:445] Audit-Id: 277f9a0b-bdbe-429f-a08c-2e1e8b6e9231
I0606 10:15:51.494473 1 round_trippers.go:445] Content-Type: application/json
I0606 10:15:51.494478 1 round_trippers.go:445] Content-Length: 1648
I0606 10:15:51.494482 1 round_trippers.go:445] Date: Thu, 06 Jun 2019 10:15:51 GMT
I0606 10:15:51.494518 1 request.go:836] Response Body: {"kind":"ConfigMap","apiVersion":"v1","metadata":{"name":"extension-apiserver-authentication","namespace":"kube-system","selfLink":"/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication","uid":"2e07486e-836e-11e9-85f1-027a37d4731e","resourceVersion":"56","creationTimestamp":"2019-05-31T06:34:45Z"},"data":{"requestheader-allowed-names":"["front-proxy-client"]","requestheader-client-ca-file":"-----BEGIN CERTIFICATE-----\nMIIC0DCCAbigAwIBAgIBADANBgkqhkiG9w0BAQsFADAZMRcwFQYDVQQDEw5mcm9u\ndC1wcm94eS1jYTAeFw0xOTA1MzEwNjM0MDNaFw0yOTA1MjgwNjM0MDNaMBkxFzAV\nBgNVBAMTDmZyb250LXByb3h5LWNhMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIB\nCgKCAQEAtpd8oUCa2DYD+O1LSRlaePTLAbrUzdwkxyAssPKNTZgAA3uau4m7HDAW\nAhzpD3JvCohFKTBQ5Dz9kDKesxlQJ9sUVVECuBtskvpNy1GWJPXkDOYRyVDqPmfo\nWDyyGLJwcl34Ztzya5ybcG0nrXXEUElZ+CHvA3jHG30GLAF6kMJHkSFiT943zQml\nnxAsLsb0PYPtrACjYmh3B20Tn3NVuoQ2xPe2lmgjgzY45Ebj9ezjEv0Qli3sdsGC\nW5L+3f/y40vogjSHbgTnhBhedzNhJYco/WFDWxJBvawkKuBuhAhVA3WYxySRffPu\nEdSWK2wqL56cn9vLY0CBU+ZHsoVm1QIDAQABoyMwITAOBgNVHQ8BAf8EBAMCAqQw\nDwYDVR0TAQH/BAUwAwEB/zANBgkqhkiG9w0BAQsFAAOCAQEAL25JNmj4gvczywkS\nMQ97URsfAYjMCne7TMYxe01bfL+uFnXBRu4DeMA9Q+NQyYmneBiuKJRvhuP0l9U+\nTUWPpiFUBqljQI3azpd0A2E7RWLwUuhcMY37vQxpA+n2D3bfJwx4muQg1RmigX22\n4+tUsmNyzVVAj8H/knk/4oTls5VNSXhMVJ3qyHGmKjUWa+deHHiPT3mBY0S4SP35\nIqL42alcGEju3Vmak4cih848FbgFPaKv2SGpDsaCJNsKWTEs+utAmVuyaCY/iuNj\nyG4wNHpV13GYQ1FLefjCB2uqdAzFD+Wdow6ZvtZXSscnexcmRUNTk+i4j69dtfG4\ntDc0IA==\n-----END CERTIFICATE-----\n","requestheader-extra-headers-prefix":"["X-Remote-Extra-"]","requestheader-group-headers":"["X-Remote-Group"]","requestheader-username-headers":"["X-Remote-User"]"}}
Error: cluster doesn't provide client-ca-file
Usage:
[flags]

Flags:
--alsologtostderr log to standard error as well as files
--authentication-kubeconfig string kubeconfig file pointing at the 'core' kubernetes server with enough rights to create tokenaccessreviews.authentication.k8s.io.
--authentication-skip-lookup If false, the authentication-kubeconfig will be used to lookup missing authentication configuration from the cluster.
--authentication-token-webhook-cache-ttl duration The duration to cache responses from the webhook token authenticator. (default 10s)
--authorization-kubeconfig string kubeconfig file pointing at the 'core' kubernetes server with enough rights to create subjectaccessreviews.authorization.k8s.io.
--authorization-webhook-cache-authorized-ttl duration The duration to cache 'authorized' responses from the webhook authorizer. (default 10s)
--authorization-webhook-cache-unauthorized-ttl duration The duration to cache 'unauthorized' responses from the webhook authorizer. (default 10s)
--bind-address ip The IP address on which to listen for the --secure-port port. The associated interface(s) must be reachable by the rest of the cluster, and by CLI/web clients. If blank, all interfaces will be used (0.0.0.0). (default 0.0.0.0)
--cert-dir string The directory where the TLS certs are located. If --tls-cert-file and --tls-private-key-file are provided, this flag will be ignored. (default "apiserver.local.config/certificates")
--client-ca-file string If set, any request presenting a client certificate signed by one of the authorities in the client-ca-file is authenticated with an identity corresponding to the CommonName of the client certificate.
--contention-profiling Enable lock contention profiling, if profiling is enabled
--discovery-interval duration interval at which to refresh API discovery information (default 10m0s)
--enable-swagger-ui Enables swagger ui on the apiserver at /swagger-ui
--lister-kubeconfig string kubeconfig file pointing at the 'core' kubernetes server with enough rights to list any described objets
--log-flush-frequency duration Maximum number of seconds between log flushes (default 5s)
--log_backtrace_at traceLocation when logging hits line file:N, emit a stack trace (default :0)
--log_dir string If non-empty, write log files in this directory
--logtostderr log to standard error instead of files (default true)
--metrics-relist-interval duration interval at which to re-list the set of all available metrics from Prometheus (default 10m0s)
--profiling Enable profiling via web interface host:port/debug/pprof/ (default true)
--prometheus-url string URL and configuration for connecting to Prometheus. Query parameters are used to configure the connection (default "https://localhost")
--rate-interval duration period of time used to calculate rate metrics from cumulative metrics (default 5m0s)
--requestheader-allowed-names stringSlice List of client certificate common names to allow to provide usernames in headers specified by --requestheader-username-headers. If empty, any client certificate validated by the authorities in --requestheader-client-ca-file is allowed.
--requestheader-client-ca-file string Root certificate bundle to use to verify client certificates on incoming requests before trusting usernames in headers specified by --requestheader-username-headers
--requestheader-extra-headers-prefix stringSlice List of request header prefixes to inspect. X-Remote-Extra- is suggested. (default [x-remote-extra-])
--requestheader-group-headers stringSlice List of request headers to inspect for groups. X-Remote-Group is suggested. (default [x-remote-group])
--requestheader-username-headers stringSlice List of request headers to inspect for usernames. X-Remote-User is common. (default [x-remote-user])
--secure-port int The port on which to serve HTTPS with authentication and authorization. If 0, don't serve HTTPS at all. (default 443)
--stderrthreshold severity logs at or above this threshold go to stderr (default 2)
--tls-ca-file string If set, this certificate authority will used for secure access from Admission Controllers. This must be a valid PEM-encoded CA bundle. Altneratively, the certificate authority can be appended to the certificate provided by --tls-cert-file.
--tls-cert-file string File containing the default x509 Certificate for HTTPS. (CA cert, if any, concatenated after server cert). If HTTPS serving is enabled, and --tls-cert-file and --tls-private-key-file are not provided, a self-signed certificate and key are generated for the public address and saved to /var/run/kubernetes.
--tls-private-key-file string File containing the default x509 private key matching --tls-cert-file.
--tls-sni-cert-key namedCertKey A pair of x509 certificate and private key file paths, optionally suffixed with a list of domain patterns which are fully qualified domain names, possibly with prefixed wildcard segments. If no domain patterns are provided, the names of the certificate are extracted. Non-wildcard matches trump over wildcard matches, explicit domain patterns trump over extracted names. For multiple key/certificate pairs, use the --tls-sni-cert-key multiple times. Examples: "example.crt,example.key" or "foo.crt,foo.key:*.foo.com,foo.com". (default [])
-v, --v Level log level for V logs (default 0)
--vmodule moduleSpec comma-separated list of pattern=N settings for file-filtered logging

panic: cluster doesn't provide client-ca-file

goroutine 1 [running]:
main.main()
/go/src/github.com/directxman12/k8s-prometheus-adapter/cmd/adapter/adapter.go:41 +0x114

@dominicicerse
Copy link
Author

Version:
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.7", GitCommit:"6f482974b76db3f1e0f5d24605a9d1d38fad9a2b", GitTreeState:"clean", BuildDate:"2019-03-29T16:15:10Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"12+", GitVersion:"v1.12.6-eks-d69f1b", GitCommit:"d69f1bf3669bf00b7f4a758e978e0e7a1e3a68f7", GitTreeState:"clean", BuildDate:"2019-02-28T20:26:10Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"}

@danielepolencic
Copy link
Contributor

The official docs suggest installing a specific version of the custom metric server. https://docs.aws.amazon.com/eks/latest/userguide/metrics-server.html

Which version are you using?

@dominicicerse
Copy link
Author

It should be 0.3.2

Pod Template:
Labels: k8s-app=metrics-server
Service Account: metrics-server
Containers:
metrics-server:
Image: k8s.gcr.io/metrics-server-amd64:v0.3.2

@yrsurya
Copy link

yrsurya commented Mar 21, 2020

Any way we can use for EKS , when I tried to setup hpa it spinning up max pods( which is set to 10) while initializing as memory is high

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants