Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improved message when not authorised to list namespaces #71

Open
vgibilmanno opened this issue Nov 27, 2019 · 63 comments
Open

Improved message when not authorised to list namespaces #71

vgibilmanno opened this issue Nov 27, 2019 · 63 comments

Comments

@vgibilmanno
Copy link

vgibilmanno commented Nov 27, 2019

I can't use kubebox when I only have access to a namespace in a cluster. It would be awesome if I could use kubebox just for the namespace I have access to, ignoring everything outside the namespace.

@astefanutti
Copy link
Owner

You should be able to use Kubebox just for the namespaces you're granted access to. What error do you face?

@vgibilmanno
Copy link
Author

The login fails with "Authentication failed" and I'm being asked for an username and a password. The cluster runs in AWS and I have the AWS credentials set. Because of that I don't really have a username or password to set. Kubectl can be executed perfectly fine.

@astefanutti
Copy link
Owner

astefanutti commented Nov 27, 2019

AWS EKS is supposed to be working :(

There have been some issues that may provide some context: #32 #60.

What version of Kubebox do you use? Could you provide the content of your kubeconfig file?

In the meantime, you can run:

$ aws eks get-token --cluster-name [my-cluster] 

To get a token to authenticate with Kubebox.

@vgibilmanno
Copy link
Author

Of course :) that's the content of the kubeconfig. I hope it's ok, that I replaced personal information with placeholders. But I didn't change the structure of the content.

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: DATA+OMITTED
    server: https://OMITTED.amazonaws.com
  name: SOMENAME
contexts:
- context:
    cluster: SOMENAME
    user: SOMENAME
  name: SOMENAME
current-context: SOMENAME
kind: Config
preferences: {}
users:
- name: SOMENAME
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1alpha1
      args:
      - token
      - -i
      - SOMEOTHERNAME
      - -r
      - arn:aws:iam::SOMEROLEID:role/SOMEROLE
      command: aws-iam-authenticator
      env: null

I'm using kubebox v0.6.1

How can I replace the token in the login screen? I'm too stupid right now sry

@astefanutti
Copy link
Owner

Thanks for the details. Nothing obvious so we'll have to try reproducing.

There is a token field in the login window, below the username and password fields. You can paste the token retrieve from AWS CLI.

@astefanutti astefanutti changed the title Using kubebox with cluster restrictions Authentication failed with AWS Nov 27, 2019
@johnpoth
Copy link
Collaborator

@vgibilmanno perhaps some useful info would be to run the command locally yourself:

aws-iam-authenticator token -i SOMEOTHERNAME -r arn:aws:iam::SOMEROLEID:role/SOMEROLE and maybe show us the output ? Of course censuring private data. Thanks !

@vgibilmanno
Copy link
Author

@johnpoth I get the following

{
    "kind": "ExecCredential",
    "apiVersion": "client.authentication.k8s.io/v1alpha1",
    "spec": {},
    "status": {
        "expirationTimestamp": "2019-11-27T15:51:10Z",
        "token": "SOMETOKEN"
    }
}

@astefanutti
Copy link
Owner

@johnpoth could it be that the provideAuth method does not get a first token with refresh_token when there is no token information?

@astefanutti
Copy link
Owner

@johnpoth the refresh_token method returns false when there is no expiration date:

console.log((undefined - Date.now()) < 5000)
> false

So we may have to add a condition to check whether a token exists:

if (this.has_token_expired()) {

WDYT?

@johnpoth
Copy link
Collaborator

johnpoth commented Nov 27, 2019

Hi @astefanutti so the expired is always set to now:

ExecAuthProvider ... this.expiry = Date.now();

So the token is always refreshed even if it's still valid on the first run. Therefore I think the problem lies elsewhere but I may be wrong...

@vgibilmanno have you tried pasting the TOKEN you got from running the command into the kubebox login screen ? This should help narrow things down a bit... thanks

@astefanutti
Copy link
Owner

@johnpoth I've just realised the expiry is set there:

this.expiry = Date.now();

It seems it'd be easier to reason about if it would not be initialised... Could it be the login sequence takes less than 5ms so that the check fails?

@vgibilmanno
Copy link
Author

@johnpoth I can't paste the token in the login screen. The paste command is not working and I would have to type in the whole token manually. I'm using alpine linux v3.10 in case that's relevant.

@johnpoth
Copy link
Collaborator

@astefanutti I hadn't thought about that... judging from the feedback #60 I'm thinking it might be something different ?
That being said I don't see what could be causing this from the Kubebox Exec plugin perspective...

It may be interesting for @vgibilmanno to use the Token directly and see if Kubebox can connect to the cluster ?

@astefanutti
Copy link
Owner

astefanutti commented Nov 27, 2019

@vgibilmanno could you update your kubeconfig manually and add a token field with the copied token returned by AWS CLI, e.g.:

users:
- name: SOMENAME
  user:
    token: PASTE THERE

Then run Kubebox? The token should be picked-up by Kubebox.

@vgibilmanno
Copy link
Author

@astefanutti Doesn't work. The token displayed in the login screen is not the same as I entered in the kubeconfig. My config file is located under /root/.kube/config

@astefanutti
Copy link
Owner

@vgibilmanno the token may just be truncated in the UI. Have you changed the KUBECONFIG environment variable or does you /root/.kube/config file contain multiple contexts?

@astefanutti
Copy link
Owner

@vgibilmanno Can you also just try a second time connecting, by just pressing enter when the login widget displays with the error message?

@vgibilmanno
Copy link
Author

vgibilmanno commented Nov 28, 2019

@astefanutti I tried, but it doesn't work. I'm getting the "authentication failed" message. The token doesn't seem to be the one I define in the kubeconfig. I looked for some substring (last 5 characters) but it doesn't match

@astefanutti
Copy link
Owner

@vgibilmanno thanks for the feedback. Would you mind checking you have only one context defined in your kubeconfig file?

@vgibilmanno
Copy link
Author

@astefanutti There is only one context defined. It's exactly like the one I posted before with the addition, that tere is a token value inside the user field.

@astefanutti
Copy link
Owner

@vgibilmanno thanks. May I ask, is there a token filled in the login widget when using the original kubeconfig, before adding the token manually to it? Finally could you double check the cluster URL displayed in the login widget is the correct one?

@vgibilmanno
Copy link
Author

@astefanutti When I use the original kubeconfig, there is a token filled in the login widget. The cluster URL displayed in the login widget is correct.

@astefanutti
Copy link
Owner

@vgibilmanno thanks again. Just one last test if you don't mind (sorry). Could you re-try the same test of adding the token manually to your kubeconfig file AND removing the whole exec section? Then check the token is the same as the one displayed in the login widget?

@vgibilmanno
Copy link
Author

@astefanutti Now the correct token is displayed, but the authentication fails. I tried pressing enter/log-in multiple times in series

@astefanutti
Copy link
Owner

OK thanks a lot! So there is a request Kubebox does and that's not authorised for your user. Let me check and come back to you. Thanks again!

@astefanutti
Copy link
Owner

@vgibilmanno if you could try executing the following commands and provide the results:

$ curl -k -v -H "Authorization: Bearer <TOKEN>" https://OMITTED.amazonaws.com/

$ curl -k -v -H "Authorization: Bearer <TOKEN>" https://OMITTED.amazonaws.com/api

$ curl -k -v -H "Authorization: Bearer <TOKEN>" https://OMITTED.amazonaws.com/api/v1/namespaces

@vgibilmanno
Copy link
Author

@astefanutti All 3 requests fail with the following message

*   Trying SOMEIP:443...
* TCP_NODELAY set
* Connected to SOMESERVER.amazonaws.com (SOMEIP) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Request CERT (13):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=kube-apiserver
*  start date: Jan 23 14:39:32 2019 GMT
*  expire date: Nov 17 23:02:45 2020 GMT
*  issuer: CN=kubernetes
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x56318da783e0)
> GET / HTTP/2
> Host: SOMEHOST.amazonaws.com
> User-Agent: curl/7.66.0
> Accept: */*
> Authorization: Bearer SOMETOKEN
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
< HTTP/2 401
< audit-id: 6562ed5f-ef44-4d1b-8210-5c98f0c3cea5
< content-type: application/json
< content-length: 165
< date: Thu, 28 Nov 2019 17:23:36 GMT
<
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {

  },
  "status": "Failure",
  "message": "Unauthorized",
  "reason": "Unauthorized",
  "code": 401
* Connection #0 to host SOMEHOST.amazonaws.com left intact

@astefanutti
Copy link
Owner

@vgibilmanno thanks, just to be sure, could you try with a fresh token that you've just retrieve with aws-iam-authenticator token -i SOMEOTHERNAME -r arn:aws:iam::SOMEROLEID:role/SOMEROLE?

@astefanutti
Copy link
Owner

astefanutti commented Nov 29, 2019

@johnpoth I see the Authorization header set when I run the command locally. Not sure about how the CA data could interfere.

@astefanutti
Copy link
Owner

astefanutti commented Nov 29, 2019

401 means that the provided token cannot be authenticated to a known identity. It's like the token used is not correct. Maybe there is an encoding issue when Kubebox reads from stdout and parse the response returned from the executed AWS CLI command.

@astefanutti
Copy link
Owner

@vgibilmanno by any chance, would you be able to clone this repository and run Kubebox from source?

@vgibilmanno
Copy link
Author

@astefanutti Yes I'm able to 👍

@astefanutti
Copy link
Owner

@vgibilmanno awesome! You'll need Node.js installed, so if you can:

  • Clone this repository: git clone https://github.com/astefanutti/kubebox.git && cd kubebox
  • Then install dependencies: npm i
  • Add require('fs').writeFileSync('reponse.json', response); above the following line:
    this.auth_provider.expiry = Date.parse(json.status.expirationTimestamp);
  • Then run node index.js
  • Finally cat response.json

Thanks again!

@vgibilmanno
Copy link
Author

@astefanutti I get the following in the response.json

{
    "kind": "ExecCredential",
    "apiVersion": "client.authentication.k8s.io/v1alpha1",
    "spec": {},
    "status": {
        "expirationTimestamp": "2019-11-29T11:07:51Z",
        "token": "SOMETOKEN"
    }
}

@astefanutti
Copy link
Owner

@vgibilmanno thanks. Could you try running:

$ aws-iam-authenticator verify -i <clusterid> -t SOMETOKEN

by taking SOMETOKEN from response.json.

@vgibilmanno
Copy link
Author

@astefanutti When I enter the cluster name in I get the following:

could not verify token: sts getCallerIdentity failed: error from AWS (expected 200, got 403)

When I enter the cluster address in I get the following:

could not verify token: input token was not properly formatted: non-whitelisted query parameter "X-Amz-SignedHeade"

@astefanutti
Copy link
Owner

@vgibilmanno could you try:

$ aws-iam-authenticator verify -i SOMEOTHERNAME -t SOMETOKEN

and provide the output of:

$ env

I suspect there are some environment variables that come into the play.

@vgibilmanno
Copy link
Author

@astefanutti So the command again fails with:

could not verify token: sts getCallerIdentity failed: error from AWS (expected 200, got 403)

And have the following entries in env:

HOSTNAME=e39ab07991c2
PYTHON_PIP_VERSION=19.3.1
SHLVL=2
HOME=/root
GPG_KEY=E3FF2839C048B25C084DEBE9B26995E310250568
PYTHON_GET_PIP_URL=https://github.com/pypa/get-pip/raw/ffe826207a010164265d9cc807978e3604d18ca0/get-pip.py
TERM=xterm
PATH=/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
LANG=C.UTF-8
PYTHON_VERSION=3.8.0
PWD=/
PYTHON_GET_PIP_SHA256=b86f36cc4345ae87bfd4f10ef6b2dbfa7a872fbff70608a1e43944d283fd0eee

@astefanutti
Copy link
Owner

astefanutti commented Nov 29, 2019

Damn, would you mind now trying:

$ aws-iam-authenticator verify -i SOMEOTHERNAME -t `aws-iam-authenticator token -i SOMEOTHERNAME -r arn:aws:iam::SOMEROLEID:role/SOMEROLE | jq -r .status.token`

(if you don't have jq, you can copy the token manually)

@vgibilmanno
Copy link
Author

@astefanutti
&{ARN:arn:aws:sts::SOMEROLEID:assumed-role/SOMEROLE/SOMESESSIONNAME CanonicalARN:arn:aws:iam::SOMEROLEID:role/SOMEROLE AccountID:SOMEACCOUNTID UserID:SOMEUSERID SessionName:SOMESESSIONNAME}

@astefanutti
Copy link
Owner

@vgibilmanno could you check any difference between the token from response.json and the one returned by aws-iam-authenticator token -i SOMEOTHERNAME -r arn:aws:iam::SOMEROLEID:role/SOMEROLE?

@vgibilmanno
Copy link
Author

@astefanutti The tokens are different. The first 190 characters are identical. The next some thousand characters are different.

@astefanutti
Copy link
Owner

astefanutti commented Nov 29, 2019

I don't understand why it does not work when a token generated with AWS CLI, that verifies correctly, is copied into the kubeconfig file and used by Kubebox. And why a token generated within Kubebox does not verify!

I've stumbled upon kubernetes-sigs/aws-iam-authenticator#157. I'm not sure if that applies to that issue to some extent.

@astefanutti
Copy link
Owner

@vgibilmanno could you try:

$ curl -k -v -H "Authorization: Bearer `aws-iam-authenticator token -i SOMEOTHERNAME -r arn:aws:iam::SOMEROLEID:role/SOMEROLE | jq -r .status.token`" https://OMITTED.amazonaws.com/api/v1/namespaces/SOMENAMESPACE/pods

@vgibilmanno
Copy link
Author

@astefanutti This one worked!

*   Trying SOMEIP:443...
* TCP_NODELAY set
* Connected to OMMITED.amazonaws.com (SOMEIP) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Request CERT (13):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=kube-apiserver
*  start date: Jan 23 14:39:32 2019 GMT
*  expire date: Nov 17 23:02:45 2020 GMT
*  issuer: CN=kubernetes
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x5653cfda06e0)
> GET /api/v1/namespaces/SOMENAMESPACE/pods HTTP/2
> Host: OMMITED.amazonaws.com
> User-Agent: curl/7.66.0
> Accept: */*
> Authorization: Bearer SOMETOKEN
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
< HTTP/2 200
< audit-id: c62fa312-7120-426b-970e-9663a90ec01f
< content-type: application/json
< date: Fri, 29 Nov 2019 15:58:25 GMT
<
{
  "kind": "PodList",
  "apiVersion": "v1",
  "metadata": {
    "selfLink": "/api/v1/namespaces/SOMENAMESPACE/pods",
    "resourceVersion": "82672454"
  },
  "items": [SOMEITEMS]
* Connection #0 to host OMMITED.amazonaws.com left intact

@astefanutti
Copy link
Owner

@vgibilmanno, good, what about:

$ curl -k -v -H "Authorization: Bearer `aws-iam-authenticator token -i SOMEOTHERNAME -r arn:aws:iam::SOMEROLEID:role/SOMEROLE | jq -r .status.token`" https://OMITTED.amazonaws.com/

$ curl -k -v -H "Authorization: Bearer `aws-iam-authenticator token -i SOMEOTHERNAME -r arn:aws:iam::SOMEROLEID:role/SOMEROLE | jq -r .status.token`" https://OMITTED.amazonaws.com/api

$ curl -k -v -H "Authorization: Bearer `aws-iam-authenticator token -i SOMEOTHERNAME -r arn:aws:iam::SOMEROLEID:role/SOMEROLE | jq -r .status.token`" https://OMITTED.amazonaws.com/api/v1/namespaces

@vgibilmanno
Copy link
Author

@astefanutti Only the 2. command worked. The other fails with statuscode 403 Forbidden. The 2. command has the following response body:

"kind": "APIVersions",
  "versions": [
    "v1"
  ],
  "serverAddressByClientCIDRs": [
    {
      "clientCIDR": "0.0.0.0/0",
      "serverAddress": "OMITTED.compute.internal:443"
    }
  ]

I tried the following command and it worked too returning a huge json object:

$ curl -k -v -H "Authorization: Bearer aws-iam-authenticator token -i SOMEOTHERNAME -r arn:aws:iam::SOMEROLEID:role/SOMEROLE | jq -r .status.token" https://OMITTED.amazonaws.com/api/v1

@astefanutti
Copy link
Owner

@vgibilmanno thanks a lot. Can you confirm the following command returns 403:

$ curl -k -v -H "Authorization: Bearer `aws-iam-authenticator token -i SOMEOTHERNAME -r arn:aws:iam::SOMEROLEID:role/SOMEROLE | jq -r .status.token`" https://OMITTED.amazonaws.com/api/v1/namespaces

@vgibilmanno
Copy link
Author

@astefanutti Yes

HTTP/2 403
< audit-id: 31bb7393-cefd-414b-8956-898135d0da83
< content-type: application/json
< x-content-type-options: nosniff
< content-length: 347
< date: Fri, 29 Nov 2019 16:56:30 GMT
<
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {

  },
  "status": "Failure",
  "message": "namespaces is forbidden: User \"SOMEUSER\" cannot list resource \"namespaces\" in API group \"\" at the cluster scope",
  "reason": "Forbidden",
  "details": {
    "kind": "namespaces"
  },
  "code": 403
* Connection #0 to host OMITTED.amazonaws.com left intact

@astefanutti
Copy link
Owner

astefanutti commented Nov 29, 2019

OK so I think we've nail down the root cause of this issue. Your user account is not granted permission to list namespaces.

As as work-around, could you update your kubeconfig file and add the namespace you have access to, e.g.:

- context:
    cluster: SOMENAME
    user: SOMENAME
    namespace: NAMESPACE
...

Then run Kubebox.

@astefanutti astefanutti removed the bug label Nov 29, 2019
@vgibilmanno
Copy link
Author

@astefanutti ok yes now it works. But it seems like I don't have permissions to see the resources usage metrics. I can execute kubectl top pod though. Well... better than nothing :) I hope I didn't waste too much of your time.

@astefanutti
Copy link
Owner

@vgibilmanno great! thanks for the feedback. Resources usage metrics requires extra permissions to proxy nodes.

Thanks a lot for your collaboration on this. I think we can let that issue open so that we improve the error message when the user is granted permission to list the namespaces and no namespace is provided.

@astefanutti astefanutti changed the title Authentication failed with AWS Improved message when not authorised to list namespaces Nov 29, 2019
@bradam12
Copy link

I'm 99% sure this issue will cover this but I get an ugly error when I try and list namespaces. I only have access to a single namespace, and it works fine otherwise, but if I hit N to list namespaces this comes up. I understand why it won't let me, but catching the error with a better message would be cool.

Screen Shot 2020-01-16 at 3 26 39 PM

@astefanutti
Copy link
Owner

@bradamson thanks a lot for the feedback. I agree with your suggestion to catch the error and display a proper message in the namespaces list box, instead of dumping the stack trace.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants