Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] admin permissions checked incorrectly #3128

Closed
Yury-Fridlyand opened this issue Aug 8, 2023 · 17 comments
Closed

[BUG] admin permissions checked incorrectly #3128

Yury-Fridlyand opened this issue Aug 8, 2023 · 17 comments
Assignees
Labels
bug Something isn't working

Comments

@Yury-Fridlyand
Copy link

What is the bug?

isAdmin check in

public boolean isAdmin(User user) {
if (isAdminDN(user.getName())) {
return true;

always fails, because in
string passed to new LdapName(...) is incorrect.

Thich check called from multiple places, for example from

private boolean userIsSuperAdmin(User user, AdminDNs adminDNs) {
return user != null && adminDNs.isAdmin(user);
}

This gives me error

{
  "error": {
    "reason": "Error occurred in OpenSearch engine: No user found for indices:data/read/search",
    "details": "OpenSearchSecurityException[No user found for indices:data/read/search]\nFor more details, please send request for Json format to see the raw response from OpenSearch engine.",
    "type": "OpenSearchSecurityException"
  },
  "status": 500
}

For built-in user admin with godlike permissions all_acccess.
Seen in different IT test setups, for example in opensearch-project/opensearch-net#311.

How can one reproduce the bug?

  1. Run new cluster with security
  2. Run any query/request

What is the expected behavior?

The check should pass.
The code should be I suppose

    public boolean isAdminDN(String dn) {

        if (dn == null) return false;

        try {
            return isAdminDN(new LdapName("CN=" + dn));
        } catch (InvalidNameException e) {
            return false;
        }
    }

What is your host/environment?

Do you have any screenshots?

image

Do you have any additional context?

I can submit a PR if you think that updating isAdminDN as I proposed is good enough.

@Yury-Fridlyand Yury-Fridlyand added bug Something isn't working untriaged Require the attention of the repository maintainers and may need to be prioritized labels Aug 8, 2023
@cwperks
Copy link
Member

cwperks commented Aug 8, 2023

@Yury-Fridlyand the internal admin user does not have permissions to do everything in a cluster. There is another category of admin (referred to as super admin) which refers to a user connecting with the admin certificate. The DN (or distinguished name) of the admin certificate is defined in opensearch.yml like this.

What is the search request you are performing where you are seeing the insufficient privileges issue?

FYI super admin can perform operations like deleting the security index, making updates to the security index directly or reload certificates and a regular admin cannot perform those operations.

@Yury-Fridlyand
Copy link
Author

I'm trying to do a search request (indices:search) authenticated as admin.
Part of my config is:

plugins.security.authcz.admin_dn: CN=admin,OU=SSL,O=Test,L=Test,C=DE

You can see all my config attached.

@cwperks
Copy link
Member

cwperks commented Aug 8, 2023

@Yury-Fridlyand The linked errors in the .NET client look related to a known issue in 2.9.0 when using testClusters and the security plugin. More details can be found on this issue: #3064

This should be resolved now after merging this PR: #3066

@cwperks
Copy link
Member

cwperks commented Aug 8, 2023

I added backport 2.9 label to #3066

@Yury-Fridlyand
Copy link
Author

I use a 3.0.0 shapshot with this fix already.
I discovered more info to clarify the issue:
Request to http://127.0.0.1:62704/<index>/_search?pretty passes, but to http://127.0.0.1:62704/*:<index>/_search?pretty fails (when index referenced by *:<index> or <cluster>:<index>) with

{
  "error" : {
    "root_cause" : [
      {
        "type" : "security_exception",
        "reason" : "No user found for indices:data/read/search"
      }
    ],
    "type" : "security_exception",
    "reason" : "No user found for indices:data/read/search"
  },
  "status" : 500
}

@cwperks
Copy link
Member

cwperks commented Aug 8, 2023

It looks like colon in the index name is problematic? Thank you for reporting the issue @Yury-Fridlyand!

@Yury-Fridlyand
Copy link
Author

Maybe I have a misconfiguration.
Can you test this on your side please @cwperks ?

@cwperks
Copy link
Member

cwperks commented Aug 8, 2023

I will try to reproduce and report back. Is this around cross cluster search?

@Yury-Fridlyand
Copy link
Author

Yes

@Xtansia
Copy link
Contributor

Xtansia commented Aug 8, 2023

The .NET client is doing approximately:

PUT _cluster/settings
{
  "transient": {
    "cluster": {
      "remote": {
        "cluster_one": {
          "seeds": ["127.0.0.1:9300", "127.0.0.1:9301"]
        },
        "cluster_two": {
          "seeds": ["127.0.0.1:9300"]
        }
      }
    }
  }
}

PUT project
{}

POST cluster_one:project,cluster_two:project/_search
{
  "query": {
    "match_all": {}
  }
}

On 2.8.0:
Screenshot 2023-08-09 at 9 11 09 AM

On 2.9.0:
Screenshot 2023-08-08 at 4 33 12 PM

@DarshitChanpura
Copy link
Member

DarshitChanpura commented Aug 9, 2023

@Xtansia Is your cluster set-up multi-node? There is a known issue in multi-node cluster setup in same JVM which causes No user found... errors. #3128 (comment)

Update: This is fixed for 2.9: #3129

@peternied peternied removed the untriaged Require the attention of the repository maintainers and may need to be prioritized label Aug 9, 2023
@DarshitChanpura
Copy link
Member

Hi @Yury-Fridlyand, I was not able to reproduce this with the latest changes from main as of 09/08:

Steps Followed when trying to re-produce:

Cluster setup:

  1. Follow DEVELOPER_GUIDE to get 2 OpenSearch clusters ready with security enabled (run it from 2 separate install locations to keep track easily)

  2. Replace the opensearch.yml contents of each cluster with this:
    a. For cluster 1: opensearch-cluster1.yml.txt
    b. For cluster 2: opensearch-cluster2.yml.txt

  3. Verify they are running.

~ curl -XGET -k -u '<admin-creds>' 'https://localhost:9200/'
{
  "name" : "node-1",
  "cluster_name" : "cluster-1",
  ...
}
~ curl -XGET -k -u '<admin-creds>' 'https://localhost:9250/'
{
  "name" : "node-2",
  "cluster_name" : "cluster-2",
  ...
}

CCS Setup:

  1. Add cluster-2 as a remote host for cluster-1:
~ curl -k -XPUT -H 'Content-Type: application/json' -u '<admin-creds>' 'https://localhost:9200/_cluster/settings' -d '
{
  "persistent": {
    "cluster.remote": {
      "cluster-2": {
        "seeds": ["127.0.0.1:9300"]
      }
    }
  }
}'
{"acknowledged":true,"persistent":{"cluster":{"remote":{"cluster-2":{"seeds":["127.0.0.1:9300"]}}}},"transient":{}}%
  1. Verify remote cluster is successfully connected:
~ curl -XGET -k -u '<admin-creds>' 'https://localhost:9200/_remote/info'
{"cluster-2":{"connected":true,"mode":"sniff","seeds":["127.0.0.1:9300"],"num_nodes_connected":1,"max_connections_per_cluster":3,"initial_connect_timeout":"30s","skip_unavailable":false}}%

Test the setup:

  1. Create a books index on cluster-2:
~ curl -XPUT -k -H 'Content-Type: application/json' -u '<admin-creds>' 'https://localhost:9250/books/_doc/1' -d '{"Dracula": "Bram Stoker"}'
{"_index":"books","_id":"1","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":0,"_primary_term":1}%
  1. Search for book by requesting cluster-1:
~ curl -XGET -k -u '<admin-creds>' 'https://localhost:9200/cluster-2:books/_search?pretty'

{
  "took" : 119,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "_clusters" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0
  },
  "hits" : {
    "total" : {
      "value" : 1,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "cluster-2:books",
        "_id" : "1",
        "_score" : 1.0,
        "_source" : {
          "Dracula" : "Bram Stoker"
        }
      }
    ]
  }
}
  1. query with * pattern for cross-cluster search:
~ curl -XGET -k -u '<admin-creds>' 'https://localhost:9200/*:books/_search?pretty'

{
  "took" : 22,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "_clusters" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0
  },
  "hits" : {
    "total" : {
      "value" : 1,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "cluster-2:books",
        "_id" : "1",
        "_score" : 1.0,
        "_source" : {
          "Dracula" : "Bram Stoker"
        }
      }
    ]
  }
}

Note: I get index_not_found_exception for following search, but that could be due to a different issue:

~ curl -XGET -k -u '<admin-creds>' 'https://localhost:9200/cluster-2:books,cluster-1:books/_search?pretty'

{
  "error" : {
    "root_cause" : [
      {
        "type" : "index_not_found_exception",
        "reason" : "no such index [cluster-1:books]",
        "index" : "cluster-1:books",
        "resource.id" : "cluster-1:books",
        "resource.type" : "index_or_alias",
        "index_uuid" : "_na_"
      }
    ],
    "type" : "index_not_found_exception",
    "reason" : "no such index [cluster-1:books]",
    "index" : "cluster-1:books",
    "resource.id" : "cluster-1:books",
    "resource.type" : "index_or_alias",
    "index_uuid" : "_na_"
  },
  "status" : 404
}

@Yury-Fridlyand
Copy link
Author

Reproduced with opensearch 7278f434cd83f095623e34c2105f1b79d247af18 + security 46989b5

  1. Build OS, unpack
  2. Build plugin, install
  3. cd to plugins/opensearch-security/tools and run install_demo_configuration
  4. Run cluster
  5. Query for https://localhost:9200/*:security-auditlog-*/_search?pretty - works fine
  6. Configure remote cluster
curl --location --request PUT 'https://localhost:9200/_cluster/settings' \
--header 'Content-Type: application/json' \
--header 'Authorization: Basic YWRtaW46YWRtaW4=' \
--data '{
  "transient": {
    "cluster": {
      "remote": {
        "cluster_one": {
          "seeds": ["127.0.0.1:9300", "127.0.0.1:9301"]
        },
        "cluster_two": {
          "seeds": ["127.0.0.1:9300"]
        }
      }
    }
  }
}'
  1. Repeat the query - fails

@DarshitChanpura
Copy link
Member

DarshitChanpura commented Aug 10, 2023

@Yury-Fridlyand I followed the steps you mentioned and I was not able to reproduce it. (One small change I made was I had to add -k to the request you suggested in step 6). Please see below the output log.

Output log
~ curl -XGET -u '<admin-creds>' -k 'https://localhost:9200/'
{
  "name" : "node-1",
  "cluster_name" : "cluster-1",
  "cluster_uuid" : "fJQ9dOCCSXKsk3X8WGqx4Q",
  "version" : {
    "distribution" : "opensearch",
    "number" : "3.0.0-SNAPSHOT",
    "build_type" : "tar",
    "build_hash" : "ccf0b9a5036b8779514d4bfd9034e147e4617da8",
    "build_date" : "2023-08-09T17:25:06.853863Z",
    "build_snapshot" : true,
    "lucene_version" : "9.8.0",
    "minimum_wire_compatibility_version" : "2.10.0",
    "minimum_index_compatibility_version" : "2.0.0"
  },
  "tagline" : "The OpenSearch Project: https://opensearch.org/"
}

➜  ~ curl -XGET -u '<admin-creds>' -k 'https://localhost:9200/*:security-auditlog-*/_search?pretty'
{
  "took" : 151,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "_clusters" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0
  },
  "hits" : {
    "total" : {
      "value" : 2,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "cluster-2:security-auditlog-2023.08.09",
        "_id" : "21l524kBBr9mJxlzArLn",
        "_score" : 1.0,
        "_source" : {
          "audit_cluster_name" : "cluster-2",
          "audit_transport_headers" : {
            "_opendistro_security_remotecn" : "cluster-2",
            "_opendistro_security_initial_action_class_header" : "BulkShardRequest",
            "_opendistro_security_origin_header" : "REST"
          },
          "audit_node_name" : "node-2",
          "audit_trace_task_id" : "AA-8sdP4RdGPL4KYdWGUTw:225",
          "audit_transport_request_type" : "PutMappingRequest",
          "audit_category" : "INDEX_EVENT",
          "audit_request_origin" : "REST",
          "audit_request_body" : "{\"_doc\":{\"properties\":{\"Dracula\":{\"type\":\"text\",\"fields\":{\"keyword\":{\"type\":\"keyword\",\"ignore_above\":256}}}}}}",
          "audit_node_id" : "AA-8sdP4RdGPL4KYdWGUTw",
          "audit_request_layer" : "TRANSPORT",
          "@timestamp" : "2023-08-09T18:04:48.230+00:00",
          "audit_format_version" : 4,
          "audit_request_remote_address" : "127.0.0.1",
          "audit_request_privilege" : "indices:admin/mapping/auto_put",
          "audit_node_host_address" : "127.0.0.1",
          "audit_request_effective_user" : "admin",
          "audit_trace_resolved_indices" : [
            "books"
          ],
          "audit_node_host_name" : "127.0.0.1"
        }
      },
      {
        "_index" : "cluster-2:security-auditlog-2023.08.09",
        "_id" : "3Fl524kBBr9mJxlzA7JC",
        "_score" : 1.0,
        "_source" : {
          "audit_cluster_name" : "cluster-2",
          "audit_transport_headers" : {
            "_system_index_access_allowed" : "false"
          },
          "audit_node_name" : "node-2",
          "audit_trace_task_id" : "AA-8sdP4RdGPL4KYdWGUTw:213",
          "audit_transport_request_type" : "CreateIndexRequest",
          "audit_category" : "INDEX_EVENT",
          "audit_request_origin" : "REST",
          "audit_request_body" : "{}",
          "audit_node_id" : "AA-8sdP4RdGPL4KYdWGUTw",
          "audit_request_layer" : "TRANSPORT",
          "@timestamp" : "2023-08-09T18:04:47.911+00:00",
          "audit_format_version" : 4,
          "audit_request_remote_address" : "127.0.0.1",
          "audit_request_privilege" : "indices:admin/auto_create",
          "audit_node_host_address" : "127.0.0.1",
          "audit_request_effective_user" : "admin",
          "audit_trace_indices" : [
            "books"
          ],
          "audit_node_host_name" : "127.0.0.1"
        }
      }
    ]
  }
}
➜  ~ curl --location --request PUT 'https://localhost:9200/_cluster/settings' --header 'Content-Type: application/json' --header 'Authorization: Basic YWRtaW46YWRtaW4=' --data '{
  "transient": {
    "cluster": {
      "remote": {
        "cluster-1": {
          "seeds": ["127.0.0.1:9300", "127.0.0.1:9301"]
        },
        "cluster-2": {
          "seeds": ["127.0.0.1:9300"]
        }
      }
    }
  }
}'
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
➜  ~ curl --location --request PUT 'https://localhost:9200/_cluster/settings' --header 'Content-Type: application/json' --header 'Authorization: Basic YWRtaW46YWRtaW4=' -k --data '{
  "transient": {
    "cluster": {
      "remote": {
        "cluster-1": {
          "seeds": ["127.0.0.1:9300", "127.0.0.1:9301"]
        },
        "cluster-2": {
          "seeds": ["127.0.0.1:9300"]
        }
      }
    }
  }
}'
{"acknowledged":true,"persistent":{},"transient":{"cluster":{"remote":{"cluster-1":{"seeds":["127.0.0.1:9300","127.0.0.1:9301"]},"cluster-2":{"seeds":["127.0.0.1:9300"]}}}}}%
➜  ~ curl -XGET -u '<admin-creds> -k 'https://localhost:9200/*:security-auditlog-*/_search?pretty'
{
  "took" : 17,
  "timed_out" : false,
  "num_reduce_phases" : 3,
  "_shards" : {
    "total" : 2,
    "successful" : 2,
    "skipped" : 0,
    "failed" : 0
  },
  "_clusters" : {
    "total" : 2,
    "successful" : 2,
    "skipped" : 0
  },
  "hits" : {
    "total" : {
      "value" : 4,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "cluster-1:security-auditlog-2023.08.09",
        "_id" : "21l524kBBr9mJxlzArLn",
        "_score" : 1.0,
        "_source" : {
          "audit_cluster_name" : "cluster-2",
          "audit_transport_headers" : {
            "_opendistro_security_remotecn" : "cluster-2",
            "_opendistro_security_initial_action_class_header" : "BulkShardRequest",
            "_opendistro_security_origin_header" : "REST"
          },
          "audit_node_name" : "node-2",
          "audit_trace_task_id" : "AA-8sdP4RdGPL4KYdWGUTw:225",
          "audit_transport_request_type" : "PutMappingRequest",
          "audit_category" : "INDEX_EVENT",
          "audit_request_origin" : "REST",
          "audit_request_body" : "{\"_doc\":{\"properties\":{\"Dracula\":{\"type\":\"text\",\"fields\":{\"keyword\":{\"type\":\"keyword\",\"ignore_above\":256}}}}}}",
          "audit_node_id" : "AA-8sdP4RdGPL4KYdWGUTw",
          "audit_request_layer" : "TRANSPORT",
          "@timestamp" : "2023-08-09T18:04:48.230+00:00",
          "audit_format_version" : 4,
          "audit_request_remote_address" : "127.0.0.1",
          "audit_request_privilege" : "indices:admin/mapping/auto_put",
          "audit_node_host_address" : "127.0.0.1",
          "audit_request_effective_user" : "admin",
          "audit_trace_resolved_indices" : [
            "books"
          ],
          "audit_node_host_name" : "127.0.0.1"
        }
      },
      {
        "_index" : "cluster-1:security-auditlog-2023.08.09",
        "_id" : "3Fl524kBBr9mJxlzA7JC",
        "_score" : 1.0,
        "_source" : {
          "audit_cluster_name" : "cluster-2",
          "audit_transport_headers" : {
            "_system_index_access_allowed" : "false"
          },
          "audit_node_name" : "node-2",
          "audit_trace_task_id" : "AA-8sdP4RdGPL4KYdWGUTw:213",
          "audit_transport_request_type" : "CreateIndexRequest",
          "audit_category" : "INDEX_EVENT",
          "audit_request_origin" : "REST",
          "audit_request_body" : "{}",
          "audit_node_id" : "AA-8sdP4RdGPL4KYdWGUTw",
          "audit_request_layer" : "TRANSPORT",
          "@timestamp" : "2023-08-09T18:04:47.911+00:00",
          "audit_format_version" : 4,
          "audit_request_remote_address" : "127.0.0.1",
          "audit_request_privilege" : "indices:admin/auto_create",
          "audit_node_host_address" : "127.0.0.1",
          "audit_request_effective_user" : "admin",
          "audit_trace_indices" : [
            "books"
          ],
          "audit_node_host_name" : "127.0.0.1"
        }
      },
      {
        "_index" : "cluster-2:security-auditlog-2023.08.09",
        "_id" : "21l524kBBr9mJxlzArLn",
        "_score" : 1.0,
        "_source" : {
          "audit_cluster_name" : "cluster-2",
          "audit_transport_headers" : {
            "_opendistro_security_remotecn" : "cluster-2",
            "_opendistro_security_initial_action_class_header" : "BulkShardRequest",
            "_opendistro_security_origin_header" : "REST"
          },
          "audit_node_name" : "node-2",
          "audit_trace_task_id" : "AA-8sdP4RdGPL4KYdWGUTw:225",
          "audit_transport_request_type" : "PutMappingRequest",
          "audit_category" : "INDEX_EVENT",
          "audit_request_origin" : "REST",
          "audit_request_body" : "{\"_doc\":{\"properties\":{\"Dracula\":{\"type\":\"text\",\"fields\":{\"keyword\":{\"type\":\"keyword\",\"ignore_above\":256}}}}}}",
          "audit_node_id" : "AA-8sdP4RdGPL4KYdWGUTw",
          "audit_request_layer" : "TRANSPORT",
          "@timestamp" : "2023-08-09T18:04:48.230+00:00",
          "audit_format_version" : 4,
          "audit_request_remote_address" : "127.0.0.1",
          "audit_request_privilege" : "indices:admin/mapping/auto_put",
          "audit_node_host_address" : "127.0.0.1",
          "audit_request_effective_user" : "admin",
          "audit_trace_resolved_indices" : [
            "books"
          ],
          "audit_node_host_name" : "127.0.0.1"
        }
      },
      {
        "_index" : "cluster-2:security-auditlog-2023.08.09",
        "_id" : "3Fl524kBBr9mJxlzA7JC",
        "_score" : 1.0,
        "_source" : {
          "audit_cluster_name" : "cluster-2",
          "audit_transport_headers" : {
            "_system_index_access_allowed" : "false"
          },
          "audit_node_name" : "node-2",
          "audit_trace_task_id" : "AA-8sdP4RdGPL4KYdWGUTw:213",
          "audit_transport_request_type" : "CreateIndexRequest",
          "audit_category" : "INDEX_EVENT",
          "audit_request_origin" : "REST",
          "audit_request_body" : "{}",
          "audit_node_id" : "AA-8sdP4RdGPL4KYdWGUTw",
          "audit_request_layer" : "TRANSPORT",
          "@timestamp" : "2023-08-09T18:04:47.911+00:00",
          "audit_format_version" : 4,
          "audit_request_remote_address" : "127.0.0.1",
          "audit_request_privilege" : "indices:admin/auto_create",
          "audit_node_host_address" : "127.0.0.1",
          "audit_request_effective_user" : "admin",
          "audit_trace_indices" : [
            "books"
          ],
          "audit_node_host_name" : "127.0.0.1"
        }
      }
    ]
  }
}
➜  ~

Seems like there might be a configuration issue on your end. Can you please post your set-up (opensearch.yml) and failure log?

@Yury-Fridlyand
Copy link
Author

@DarshitChanpura
Please see my logs and config:
logs.zip
config.zip

@DarshitChanpura
Copy link
Member

@Yury-Fridlyand Here is a summary of our discussion on slack:

  • Tried to reproduce the issue with your setup
  • Figured out that when trying to add a remote-cluster, without it running leads to _cluster/settings still be successful since the IPv4 address is 127.0.0.1:9300 which is going to be same for all local clusters. This scenario won't happen in a production environment.
  • This lead to index-search requests being failed.
  • Solution: Follow steps mentioned here: [BUG] admin permissions checked incorrectly #3128 (comment)

Closing it as it was a local cluster setup issue.

@Yury-Fridlyand
Copy link
Author

I still insist that there is a bug1.
Reproduce scenario:

  1. Start cluster
  2. Configure remote cluster
  3. Try accessing any index on any cluster using CCR prefix, for example *:* (search on all indices in all clusters)
  4. ERROR ❗
  5. Start remote cluster
  6. No error

So until remote cluster started, I can't access data on the main cluster.

Footnotes

  1. Actually, two bugs. isAdminDN check is also incorrect and always fails.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

5 participants