Connect Opensearch to Active Directory

[sevenval-admins]

Describe the bug
Cannot connect Opensearch with Active Directory

To Reproduce
Steps to reproduce the behavior:

1. Create custom config file in order to connect opensearch to LDAP-Server
2. Launch docker-compose up with custom config file mounted as Volume in docker-compose.yml
3. The Cluster creation starts
4. When giving http://localhost:5601 on address bar, the Dashboard is unreachable
5. Error on UI "OpenSearch Dashboards server is not ready yet"
6. opensearch error:

[root@localhost] opensearch # curl -XGET https://localhost:9200/_cat -u 'admin:admin' --insecure
OpenSearch Security not initialized.[11:39 0.07 ] ✓

6. Without a custom config file both services are reachable

Expected behavior
Using a custom config file should not prevent to run the services

Plugins
Please list all plugins currently enabled.

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

• OS: CentOS 7
• docker-compose version 1.18.0

Additional context
According to the instructions here, I used docker-compose to start a Cluster. So far no problem, the containers start, I can log into the Dashboard and start sending logs to the Opensearch instances.
According to the instructions here I rewrote the file /usr/share/opensearch/plugins/opensearch-security/securityconfig/config.yml to connect Opensearch with my LDAP-Server.

authc
      ldap:
        description: "Authenticate via LDAP or Active Directory"
        http_enabled: true
        transport_enabled: true
        order: 1
        http_authenticator:
          type: basic
          challenge: false
        authentication_backend:
          # LDAP authentication backend (authenticate users against a LDAP or Active Directory)
          type: ldap
          config:
            # enable ldaps
            enable_ssl: true
            # enable start tls, enable_ssl should be false
            enable_start_tls: false
            # send client certificate
            enable_ssl_client_auth: false
            # verify ldap hostname
            verify_hostnames: true
            hosts:
            - my-ldap-server:636
            bind_dn: CN=user,OU=OU Service Accounts,OU=OU Organisation,DC=example,DC=com
            password: <mypassword>
            userbase: 'DC=example,DC=com'
            # Filter to search for users (currently in the whole subtree beneath userbase)
            # {0} is substituted with the username
            usersearch: '(sAMAccountName=%s)'
            # Use this attribute from the user as username (if not set then DN is used)
            username_attribute: uid
    authz:
      roles_from_myldap:
        description: "Authorize via LDAP or Active Directory"
        http_enabled: true
        transport_enabled: true
        authorization_backend:
          # LDAP authorization backend (gather roles from a LDAP or Active Directory, you have to configure the above LDAP authentication backend settings too)
          type: ldap
          # LDAP authorization backend (gather roles from a LDAP or Active Directory, you have to configure the above LDAP authentication backend settings too)
          type: ldap
          config:
            # enable ldaps
            enable_ssl: true
            # enable start tls, enable_ssl should be false
            enable_start_tls: false
            # send client certificate
            enable_ssl_client_auth: false
            # verify ldap hostname
            verify_hostnames: true
            hosts:
            - my-ldap-server:636
            bind_dn: CN=user,OU=OU Service Accounts,OU=OU Organisation,DC=example,DC=com
            password: <mypassword>
            userbase: 'DC=example,DC=com'
            # Filter to search for roles (currently in the whole subtree beneath rolebase)
            # {0} is substituted with the DN of the user
            # {1} is substituted with the username
            # {2} is substituted with an attribute value from user's directory entry, of the authenticated user. Use userroleattribute to specify the name of the attribute
            rolesearch: '(member={0})'
            # Specify the name of the attribute which value should be substituted with {2} above
            userroleattribute: null
            # Roles as an attribute of the user entry
            userrolename: none
            #userrolename: memberOf
            # The attribute in a role entry containing the name of that role, Default is "name".
            # Can also be "dn" to use the full DN as rolename.
            rolename: cn
            # Resolve nested roles transitive (roles which are members of other roles and so on ...)
            resolve_nested_roles: true
            userbase: 'DC=example,DC=com'
            # Filter to search for users (currently in the whole subtree beneath userbase)
            # {0} is substituted with the username
            usersearch: '(uid={0})'
            # Skip users matching a user name, a wildcard or a regex pattern
            skip_users:
              - kibanaserver

The created file I then mounted as a volume inside the docker-compose.yml:

    volumes:
      - opensearch-data1:/usr/share/opensearch/data
      - /path/to-my-custom/config.yml:/usr/share/opensearch/plugins/opensearch-security/securityconfig/config.yml

With docker-compose down -v I stopped the cluster and deleted the previously created volumes and with docker-compose up I restarted the cluster construction. The containers start but both the opesearch service and opensearch-dashboard are not reachable.
Opensearch-Dashboard Logs:

{"type":"log","@timestamp":"2021-09-15T09:23:44Z","tags":["info","savedobjects-service"],"pid":1,"message":"Waiting until all OpenSearch nodes are compatible with OpenSearch Dashboards before starting saved objects migrations..."}
{"type":"log","@timestamp":"2021-09-15T09:23:44Z","tags":["error","opensearch","data"],"pid":1,"message":"[ConnectionError]: connect ECONNREFUSED 10.254.10.2:9200"}
{"type":"log","@timestamp":"2021-09-15T09:23:44Z","tags":["error","savedobjects-service"],"pid":1,"message":"Unable to retrieve version information from OpenSearch nodes."}
{"type":"log","@timestamp":"2021-09-15T09:23:46Z","tags":["error","opensearch","data"],"pid":1,"message":"[ConnectionError]: connect ECONNREFUSED 10.254.10.2:9200"}
{"type":"log","@timestamp":"2021-09-15T09:23:49Z","tags":["error","opensearch","data"],"pid":1,"message":"[ConnectionError]: connect ECONNREFUSED 10.254.10.2:9200"}
{"type":"log","@timestamp":"2021-09-15T09:23:51Z","tags":["error","opensearch","data"],"pid":1,"message":"[ResponseError]: Response Error"}
{"type":"log","@timestamp":"2021-09-15T09:23:54Z","tags":["error","opensearch","data"],"pid":1,"message":"[ResponseError]: Response Error"}
{"type":"log","@timestamp":"2021-09-15T09:23:56Z","tags":["error","opensearch","data"],"pid":1,"message":"[ResponseError]: Response Error"}
{"type":"log","@timestamp":"2021-09-15T09:23:59Z","tags":["error","opensearch","data"],"pid":1,"message":"[ResponseError]: Response Error"}
{"type":"log","@timestamp":"2021-09-15T09:24:01Z","tags":["error","opensearch","data"],"pid":1,"message":"[ResponseError]: Response Error"}
{"type":"log","@timestamp":"2021-09-15T09:24:04Z","tags":["error","opensearch","data"],"pid":1,"message":"[ResponseError]: Response Error"}
{"type":"log","@timestamp":"2021-09-15T09:24:06Z","tags":["error","opensearch","data"],"pid":1,"message":"[ResponseError]: Response Error"}
{"type":"log","@timestamp":"2021-09-15T09:24:09Z","tags":["error","opensearch","data"],"pid":1,"message":"[ResponseError]: Response Error"}
{"type":"log","@timestamp":"2021-09-15T09:24:11Z","tags":["error","opensearch","data"],"pid":1,"message":"[ResponseError]: Response Error"}
{"type":"log","@timestamp":"2021-09-15T09:24:14Z","tags":["error","opensearch","data"],"pid":1,"message":"[ResponseError]: Response Error"}
{"type":"log","@timestamp":"2021-09-15T09:24:16Z","tags":["error","opensearch","data"],"pid":1,"message":"[ResponseError]: Response Error"}
{"type":"log","@timestamp":"2021-09-15T09:24:19Z","tags":["error","opensearch","data"],"pid":1,"message":"[ResponseError]: Response Error"}
{"type":"log","@timestamp":"2021-09-15T09:24:21Z","tags":["error","opensearch","data"],"pid":1,"message":"[ResponseError]: Response Error"}
{"type":"log","@timestamp":"2021-09-15T09:24:24Z","tags":["error","opensearch","data"],"pid":1,"message":"[ResponseError]: Response Error"}
{"type":"log","@timestamp":"2021-09-15T09:24:26Z","tags":["error","opensearch","data"],"pid":1,"message":"[ResponseError]: Response Error"}
{"type":"log","@timestamp":"2021-09-15T09:24:29Z","tags":["error","opensearch","data"],"pid":1,"message":"[ResponseError]: Response Error"}
{"type":"log","@timestamp":"2021-09-15T09:24:31Z","tags":["error","opensearch","data"],"pid":1,"message":"[ResponseError]: Response Error"}
{"type":"log","@timestamp":"2021-09-15T09:24:34Z","tags":["error","opensearch","data"],"pid":1,"message":"[ResponseError]: Response Error"}
{"type":"log","@timestamp":"2021-09-15T09:24:36Z","tags":["error","opensearch","data"],"pid":1,"message":"[ResponseError]: Response Error"}
{"type":"log","@timestamp":"2021-09-15T09:24:39Z","tags":["error","opensearch","data"],"pid":1,"message":"[ResponseError]: Response Error"}

Opensearch-node Logs:

[2021-09-14T14:00:37,468][WARN ][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-node1] No data for internalusers while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, AUDIT]  (index=.opendistro_security and type=null)
[2021-09-14T14:00:37,468][WARN ][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-node1] No data for actiongroups while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, AUDIT]  (index=.opendistro_security and type=null)
[2021-09-14T14:00:37,468][WARN ][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-node1] No data for config while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, AUDIT]  (index=.opendistro_security and type=null)
[2021-09-14T14:00:37,468][WARN ][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-node1] No data for roles while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, AUDIT]  (index=.opendistro_security and type=null)
[2021-09-14T14:00:37,468][WARN ][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-node1] No data for rolesmapping while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, AUDIT]  (index=.opendistro_security and type=null)
[2021-09-14T14:00:37,468][WARN ][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-node1] No data for tenants while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, AUDIT]  (index=.opendistro_security and type=null)
[2021-09-14T14:00:39,564][ERROR][o.o.s.a.BackendRegistry  ] [opensearch-node2] Not yet initialized (you may need to run securityadmin)
[2021-09-14T14:00:39,568][ERROR][o.o.s.a.BackendRegistry  ] [opensearch-node2] Not yet initialized (you may need to run securityadmin)

I saw that there is already an open Issue concerning the problem that when using custom config files, the containers do not work correctly.

[saratvemulapalli]

Taking a first stab at the issue.
Looks like securityadmin.sh is not run yet and it is needed to initialize the OpenSearch security plugin.
For more details you could look through the documentation: https://opensearch.org/docs/security-plugin/configuration/security-admin/

Having said that, I'll transfer this to the security plugin as they might have more insights.

[sevenval-admins]

Hi and thanks for your reply and for taking care to transfer the issue.
I have used the securityadmin script when I have set up opensearch with the tarball option, in this case I am not sure to understand his utility: do you means that once I started the cluster with docker-compose up I have to log into one of the opensearch containers and run the securityadmin script? Does it make sense this sort of manual approach when docker compose should be censed to fire up the cluster/services described in Dockercompose file without any manual intervention?

[AntonEliatra]

@sevenval-admins The script should indeed be executed automatically assuming you have not mapped custom opensearch.yml config file with other settings. Did you try executing the script manually? Did it give you any errors during execution?

[ngalanis930]

As @sevenval-admins mentioned I haven't find a way to run the securityadmin script on startup. In case of changes, for example in config.yml I have to jump on a bash shell in the opensearch container and apply the changes via the script. Any insights from the security plugin team?

[davidlago]

Closing and tagging this issue (#1755) to the overall tools deprecation initiative as this no longer looks like an AD issue but a question about better ways of running the security admin tools.