Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update indexer base with Opensearch 2.1.0 #1723

Closed
7 tasks done
c-bordon opened this issue Jul 4, 2022 · 7 comments
Closed
7 tasks done

Update indexer base with Opensearch 2.1.0 #1723

c-bordon opened this issue Jul 4, 2022 · 7 comments
Assignees
@c-bordon

Comments

@c-bordon
Copy link
Member

c-bordon commented Jul 4, 2022 •
edited by alberpilot
Loading

  • Update the database with the Opensearch 2.1.0 tar
  • Validate package creation without errors
  • Validate installation without errors
  • Validate start without errors
  • Compare changes between packages created by Opensearch and packages created by us
  • Compare systemd and init.d services between packages
  • Validate correct file permissions

Opensearch is working on deprecating some tools, they have the following issue opensearch-project/security#1755
@c-bordon c-bordon mentioned this issue Jul 4, 2022
Closed
2 tasks
@c-bordon c-bordon self-assigned this Jul 4, 2022
@c-bordon
Copy link
Member Author

c-bordon commented Jul 6, 2022

In Opensearch 2.0 the different configuration directories were moved to /usr/share/opensearch/config:

ls -la opensearch-2.0.1/config/
total 56
drwxr-xr-x  9 cbordon cbordon  4096 jun 15 06:06 .
drwxr-xr-x 10 cbordon cbordon  4096 jun 15 06:06 ..
-rw-rw----  1 cbordon cbordon  2413 dic 31  1969 jvm.options
drwxr-x---  2 cbordon cbordon  4096 dic 31  1969 jvm.options.d
-rw-rw----  1 cbordon cbordon 12134 dic 31  1969 log4j2.properties
drwxr-x---  2 cbordon cbordon  4096 jun 15 06:06 opensearch-notifications
drwxr-x---  2 cbordon cbordon  4096 jun 15 06:06 opensearch-notifications-core
drwxr-x---  2 cbordon cbordon  4096 jun 15 06:06 opensearch-observability
drwxr-x---  2 cbordon cbordon  4096 jun 15 06:06 opensearch-performance-analyzer
drwxr-x---  2 cbordon cbordon  4096 jun 15 06:06 opensearch-reports-scheduler
drwxr-x---  2 cbordon cbordon  4096 jun 15 06:06 opensearch-security
-rw-rw----  1 cbordon cbordon  2800 dic 31  1969 opensearch.yml

If we want to keep our change made in 1.2.4 where the config directory does not exist, we will have to move all these directories to the previous destination and make the relevant changes on the code side.

@c-bordon
Copy link
Member Author

c-bordon commented Jul 6, 2022

Investigating a little more, this is "solved" with the installation by package, I am investigating how to replicate this

ls -la /etc/opensearch
total 72
drwxr-sr-x.  9 opensearch opensearch  4096 Jul  6 19:16 .
drwxr-xr-x. 80 root       root        8192 Jul  6 18:38 ..
-rw-r--r--.  1 opensearch opensearch  1704 Jul  6 18:38 esnode-key.pem
-rw-r--r--.  1 opensearch opensearch  1720 Jul  6 18:38 esnode.pem
-rw-r--r--.  1 opensearch opensearch  2714 Jul  6 18:38 jvm.options
drwxr-sr-x.  2 opensearch opensearch     6 Jun 15 08:53 jvm.options.d
-rw-r--r--.  1 opensearch opensearch  1704 Jul  6 18:38 kirk-key.pem
-rw-r--r--.  1 opensearch opensearch  1610 Jul  6 18:38 kirk.pem
-rw-r--r--.  1 opensearch opensearch 12134 Jun 15 08:53 log4j2.properties
drwxr-xr-x.  2 opensearch opensearch    31 Jul  6 18:38 opensearch-notifications
drwxr-xr-x.  2 opensearch opensearch    36 Jul  6 18:38 opensearch-notifications-core
drwxr-xr-x.  2 opensearch opensearch    31 Jul  6 18:38 opensearch-observability
drwxr-xr-x.  2 opensearch opensearch   245 Jul  6 18:38 opensearch-performance-analyzer
drwxr-xr-x.  2 opensearch opensearch    35 Jul  6 18:38 opensearch-reports-scheduler
drwxr-xr-x.  2 opensearch opensearch   224 Jul  6 18:38 opensearch-security
-rw-rw----.  1 opensearch opensearch   196 Jul  6 19:16 opensearch.keystore
-rw-r--r--.  1 opensearch opensearch  4464 Jul  6 18:38 opensearch.yml
-rw-r--r--.  1 opensearch opensearch     5 Jun 15 09:12 performance_analyzer_enabled.conf
-rw-r--r--.  1 opensearch opensearch     5 Jun 15 09:12 rca_enabled.conf
-rw-r--r--.  1 opensearch opensearch  1444 Jul  6 18:38 root-ca.pem
[root@centos7-1 ~]# ls -la /etc/opensearch/opensearch-security/
total 68
drwxr-xr-x. 2 opensearch opensearch   224 Jul  6 18:38 .
drwxr-sr-x. 9 opensearch opensearch  4096 Jul  6 19:16 ..
-rw-r--r--. 1 opensearch opensearch    50 Jun 15 09:12 action_groups.yml
-rw-r--r--. 1 opensearch opensearch  2541 Jun 15 09:12 audit.yml
-rw-r--r--. 1 opensearch opensearch  9956 Jun 15 09:12 config.yml
-rw-r--r--. 1 opensearch opensearch  1352 Jun 15 09:12 internal_users.yml
-rw-r--r--. 1 opensearch opensearch   154 Jun 15 09:12 nodes_dn.yml
-rw-r--r--. 1 opensearch opensearch 12615 Jun 15 09:12 opensearch.yml.example
-rw-r--r--. 1 opensearch opensearch  7829 Jun 15 09:12 roles.yml
-rw-r--r--. 1 opensearch opensearch   844 Jun 15 09:12 roles_mapping.yml
-rw-r--r--. 1 opensearch opensearch   170 Jun 15 09:12 tenants.yml
-rw-r--r--. 1 opensearch opensearch  1973 Jun 15 09:12 whitelist.yml

@c-bordon
Copy link
Member Author

c-bordon commented Jul 7, 2022

I have managed to make the necessary changes to keep the opensearch configuration, but I get this error when starting the security admin, which I am investigating:

[root@centos72 ~]#             /usr/share/wazuh-indexer/bin/indexer-security-init.sh
**************************************************************************
** This tool will be deprecated in the next major release of OpenSearch **
** https://github.com/opensearch-project/security/issues/1755           **
**************************************************************************
Security Admin v7
Will connect to 127.0.0.1:9300 ... done
ERR: An unexpected RuntimeException occured: error while performing request
Trace:
java.lang.RuntimeException: error while performing request
        at org.opensearch.client.RestClient.extractAndWrapCause(RestClient.java:932)
        at org.opensearch.client.RestClient.performRequest(RestClient.java:307)
        at org.opensearch.client.RestClient.performRequest(RestClient.java:295)
        at org.opensearch.security.tools.SecurityAdmin.execute(SecurityAdmin.java:464)
        at org.opensearch.security.tools.SecurityAdmin.main(SecurityAdmin.java:161)
Caused by: org.apache.http.ProtocolException: Not a valid protocol version: This is not an HTTP port
        at org.apache.http.impl.nio.codecs.AbstractMessageParser.parse(AbstractMessageParser.java:209)
        at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:245)
        at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81)
        at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39)
        at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:121)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
        at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
        at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: org.apache.http.ParseException: Not a valid protocol version: This is not an HTTP port
        at org.apache.http.message.BasicLineParser.parseProtocolVersion(BasicLineParser.java:148)
        at org.apache.http.message.BasicLineParser.parseStatusLine(BasicLineParser.java:366)
        at org.apache.http.impl.nio.codecs.DefaultHttpResponseParser.createMessage(DefaultHttpResponseParser.java:112)
        at org.apache.http.impl.nio.codecs.DefaultHttpResponseParser.createMessage(DefaultHttpResponseParser.java:50)
        at org.apache.http.impl.nio.codecs.AbstractMessageParser.parseHeadLine(AbstractMessageParser.java:156)
        at org.apache.http.impl.nio.codecs.AbstractMessageParser.parse(AbstractMessageParser.java:207)
        ... 11 more

[root@centos72 ~]# systemctl status wazuh-indexer
● wazuh-indexer.service - Wazuh-indexer
   Loaded: loaded (/usr/lib/systemd/system/wazuh-indexer.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2022-07-07 19:18:54 UTC; 20min ago
     Docs: https://documentation.wazuh.com
 Main PID: 25256 (java)
   CGroup: /system.slice/wazuh-indexer.service
           └─25256 /usr/share/wazuh-indexer/jdk/bin/java -Xshare:auto -Dopensearch.networkaddress.cache.ttl=60 -Dopensearch.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8...

Jul 07 19:18:46 centos72 systemd[1]: Starting Wazuh-indexer...
Jul 07 19:18:47 centos72 systemd-entrypoint[25256]: WARNING: A terminally deprecated method in java.lang.System has been called
Jul 07 19:18:47 centos72 systemd-entrypoint[25256]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/wazuh-indexer/lib/opensearch-2.0.1.jar)
Jul 07 19:18:47 centos72 systemd-entrypoint[25256]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
Jul 07 19:18:47 centos72 systemd-entrypoint[25256]: WARNING: System::setSecurityManager will be removed in a future release
Jul 07 19:18:48 centos72 systemd-entrypoint[25256]: WARNING: A terminally deprecated method in java.lang.System has been called
Jul 07 19:18:48 centos72 systemd-entrypoint[25256]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/wazuh-indexer/lib/opensearch-2.0.1.jar)
Jul 07 19:18:48 centos72 systemd-entrypoint[25256]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
Jul 07 19:18:48 centos72 systemd-entrypoint[25256]: WARNING: System::setSecurityManager will be removed in a future release
Jul 07 19:18:54 centos72 systemd[1]: Started Wazuh-indexer.

@c-bordon
Copy link
Member Author

c-bordon commented Jul 8, 2022

Regarding the error mentioned above, some changes were made in opensearch and the security admin must be pointed to port 9200 and not 9300 as in the previous version:

[root@centos72 ~]# /usr/share/wazuh-indexer/bin/indexer-security-init.sh 
**************************************************************************
** This tool will be deprecated in the next major release of OpenSearch **
** https://github.com/opensearch-project/security/issues/1755           **
**************************************************************************
Security Admin v7
Will connect to 127.0.0.1:9200 ... done
Connected as "CN=admin,OU=Wazuh,O=Wazuh,L=California,C=US"
OpenSearch Version: 2.0.1
Contacting opensearch cluster 'opensearch' and wait for YELLOW clusterstate ...
Clustername: wazuh-cluster
Clusterstate: GREEN
Number of nodes: 1
Number of data nodes: 1
.opendistro_security index does not exists, attempt to create it ... done (0-all replicas)
Populate config from /usr/share/wazuh-indexer/plugins/opensearch-security/securityconfig/
ERR: Seems /usr/share/wazuh-indexer/plugins/opensearch-security/securityconfig/config.yml is not in OpenSearch Security 7 format: java.io.FileNotFoundException: /usr/share/wazuh-indexer/plugins/opensearch-security/securityconfig/config.yml (No such file or directory)
ERR: Seems /usr/share/wazuh-indexer/plugins/opensearch-security/securityconfig/roles.yml is not in OpenSearch Security 7 format: java.io.FileNotFoundException: /usr/share/wazuh-indexer/plugins/opensearch-security/securityconfig/roles.yml (No such file or directory)
ERR: Seems /usr/share/wazuh-indexer/plugins/opensearch-security/securityconfig/roles_mapping.yml is not in OpenSearch Security 7 format: java.io.FileNotFoundException: /usr/share/wazuh-indexer/plugins/opensearch-security/securityconfig/roles_mapping.yml (No such file or directory)
ERR: Seems /usr/share/wazuh-indexer/plugins/opensearch-security/securityconfig/internal_users.yml is not in OpenSearch Security 7 format: java.io.FileNotFoundException: /usr/share/wazuh-indexer/plugins/opensearch-security/securityconfig/internal_users.yml (No such file or directory)
ERR: Seems /usr/share/wazuh-indexer/plugins/opensearch-security/securityconfig/action_groups.yml is not in OpenSearch Security 7 format: java.io.FileNotFoundException: /usr/share/wazuh-indexer/plugins/opensearch-security/securityconfig/action_groups.yml (No such file or directory)
ERR: Seems /usr/share/wazuh-indexer/plugins/opensearch-security/securityconfig/tenants.yml is not in OpenSearch Security 7 format: java.io.FileNotFoundException: /usr/share/wazuh-indexer/plugins/opensearch-security/securityconfig/tenants.yml (No such file or directory)
ERR: Seems /usr/share/wazuh-indexer/plugins/opensearch-security/securityconfig/nodes_dn.yml is not in OpenSearch Security 7 format: java.io.FileNotFoundException: /usr/share/wazuh-indexer/plugins/opensearch-security/securityconfig/nodes_dn.yml (No such file or directory)
ERR: Seems /usr/share/wazuh-indexer/plugins/opensearch-security/securityconfig/whitelist.yml is not in OpenSearch Security 7 format: java.io.FileNotFoundException: /usr/share/wazuh-indexer/plugins/opensearch-security/securityconfig/whitelist.yml (No such file or directory)
ERR: cannot upload configuration, see errors above

Documentation:
https://forum.opensearch.org/t/securityadmin-sh-uses-http-port-9200-and-not-transport-port-9300-in-opensearch-2-0-0/9760/4
https://opensearch.org/docs/latest/security-plugin/configuration/security-admin/#basic-usage
opensearch-project/documentation-website#695

@c-bordon
Copy link
Member Author

c-bordon commented Jul 8, 2022

The configuration error was in the indexer-security-init.sh script

a new variable is added with the path of the securityadmin configuration files

[root@centos72 ~]#             /usr/share/wazuh-indexer/bin/indexer-security-init.sh
**************************************************************************
** This tool will be deprecated in the next major release of OpenSearch **
** https://github.com/opensearch-project/security/issues/1755           **
**************************************************************************
Security Admin v7
Will connect to 127.0.0.1:9200 ... done
Connected as "CN=admin,OU=Wazuh,O=Wazuh,L=California,C=US"
OpenSearch Version: 2.0.1
Contacting opensearch cluster 'opensearch' and wait for YELLOW clusterstate ...
Clustername: wazuh-cluster
Clusterstate: GREEN
Number of nodes: 1
Number of data nodes: 1
.opendistro_security index does not exists, attempt to create it ... done (0-all replicas)
Populate config from /etc/wazuh-indexer/opensearch-security/
Will update '/config' with /etc/wazuh-indexer/opensearch-security/config.yml 
   SUCC: Configuration for 'config' created or updated
Will update '/roles' with /etc/wazuh-indexer/opensearch-security/roles.yml 
   SUCC: Configuration for 'roles' created or updated
Will update '/rolesmapping' with /etc/wazuh-indexer/opensearch-security/roles_mapping.yml 
   SUCC: Configuration for 'rolesmapping' created or updated
Will update '/internalusers' with /etc/wazuh-indexer/opensearch-security/internal_users.yml 
   SUCC: Configuration for 'internalusers' created or updated
Will update '/actiongroups' with /etc/wazuh-indexer/opensearch-security/action_groups.yml 
   SUCC: Configuration for 'actiongroups' created or updated
Will update '/tenants' with /etc/wazuh-indexer/opensearch-security/tenants.yml 
   SUCC: Configuration for 'tenants' created or updated
Will update '/nodesdn' with /etc/wazuh-indexer/opensearch-security/nodes_dn.yml 
   SUCC: Configuration for 'nodesdn' created or updated
Will update '/whitelist' with /etc/wazuh-indexer/opensearch-security/whitelist.yml 
   SUCC: Configuration for 'whitelist' created or updated
Will update '/audit' with /etc/wazuh-indexer/opensearch-security/audit.yml 
   SUCC: Configuration for 'audit' created or updated
SUCC: Expected 9 config types for node {"updated_config_types":["tenants","rolesmapping","nodesdn","audit","roles","whitelist","internalusers","actiongroups","config"],"updated_config_size":9,"message":null} is 9 (["tenants","rolesmapping","nodesdn","audit","roles","whitelist","internalusers","actiongroups","config"]) due to: null
Done with success

@c-bordon
Copy link
Member Author

c-bordon commented Jul 8, 2022

Wazuh indexer 4.4.0 installation success:

[root@centos72 ~]# yum localinstall -y /home/vagrant/wazuh-indexer-4.4.0-1.x86_64.rpm 
Failed to set locale, defaulting to C
Loaded plugins: fastestmirror
Examining /home/vagrant/wazuh-indexer-4.4.0-1.x86_64.rpm: wazuh-indexer-4.4.0-1.x86_64
Marking /home/vagrant/wazuh-indexer-4.4.0-1.x86_64.rpm to be installed
Resolving Dependencies
--> Running transaction check
---> Package wazuh-indexer.x86_64 0:4.4.0-1 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

=============================================================================================================================================================================================================================================
 Package                                                 Arch                                             Version                                              Repository                                                               Size
=============================================================================================================================================================================================================================================
Installing:
 wazuh-indexer                                           x86_64                                           4.4.0-1                                              /wazuh-indexer-4.4.0-1.x86_64                                           642 M

Transaction Summary
=============================================================================================================================================================================================================================================
Install  1 Package

Total size: 642 M
Installed size: 642 M
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : wazuh-indexer-4.4.0-1.x86_64                                                                                                                                                                                              1/1 
Created opensearch keystore in /etc/wazuh-indexer/opensearch.keystore
  Verifying  : wazuh-indexer-4.4.0-1.x86_64                                                                                                                                                                                              1/1 

Installed:
  wazuh-indexer.x86_64 0:4.4.0-1                                                                                                                                                                                                             

Complete!

indexer-security-init.sh

[root@centos72 ~]#             /usr/share/wazuh-indexer/bin/indexer-security-init.sh
**************************************************************************
** This tool will be deprecated in the next major release of OpenSearch **
** https://github.com/opensearch-project/security/issues/1755           **
**************************************************************************
Security Admin v7
Will connect to 127.0.0.1:9200 ... done
Connected as "CN=admin,OU=Wazuh,O=Wazuh,L=California,C=US"
OpenSearch Version: 2.0.1
Contacting opensearch cluster 'opensearch' and wait for YELLOW clusterstate ...
Clustername: wazuh-cluster
Clusterstate: GREEN
Number of nodes: 1
Number of data nodes: 1
.opendistro_security index does not exists, attempt to create it ... done (0-all replicas)
Populate config from /etc/wazuh-indexer/opensearch-security/
Will update '/config' with /etc/wazuh-indexer/opensearch-security/config.yml 
   SUCC: Configuration for 'config' created or updated
Will update '/roles' with /etc/wazuh-indexer/opensearch-security/roles.yml 
   SUCC: Configuration for 'roles' created or updated
Will update '/rolesmapping' with /etc/wazuh-indexer/opensearch-security/roles_mapping.yml 
   SUCC: Configuration for 'rolesmapping' created or updated
Will update '/internalusers' with /etc/wazuh-indexer/opensearch-security/internal_users.yml 
   SUCC: Configuration for 'internalusers' created or updated
Will update '/actiongroups' with /etc/wazuh-indexer/opensearch-security/action_groups.yml 
   SUCC: Configuration for 'actiongroups' created or updated
Will update '/tenants' with /etc/wazuh-indexer/opensearch-security/tenants.yml 
   SUCC: Configuration for 'tenants' created or updated
Will update '/nodesdn' with /etc/wazuh-indexer/opensearch-security/nodes_dn.yml 
   SUCC: Configuration for 'nodesdn' created or updated
Will update '/whitelist' with /etc/wazuh-indexer/opensearch-security/whitelist.yml 
   SUCC: Configuration for 'whitelist' created or updated
Will update '/audit' with /etc/wazuh-indexer/opensearch-security/audit.yml 
   SUCC: Configuration for 'audit' created or updated
SUCC: Expected 9 config types for node {"updated_config_types":["tenants","rolesmapping","nodesdn","audit","roles","whitelist","internalusers","actiongroups","config"],"updated_config_size":9,"message":null} is 9 (["tenants","rolesmapping","nodesdn","audit","roles","whitelist","internalusers","actiongroups","config"]) due to: null
Done with success

@c-bordon
Copy link
Member Author

Making the comparisons between the packages I did not find very big changes, I detail some:

/etc/opensearch/opensearch.yml

plugins.security.system_indices.indices: [".plugins-ml-model", ".plugins-ml-task", ".opendistro-alerting-config", ".opendistro-alerting-alert*", ".opendistro-anomaly-results*", ".opendistro-anomaly-detector*", ".opendistro-anomaly-checkpoints", ".opendistro-anomaly-detection-state", ".opendistro-reports-*", ".opensearch-notifications-*", ".opensearch-notebooks", ".opensearch-observability", ".opendistro-asynchronous-search-response*", ".replication-metadata-store"]

/etc/wazuh-indexer/opensearch.yml

plugins.security.system_indices.indices: [".opendistro-alerting-config", ".opendistro-alerting-alert*", ".opendistro-anomaly-results*", ".opendistro-anomaly-detector*", ".opendistro-anomaly-checkpoints", ".opendistro-anomaly-detection-state", ".opendistro-reports-*", ".opendistro-notifications-*", ".opendistro-notebooks", ".opensearch-observability", ".opendistro-asynchronous-search-response*", ".replication-metadata-store"]

/etc/sysconfig/opensearch

# Notification for systemd
OPENSEARCH_SD_NOTIFY=true

@c-bordon c-bordon mentioned this issue Jul 27, 2022
Merged
30 tasks
@alberpilot alberpilot changed the title Update indexer base with Opensearch 2.0.1 Update indexer base with Opensearch 2.1.0 Jul 29, 2022
@c-bordon c-bordon closed this as completed Aug 2, 2022
@MiguelCasaresRobles MiguelCasaresRobles mentioned this issue Aug 8, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@c-bordon c-bordon

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@c-bordon