[BUG] ISM should honor plugins namespace more strictly

[r1walz]

What is the bug?

ISM setting exists between two namespaces, plugins. and opendistro.. opendistro namespace is reminiscence but still not fully deprecated as some of the settings are required for migration purposes.

ISM utilizes index_state_management.auto_manage setting to get to know which indices to manage. But, since the setting exists under two namespaces plugins. and opendistro., if someone does not want ISM to manage the index, they need to disable setting under both namespaces ...

index-management/src/main/kotlin/org/opensearch/indexmanagement/IndexManagementPlugin.kt

Line 499 in 417d0d9

ManagedIndexSettings.INDEX_STATE_MANAGEMENT_ENABLED,

index-management/src/main/kotlin/org/opensearch/indexmanagement/IndexManagementPlugin.kt

Line 537 in 417d0d9

LegacyOpenDistroManagedIndexSettings.INDEX_STATE_MANAGEMENT_ENABLED,

... as disabling only one is going to violate the following check:

index-management/src/main/kotlin/org/opensearch/indexmanagement/indexstatemanagement/ManagedIndexCoordinator.kt

Lines 334 to 338 in 417d0d9

	val autoManage = if (AUTO_MANAGE.get(indexMetadata.settings)) {
	true
	} else {
	LegacyOpenDistroManagedIndexSettings.AUTO_MANAGE.get(indexMetadata.settings)
	}

How can one reproduce the bug?

Steps to reproduce the behavior:

# create ISM policy
$ http PUT :9200/_plugins/_ism/policies/sample policy:='{
    "description": "sample policy",
    "default_state": "hot",
    "states": [{
        "name": "hot",
        "actions": [{}]
    }],
    "ism_template": {
        "index_patterns": ["sample-idx-*"],
        "priority": 100
    }
}'

# create index matching policy index pattern
$ http PUT :9200/sample-idx-001

[2023-12-01T09:54:54,644][INFO ][o.o.i.i.ManagedIndexCoordinator] [integTest-0] Index [sample-idx-001] matched ISM policy template and will be managed by sample

# ISM manages index sample-idx-001

# create index matching policy index pattern with plugins.*.auto_manage disabled
$ http PUT :9200/sample-idx-002 settings:='{
    "index": {
        "plugins.index_state_management.auto_manage": "false"
    }
}'

[2023-12-01T09:54:54,644][INFO ][o.o.i.i.ManagedIndexCoordinator] [integTest-0] Index [sample-idx-002] matched ISM policy template and will be managed by sample

# ISM manages index sample-idx-002

# create index matching policy index patter with both auto_manage disabled
$ http PUT :9200/sample-idx-003 settings:='{
    "index": {
        "plugins.index_state_management.auto_manage": "false",
        "opendistro.index_state_management.auto_manage": "false"
    }
}'

... empty logs ...

# ISM does not manage index sample-idx-003

What is the expected behavior?

Can we know why opendistro still exists, other than the reason mentioned above? Is there any plan to completely deprecate opendistro namespace? Can we somehow alias the opendistro namespace over plugins namespace, so that we don't have to manually set auto_manage to both namespaces. There could be many such checks which try to honor both namespaces and fail.

What is your host/environment?

• Amazon EC2 with default Ubuntu LTS AMI

Do you have any screenshots?

• N/A

Do you have any additional context?

• N/A