Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Updating data sources docs # 7680 #7800

Merged
Merged
Show file tree
Hide file tree
Changes from 24 commits
Commits
Show all changes
40 commits
Select commit Hold shift + click to select a range
9a13153
Updating data sources docs # 7680
AntonEliatra Jul 23, 2024
ceeb9f0
Update connect-prometheus.md
AntonEliatra Jul 23, 2024
50969ec
Update data-sources.md
AntonEliatra Jul 24, 2024
e58b02f
Merge branch 'main' into adding-data-sources-documentation
hdhalter Jul 30, 2024
a3b484a
removing the screenshot from data sources
AntonEliatra Aug 7, 2024
988a2a3
Merge branch 'main' into adding-data-sources-documentation
Naarcha-AWS Aug 12, 2024
3f85568
Revise text for accuracy and readability
vagimeli Aug 13, 2024
a5fa35f
Copy edits to headings
vagimeli Aug 13, 2024
0501b54
Revise lines 39-42
vagimeli Aug 13, 2024
2b43758
Merge branch 'main' into adding-data-sources-documentation
vagimeli Aug 15, 2024
991ff62
Merge branch 'main' into adding-data-sources-documentation
vagimeli Aug 20, 2024
6aa614b
Update connect-prometheus.md
vagimeli Aug 21, 2024
7aae9dd
Update _dashboards/management/connect-prometheus.md
vagimeli Aug 27, 2024
ceb45ba
Update _dashboards/management/connect-prometheus.md
vagimeli Aug 27, 2024
847be09
Update _dashboards/management/connect-prometheus.md
vagimeli Aug 27, 2024
a0c5520
Update connect-prometheus.md
vagimeli Aug 27, 2024
ae8fa8e
Update _dashboards/management/data-sources.md
vagimeli Aug 27, 2024
aec8bc3
Update _dashboards/management/data-sources.md
vagimeli Aug 27, 2024
51c3a42
Update data-sources.md
vagimeli Aug 27, 2024
532213b
Merge branch 'main' into adding-data-sources-documentation
vagimeli Aug 27, 2024
1d03bec
Update _dashboards/management/connect-prometheus.md
vagimeli Aug 27, 2024
49e5384
Update _dashboards/management/connect-prometheus.md
vagimeli Aug 27, 2024
3daccbf
Update _dashboards/management/connect-prometheus.md
vagimeli Aug 27, 2024
5c9f050
Update _dashboards/management/connect-prometheus.md
vagimeli Aug 27, 2024
f45bf68
Update _dashboards/management/connect-prometheus.md
vagimeli Aug 27, 2024
a09f26d
Update data-sources.md
vagimeli Aug 27, 2024
ff765b7
Address editorial feedback
vagimeli Aug 27, 2024
6824b77
Update connect-prometheus.md
vagimeli Aug 27, 2024
3e8ee23
Update connect-prometheus.md
vagimeli Aug 27, 2024
ae1e46c
Update data-sources.md
vagimeli Aug 27, 2024
e8ed29c
Update _dashboards/management/connect-prometheus.md
vagimeli Aug 27, 2024
17e0e47
Update _dashboards/management/connect-prometheus.md
vagimeli Aug 27, 2024
487aea0
Update connect-prometheus.md
vagimeli Aug 27, 2024
e1a8fda
Update connect-prometheus.md
vagimeli Aug 27, 2024
48db453
Update connect-prometheus.md
vagimeli Aug 27, 2024
2cd1ec4
Update connect-prometheus.md
vagimeli Aug 27, 2024
ec765b0
Update _dashboards/management/data-sources.md
vagimeli Aug 28, 2024
19aba84
Merge branch 'main' into adding-data-sources-documentation
vagimeli Aug 28, 2024
786634d
Update connect-prometheus.md
vagimeli Aug 28, 2024
52ee85d
Merge branch 'main' into adding-data-sources-documentation
vagimeli Aug 28, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
51 changes: 51 additions & 0 deletions _dashboards/management/connect-prometheus.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
---
layout: default
title: Connecting Prometheus to OpenSearch
parent: Data sources
nav_order: 20
---

# Connecting Prometheus to OpenSearch

Check failure on line 8 in _dashboards/management/connect-prometheus.md

View workflow job for this annotation

GitHub Actions / style-job

[vale] reported by reviewdog 🐶 [OpenSearch.HeadingCapitalization] 'Connecting Prometheus to OpenSearch' is a heading and should be in sentence case. Raw Output: {"message": "[OpenSearch.HeadingCapitalization] 'Connecting Prometheus to OpenSearch' is a heading and should be in sentence case.", "location": {"path": "_dashboards/management/connect-prometheus.md", "range": {"start": {"line": 8, "column": 3}}}, "severity": "ERROR"}

vagimeli marked this conversation as resolved.
Show resolved Hide resolved
This documentation covers the key steps to connect Prometheus to OpenSearch using the OpenSearch Dashboards interface, including setting up the data source connection, modifying the connection details, and creating an index pattern for the Prometheus data.

## Prerequisites and permissions

Ensure you have configured the necessary [Prerequisites]({{site.url}}{{site.baseurl}}/dashboards/management/data-sources/#prerequisites) and [Permissions]({{site.url}}{{site.baseurl}}/dashboards/management/data-sources/#permissions).
vagimeli marked this conversation as resolved.
Show resolved Hide resolved

## Create a Prometheus data source connection

A data source connection specifies the parameters needed to connect to a data source. These parameters form a connection string for the data source. Using OpenSearch Dashboards, you can add new **Amazon S3** or **Prometheus** data source connections or manage existing ones.
vagimeli marked this conversation as resolved.
Show resolved Hide resolved

Follow these steps to connect **Prometheus** to OpenSearch:
vagimeli marked this conversation as resolved.
Show resolved Hide resolved

1. From the OpenSearch Dashboards main menu, go to **Management** > **Data sources** > **New data source** > **Prometheus**.

2. From the **Configure Prometheus data source** section:

- Under **Data source details**, provide a title and optional description.
- Under **Prometheus data location**, enter the Prometheus URI.
- Under **Authentication details**, select the appropriate authentication method from the dropdown list and enter the required details:
- **Basic authentication**: Enter a username and password.
- **AWS Signature Version 4**: Specify the **Region**, select the OpenSearch service from the **Service Name** list (**Amazon OpenSearch Service** or **Amazon OpenSearch Serverless**), and enter the **Access Key** and **Secret Key**.
- Under **Query permissions**, choose the role needed to search and index data. If you select **Restricted**, additional fields will become available to configure the required role.

3. Select **Review Configuration** > **Connect to Prometheus** to save your settings. The new connection will appear in the list of data sources.

### Modify a data source connection

To modify a data source connection, select the desired connection from the list on the **Data sources** main page. This will open the **Connection Details** window.

Within the **Connection Details** window, you can edit the **Title** and **Description** fields and then select **Save changes** in the lower-right corner to apply the changes.

To update the **Authentication Method**, choose a different method from the dropdown list, enter the required credentials (if applicable), and then select **Save changes** to apply the changes.

If the selected authentication method is **Username & Password**, you can update the password by choosing **Update stored password** next to the **Password** field. In the pop-up window, enter the new password and confirm it, then select **Update stored password** to save the changes. Select **Test connection** to confirm that the connection is valid.

Check warning on line 43 in _dashboards/management/connect-prometheus.md

View workflow job for this annotation

GitHub Actions / style-job

[vale] reported by reviewdog 🐶 [OpenSearch.Ampersand] Don't use an ampersand in place of 'and' in documentation. Raw Output: {"message": "[OpenSearch.Ampersand] Don't use an ampersand in place of 'and' in documentation.", "location": {"path": "_dashboards/management/connect-prometheus.md", "range": {"start": {"line": 43, "column": 51}}}, "severity": "WARNING"}

For **AWS Signature Version 4** authentication, you can update the credentials by selecting **Update stored AWS credential**. In the pop-up window, enter the new access key and the new secret key. Select **Update stored AWS credential** to save the changes. Select **Test connection** to confirm that the connection is valid.

To delete the data source connection, select the {::nomarkdown}<img src="{{site.url}}{{site.baseurl}}/images/dashboards/trash-can-icon.png" class="inline-icon" alt="delete icon"/>{:/} icon.

## Creating an index pattern

After creating a data source connection, the next step is to create an index pattern for that data source. For more information and a tutorial on index patterns, refer to [Index patterns]({{site.url}}{{site.baseurl}}/dashboards/management/index-patterns/).
54 changes: 12 additions & 42 deletions _dashboards/management/data-sources.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,61 +19,31 @@ Once you have installed OpenSearch and OpenSearch Dashboards, you can use Dashbo

Configuration of the [YAML files]({{site.url}}{{site.baseurl}}/install-and-configure/configuring-opensearch/#configuration-file) and installation of the `dashboards-observability` and `opensearch-sql` plugins is necessary. For more information, see [OpenSearch plugins]({{site.url}}{{site.baseurl}}/install-and-configure/plugins/).

## Permissions

To work with data sources in OpenSearch Dashboards, make sure that the user has been assigned the correct cluster-level [data source permission]({{site.url}}{{site.baseurl}}/security/access-control/permissions#data-source-permissions).



## Create a data source connection

A data source connection specifies the parameters needed to connect to a data source. These parameters form a connection string for the data source. Using Dashboards, you can add new data source connections or manage existing ones.

The following steps guide you through the basics of creating a data source connection:

1. From the OpenSearch Dashboards main menu, select **Management** > **Data sources** > **Create data source connection**. The UI is shown in the following image.
To securely store and encrypt data source connections in OpenSearch, you must add the following configuration to the `opensearch.yml` file on all the nodes:

<img src="{{site.url}}{{site.baseurl}}/images/dashboards/data-source-UI.png" alt="Connecting a data source UI" width="700"/>
`plugins.query.datasources.encryption.masterkey: "YOUR_GENERATED_MASTER_KEY_HERE"`

2. Create the data source connection by entering the appropriate information into the **Connection Details** and **Authentication Method** fields.

- Under **Connection Details**, enter a title and endpoint URL. For this tutorial, use the URL `http://localhost:5601/app/management/opensearch-dashboards/dataSources`. Entering a description is optional.
The key must be 16, 24, or 32 characters long. You can use the following command to generate a 24-character key:

- Under **Authentication Method**, select an authentication method from the dropdown list. Once an authentication method is selected, the applicable fields for that method appear. You can then enter the required details. The authentication method options are:
- **No authentication**: No authentication is used to connect to the data source.
- **Username & Password**: A basic username and password are used to connect to the data source.
- **AWS SigV4**: An AWS Signature Version 4 authenticating request is used to connect to the data source. AWS Signature Version 4 requires an access key and a secret key.
- For AWS Signature Version 4 authentication, first specify the **Region**. Next, select the OpenSearch service in the **Service Name** list. The options are **Amazon OpenSearch Service** and **Amazon OpenSearch Serverless**. Lastly, enter the **Access Key** and **Secret Key** for authorization.
`openssl rand -hex 12`

After you have populated the required fields, the **Test connection** and **Create data source** buttons become active. You can select **Test connection** to confirm that the connection is valid.
Generating 12 bytes results in a hexadecimal string that is 12 * 2 = 24 characters long.
{: .note}

3. Select **Create data source** to save your settings. The connection is created. The active window returns to the **Data sources** main page, and the new connection appears in the list of data sources.

4. To delete a data source connection, select the checkbox to the left of the data source **Title** and then select the **Delete 1 connection** button. Selecting multiple checkboxes for multiple connections is supported. An example UI is shown in the following image.

<img src="{{site.url}}{{site.baseurl}}/images/dashboards/delete-data-source.png" alt="Deleting a data source UI" width="700"/>

### Modify a data source connection

To make changes to a data source connection, select a connection in the list on the **Data sources** main page. The **Connection Details** window opens.

To make changes to **Connection Details**, edit one or both of the **Title** and **Description** fields and select **Save changes** in the lower-right corner of the screen. You can also cancel changes here. To change the **Authentication Method**, choose a different authentication method, enter your credentials (if applicable), and then select **Save changes** in the lower-right corner of the screen. The changes are saved.

When **Username & Password** is the selected authentication method, you can update the password by choosing **Update stored password** next to the **Password** field. In the pop-up window, enter a new password in the first field and then enter it again in the second field to confirm. Select **Update stored password** in the pop-up window. The new password is saved. Select **Test connection** to confirm that the connection is valid.
## Permissions

When **AWS SigV4** is the selected authentication method, you can update the credentials by selecting **Update stored AWS credential**. In the pop-up window, enter a new access key in the first field and a new secret key in the second field. Select **Update stored AWS credential** in the pop-up window. The new credentials are saved. Select **Test connection** in the upper-right corner of the screen to confirm that the connection is valid.
To work with data sources in OpenSearch Dashboards, make sure that the user has been assigned the correct cluster-level [data source permissions]({{site.url}}{{site.baseurl}}/security/access-control/permissions#data-source-permissions).

To delete the data source connection, select the delete icon ({::nomarkdown}<img src="{{site.url}}{{site.baseurl}}/images/dashboards/trash-can-icon.png" class="inline-icon" alt="delete icon"/>{:/}).
## Types of data streams

## Create an index pattern
To configure data sources through OpenSearch Dashboards, go to **Management** > **Dashboards Management** > **Data sources**. This flow can be used for OpenSearch data stream connections. See [Configuring and using multiple data sources]({{site.url}}{{site.baseurl}}/dashboards/management/multi-data-sources/) for more information.

Once you've created a data source connection, you can create an index pattern for the data source. An _index pattern_ is a template that OpenSearch uses to create indexes for data from the data source. See [Index patterns]({{site.url}}{{site.baseurl}}/dashboards/management/index-patterns/) for more information and a tutorial.
Alternatively, if you are running OpenSearch Dashboards 2.16 or later, go to **Management** > **Data sources**. This flow can be used to connect Amazon Simple Storage Service (Amazon S3) and Prometheus. See [Connecting Amazon S3 to OpenSearch]({{site.url}}{{site.baseurl}}/dashboards/management/S3-data-source/) and [Connecting Prometheus to OpenSearch]({{site.url}}{{site.baseurl}}/dashboards/management/connect-prometheus/) for more information.

## Next steps

- Learn about [managing index patterns]({{site.url}}{{site.baseurl}}/dashboards/management/index-patterns/) through OpenSearch Dashboards.
- Learn about [indexing data using Index Management]({{site.url}}{{site.baseurl}}/dashboards/im-dashboards/index/) through OpenSearch Dashboards.
- Learn about how to connect [multiple data sources]({{site.url}}{{site.baseurl}}/dashboards/management/multi-data-sources/).
- Learn about how to [connect OpenSearch and Amazon S3 through OpenSearch Dashboards]({{site.url}}{{site.baseurl}}/dashboards/management/S3-data-source/).
- Learn about the [Integrations]({{site.url}}{{site.baseurl}}/integrations/index/) tool, which gives you the flexibility to use various data ingestion methods and connect data from the Dashboards UI.

- Learn about the [Integrations]({{site.url}}{{site.baseurl}}/integrations/index/) plugin, which gives you the flexibility to use various data ingestion methods and connect data to OpenSearch Dashboards.
Binary file removed images/dashboards/data-source-UI.png
Binary file not shown.
Binary file removed images/dashboards/delete-data-source.png
Binary file not shown.
Loading