Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Discover Next] Dataset Dropdown #7289

Closed
wants to merge 24 commits into from

Conversation

sejli
Copy link
Member

@sejli sejli commented Jul 17, 2024

Description

Adds a new UI component to support datasets in Discover.

Duplicate of #7267, clean branch with rebase

Issues Resolved

Screenshot

Testing the changes

Changelog

  • feat: Add DataSet dropdown with index patterns and indices

Check List

  • All tests pass
    • yarn test:jest
    • yarn test:jest_integration
  • New functionality includes testing.
  • New functionality has been documented.
  • Update CHANGELOG.md
  • Commits are signed per the DCO using --signoff

Copy link
Contributor

❌ Empty Changelog Section

The Changelog section in your PR description is empty. Please add a valid changelog entry or entries. If you did add a changelog entry, check to make sure that it was not accidentally included inside the comment block in the Changelog section.

Copy link

codecov bot commented Jul 17, 2024

Codecov Report

Attention: Patch coverage is 18.60465% with 70 lines in your changes missing coverage. Please review.

Project coverage is 67.50%. Comparing base (54cdf23) to head (1562935).

Files Patch % Lines
.../public/ui/dataset_navigator/dataset_navigator.tsx 6.94% 67 Missing ⚠️
src/plugins/data/public/ui/settings/settings.ts 88.88% 1 Missing ⚠️
.../data_explorer/public/components/sidebar/index.tsx 0.00% 0 Missing and 1 partial ⚠️
...er/public/utils/state_management/metadata_slice.ts 50.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #7289      +/-   ##
==========================================
- Coverage   67.53%   67.50%   -0.03%     
==========================================
  Files        3504     3503       -1     
  Lines       69407    69453      +46     
  Branches    11324    11330       +6     
==========================================
+ Hits        46874    46885      +11     
- Misses      19773    19810      +37     
+ Partials     2760     2758       -2     
Flag Coverage Δ
Linux_1 33.12% <16.04%> (-0.03%) ⬇️
Linux_2 55.46% <ø> (ø)
Linux_3 ?
Linux_4 34.75% <18.07%> (-0.02%) ⬇️
Windows_1 33.14% <16.04%> (-0.03%) ⬇️
Windows_2 55.41% <ø> (ø)
Windows_3 43.01% <16.66%> (-0.06%) ⬇️
Windows_4 34.75% <18.07%> (-0.02%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Contributor

❌ Empty Changelog Section

The Changelog section in your PR description is empty. Please add a valid changelog entry or entries. If you did add a changelog entry, check to make sure that it was not accidentally included inside the comment block in the Changelog section.

@kavilla
Copy link
Member

kavilla commented Jul 18, 2024

  • todo: reset the dataset when the enhancements disabled

kavilla
kavilla previously approved these changes Jul 18, 2024
Signed-off-by: Sean Li <[email protected]>
@Swiddis
Copy link
Contributor

Swiddis commented Jul 19, 2024

I tried pulling this -- it's not letting me edit the async queries in discover even with query:dataSource:readOnly disabled. Is this still WIP or otherwise only affecting me?

@Swiddis
Copy link
Contributor

Swiddis commented Jul 19, 2024

^ Looks like this is also affecting main so not just this PR

import { IIndexPattern } from '../..';
import { getUiService, getIndexPatterns, getSearchService } from '../../services';

const getClusters = async (savedObjectsClient: SavedObjectsClientContract) => {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should this be getDataSources?

});
};

export const searchResponseToArray = (showAllIndices: boolean) => (response) => {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit. "all" indices might be ambiguous, would it be better with includeHiddenIndices?

and there's the IOpenSearchDashboardsSearchResponse type although raw response is still any by default

Suggested change
export const searchResponseToArray = (showAllIndices: boolean) => (response) => {
export const searchResponseToArray = (showAllIndices: boolean) => (response: IOpenSearchDashboardsSearchResponse) => {

Comment on lines +55 to +62
aggs: {
indices: {
terms: {
field: '_index',
size: 100,
},
},
},
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why not just cat indices?

if (showAllIndices) {
return true;
} else {
return !indexName.startsWith('.');
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

probably doesn't matter but can this filtering be done in the request rather than post processing?

.map((indexName: string) => {
return {
name: indexName,
// item: {},
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this comment useful?

const [isDataSetNavigatorOpen, setIsDataSetNavigatorOpen] = useState(false);
const [clusterList, setClusterList] = useState<SimpleSavedObject[]>([]);
const [indexList, setIndexList] = useState<DataSetOption[]>([]);
const [selectedCluster, setSelectedCluster] = useState<any>();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what exactly is a cluster? is this a dataSource?

const [selectedCluster, setSelectedCluster] = useState<any>();
const [selectedDataSet, setSelectedDataSet] = useState<DataSetOption>({
id: indexPatterns[0]?.id,
name: indexPatterns[0]?.title,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

indexPatterns[number] can be a string based on its type, accessing .id and .title of a string will crash?

this.setUserQueryEnhancementsEnabled(this.isEnabled);
this.enhancedAppNames = this.isEnabled ? this.config.supportedAppNames : [];
this.setSelectedDataSet(this.getSelectedDataSet());
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what does this do?

@@ -18,7 +17,7 @@ export function registerDataSourceConnectionsRoutes(router: IRouter) {
},
async (context, request, response) => {
const fields = ['id', 'title', 'auth.type'];
const resp = await context.core.savedObjects.client.find<DataSourceAttributes>({
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why removing this?

import { API } from '../../../common';
import { Connection, ConnectionsServiceDeps } from '../../types';
import { API } from '../../common';
import { Connection, ConnectionsServiceDeps } from '../types';

export class ConnectionsService {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this still used? can it be removed?

kavilla pushed a commit to kavilla/OpenSearch-Dashboards-1 that referenced this pull request Jul 20, 2024
running linter

Signed-off-by: Sean Li <[email protected]>

removing connections bar component

Signed-off-by: Sean Li <[email protected]>

remove unused property

Signed-off-by: Sean Li <[email protected]>

use default search interceptor for get indices

Signed-off-by: Sean Li <[email protected]>

removing unused file

Signed-off-by: Sean Li <[email protected]>

adding onselect for dataset navigator

Signed-off-by: Sean Li <[email protected]>

trying to make a commponetto call

Signed-off-by: Kawika Avilla <[email protected]>

no clue if it works

Signed-off-by: Kawika Avilla <[email protected]>

trying out changes

Signed-off-by: Sean Li <[email protected]>
kavilla pushed a commit to kavilla/OpenSearch-Dashboards-1 that referenced this pull request Jul 22, 2024
@sejli
Copy link
Member Author

sejli commented Jul 23, 2024

Closing for #7368

@sejli sejli closed this Jul 23, 2024
kavilla added a commit to kavilla/OpenSearch-Dashboards-1 that referenced this pull request Jul 23, 2024
* initial implementation for dataset dropdown

Signed-off-by: Sean Li <[email protected]>

* initial metadata commit

Signed-off-by: Sean Li <[email protected]>

* bug fixes

Signed-off-by: Sean Li <[email protected]>

* hiding datasource cluster selctor and other bug fixes

Signed-off-by: Sean Li <[email protected]>

* refactoring and fixing datasource selector button

Signed-off-by: Sean Li <[email protected]>

* fix breaking loop

Signed-off-by: Sean Li <[email protected]>

* last working commit

Signed-off-by: Sean Li <[email protected]>

* WIP indices

Signed-off-by: Sean Li <[email protected]>

* refactor to query editor

Signed-off-by: Sean Li <[email protected]>

* cleanup dataset dropdown

Signed-off-by: Sean Li <[email protected]>

* running linter

Signed-off-by: Sean Li <[email protected]>

* removing exports for deleted components

Signed-off-by: Sean Li <[email protected]>

* Changeset file for PR opensearch-project#7289 created/updated

* running linter

Signed-off-by: Sean Li <[email protected]>

* removing connections bar component

Signed-off-by: Sean Li <[email protected]>

* remove unused property

Signed-off-by: Sean Li <[email protected]>

* use default search interceptor for get indices

Signed-off-by: Sean Li <[email protected]>

* removing unused file

Signed-off-by: Sean Li <[email protected]>

* adding onselect for dataset navigator

Signed-off-by: Sean Li <[email protected]>

* trying to make a commponetto call

Signed-off-by: Kawika Avilla <[email protected]>

* no clue if it works

Signed-off-by: Kawika Avilla <[email protected]>

* trying out changes

Signed-off-by: Sean Li <[email protected]>

* initial commit for external datasources work

Signed-off-by: Sean Li <[email protected]>

* caching recently used and external datasources

Signed-off-by: Sean Li <[email protected]>

* still porting over

Signed-off-by: Kawika Avilla <[email protected]>

* more ported over

Signed-off-by: Kawika Avilla <[email protected]>

* no linter complaints

Signed-off-by: Kawika Avilla <[email protected]>

* update to sidebar

Signed-off-by: Kawika Avilla <[email protected]>

* handle import errors

Signed-off-by: Kawika Avilla <[email protected]>

* rebased

Signed-off-by: Kawika Avilla <[email protected]>

* fix a messy merge for package.json

Signed-off-by: Kawika Avilla <[email protected]>

* so lnse for catching this

Signed-off-by: Kawika Avilla <[email protected]>

* fix nav

Signed-off-by: Kawika Avilla <[email protected]>

* no recalling

Signed-off-by: Kawika Avilla <[email protected]>

* number panels

Signed-off-by: Kawika Avilla <[email protected]>

* add wait

Signed-off-by: Kawika Avilla <[email protected]>

* index patterns worknig agan

Signed-off-by: Kawika Avilla <[email protected]>

* fix and clearing it df out

Signed-off-by: Kawika Avilla <[email protected]>

* disable hooks dependency

Signed-off-by: Kawika Avilla <[email protected]>

* fix the issue with import

Signed-off-by: Kawika Avilla <[email protected]>

* fixed passing the data source id

Signed-off-by: Kawika Avilla <[email protected]>

* loading spinner

Signed-off-by: Kawika Avilla <[email protected]>

* pading

Signed-off-by: Kawika Avilla <[email protected]>

* conditionally show s3

Signed-off-by: Kawika Avilla <[email protected]>

* remove sql

Signed-off-by: Kawika Avilla <[email protected]>

* let it fail

Signed-off-by: Kawika Avilla <[email protected]>

* s3 datasources

Signed-off-by: Kawika Avilla <[email protected]>

* set data set for table

Signed-off-by: Kawika Avilla <[email protected]>

* pass it in the options

Signed-off-by: Kawika Avilla <[email protected]>

* run query different

Signed-off-by: Kawika Avilla <[email protected]>

* dont parse query string

Signed-off-by: Kawika Avilla <[email protected]>

* not capitalization

Signed-off-by: Kawika Avilla <[email protected]>

* Changeset file for PR opensearch-project#7368 created/updated

* connections imported right

Signed-off-by: Kawika Avilla <[email protected]>

* handle select external

Signed-off-by: Kawika Avilla <[email protected]>

* initial implementation for url state

Signed-off-by: Sean Li <[email protected]>

* restore the dataset metadata stuff

Signed-off-by: Kawika Avilla <[email protected]>

* build dataset in state

Signed-off-by: Kawika Avilla <[email protected]>

* null pointer

Signed-off-by: Kawika Avilla <[email protected]>

---------

Signed-off-by: Sean Li <[email protected]>
Signed-off-by: Kawika Avilla <[email protected]>
Co-authored-by: Sean Li <[email protected]>
Co-authored-by: opensearch-changeset-bot[bot] <154024398+opensearch-changeset-bot[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants