Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Batch index_metadata calls in indexing service #4921

Merged
merged 1 commit into from
May 9, 2024

Conversation

guilload
Copy link
Member

@guilload guilload commented Apr 27, 2024

Description

Add a batch indexes_metadata API to the metastore and use it in the indexing service when spawning new indexing pipelines upon applying a new indexing plan.

How was this PR tested?

Added unit test

@guilload guilload changed the title WIP Batch index_metadata call in indexing service Apr 27, 2024
@guilload guilload changed the title Batch index_metadata call in indexing service Batch index_metadata calls in indexing service Apr 27, 2024
@guilload guilload force-pushed the guilload/batch-index-metadata-metastore-calls branch 2 times, most recently from 82b74e2 to 5724e0c Compare April 28, 2024 00:54
@guilload guilload marked this pull request as ready for review April 28, 2024 00:56
Copy link
Contributor

@fulmicoton fulmicoton left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See comments... the PR can be tested in this week airmail test but I think the DELETE statement has something fishy (which was already here though)

@fulmicoton
Copy link
Contributor

This PR showed a regression on simian so I pulled it off of today's test with airmail.

With - chico(rate=300kb)*1000 on 8 indexers and with postgres, I cannot go above 200MB/s.

This was working on qw-airmail-20240423

@guilload
Copy link
Member Author

guilload commented May 6, 2024

Would you happen to have a clue as to what this PR is breaking?

@fulmicoton
Copy link
Contributor

@guilload no clue. I did not dig further. I can redo the test if you want.

@guilload guilload force-pushed the guilload/batch-index-metadata-metastore-calls branch 3 times, most recently from 5f21d28 to e5578bd Compare May 8, 2024 16:28
@guilload guilload requested a review from fulmicoton May 8, 2024 17:42
@fulmicoton
Copy link
Contributor

I confirm it now works normally.

@guilload guilload force-pushed the guilload/batch-index-metadata-metastore-calls branch 3 times, most recently from dc6e697 to e1bc348 Compare May 9, 2024 15:55
@guilload guilload force-pushed the guilload/batch-index-metadata-metastore-calls branch from e1bc348 to 3626621 Compare May 9, 2024 16:07
@guilload guilload merged commit f3c5528 into main May 9, 2024
5 checks passed
@guilload guilload deleted the guilload/batch-index-metadata-metastore-calls branch May 9, 2024 16:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants