Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Manage index settings in Vespa application package #921

Merged
merged 50 commits into from
Oct 3, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
50 commits
Select commit Hold shift + click to select a range
f78bf31
Serve index settings from json files via a custom request handler
papa99do Jul 20, 2024
e281609
Fix the jackson version issue
papa99do Jul 21, 2024
30f8e3c
Something that's working
papa99do Jul 21, 2024
71f242a
Model vespa application package.
papa99do Jul 22, 2024
718954f
Add update component logic
papa99do Jul 22, 2024
723ef06
Application package is now complete
papa99do Jul 23, 2024
1a201b2
Handle version conflict in IndexSettingStore
papa99do Jul 23, 2024
2c14761
Change some method signature to be more easily used by index management
papa99do Jul 23, 2024
9ccface
Change some method signature to be more easily used by index management
papa99do Jul 23, 2024
63adda1
Index management change is here finally
papa99do Jul 23, 2024
cc03ae6
Do not need index_setting_dao
papa99do Jul 23, 2024
de9f427
ignore validation-overrides.xml changes
papa99do Jul 26, 2024
f70bc3b
Integration test the new approach
papa99do Jul 26, 2024
c3e1c4d
Prepare for the Vespa deployment change
papa99do Jul 26, 2024
8ed29d8
Add the Method check
papa99do Jul 28, 2024
7d75faa
Use the deployment session to handle race condition during deployment
papa99do Jul 28, 2024
97c5246
Add a fix me to revert to the previous name.
papa99do Jul 28, 2024
bfcaa7b
fix format
papa99do Jul 28, 2024
1101532
Fix the test dir issue
papa99do Jul 29, 2024
d346af7
Bootstrap app package if not configured when creating/deleting indexes
papa99do Jul 29, 2024
0f8944d
Bootstrap app package before creating any indexes
papa99do Jul 29, 2024
ba8a8ed
Add unit test for IndexSettingStore
papa99do Jul 30, 2024
3ce9b78
Made some improvements to the save_file method
papa99do Jul 30, 2024
f6b7801
Extract constants and add more comments
papa99do Jul 31, 2024
668778d
backup and rollback WIP
papa99do Aug 1, 2024
e28d1af
implement the backup and rollback
papa99do Aug 4, 2024
211e793
Add rollback validation
papa99do Aug 4, 2024
4ae683d
minor changes
papa99do Aug 4, 2024
2bd2770
Handle local deployment session
papa99do Aug 5, 2024
3929044
test rollback
papa99do Aug 5, 2024
eff8c80
Fixed index management tests
papa99do Aug 5, 2024
99e73a1
Fixed minor bugs
papa99do Aug 5, 2024
abd097d
Added test coverage report
papa99do Aug 6, 2024
be7f327
Added unit test for services xml
papa99do Aug 6, 2024
63d0192
Add some more unit tests
papa99do Aug 6, 2024
ced8027
Add some more tests
papa99do Aug 6, 2024
60bb33a
An api to roll back vespa to the previous version
papa99do Aug 6, 2024
5a3f9a5
Add info level logging for bootstrapping and rollback
papa99do Aug 6, 2024
0fd48cf
Fix the bug
papa99do Aug 6, 2024
e50e52e
Make the bootstrapping work for both old and new Vespa versions
papa99do Sep 13, 2024
2a3b614
improve the test
papa99do Sep 16, 2024
1e40c80
Fail the Marqo application when Vespa bootstrapping fails
papa99do Oct 1, 2024
857521f
expose rollback-vespa API endpoint
papa99do Oct 1, 2024
57bdfb1
did some renaming
papa99do Oct 1, 2024
5b84c4d
Moved the logic in context managers to VespaApplicationPackage
papa99do Oct 1, 2024
0874dc7
Test both VespaApplicationStore implementation, increased the test co…
papa99do Oct 2, 2024
9dfa0f7
Address the review comments
papa99do Oct 2, 2024
e42850b
Rename a method to reflect it's a legacy method for migration purpose…
papa99do Oct 2, 2024
be50f19
Simplify check of bootstrapping logic.
papa99do Oct 2, 2024
67bdde9
Fix the tests
papa99do Oct 2, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions scripts/vespa_local/vespa_local.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

import os

VESPA_SERVER_VERSION='8.396.18' # version baked into marqo-base:29
VESPA_VERSION=os.getenv('VESPA_VERSION', '8.396.18') # default version baked into marqo-base:30


def start(args):
Expand All @@ -12,7 +12,7 @@ def start(args):
"--name vespa "
"--hostname vespa-container "
"--publish 8080:8080 --publish 19071:19071 --publish 2181:2181 --publish 127.0.0.1:5005:5005 "
f"vespaengine/vespa:{VESPA_SERVER_VERSION}")
f"vespaengine/vespa:{VESPA_VERSION}")


def restart(args):
Expand Down
5 changes: 5 additions & 0 deletions src/marqo/api/exceptions.py
Original file line number Diff line number Diff line change
Expand Up @@ -205,6 +205,11 @@ class OperationConflictError(__InvalidRequestError):
status_code = HTTPStatus.CONFLICT


class ApplicationRollbackError(__InvalidRequestError):
code = "application_rollback_error"
status_code = HTTPStatus.CONFLICT


# ---MARQO INTERNAL ERROR---


Expand Down
12 changes: 12 additions & 0 deletions src/marqo/core/exceptions.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,18 @@ class InternalError(MarqoError):
pass


class ApplicationNotInitializedError(MarqoError):
"""
This exception is raised when the Vespa application is not bootstrapped when receiving
index operation requests.
"""
pass


class ApplicationRollbackError(MarqoError):
pass


class IndexExistsError(MarqoError):
pass

Expand Down
626 changes: 139 additions & 487 deletions src/marqo/core/index_management/index_management.py

Large diffs are not rendered by default.

Loading
Loading