-
Notifications
You must be signed in to change notification settings - Fork 51
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Add support for custom storage endpoints
Adds a new `CUSTOM` variant of storage mappings, which allows catalogs to use a variety of S3-compatible storage services by specifying the `endpoint` explicitly. This is not yet directly exposed to end-users, since `storageMappings` are handled by the control plane. But it does give us the ability to use custom storage endpoints by configuring the storage mappings using something like: ``` {"stores":[{"provider":"CUSTOM","bucket":"the-bucket","endpoint":"some.storage.endpoint.com"}]} ``` Credentials are handled by using the tenant name of each task or collection as a `profile` in the journal storage URI. This profile is understood by the brokers and is looked up in `~/.aws/credentials` and `~/.aws/config` to provide the credentials and region configuration. In order to prevent any `CUSTOM` storage endpoints from using the `default` aws config values, an additional validation was added to ensure that tenant names cannot be `default`. One thing to point is that the catalog JSON schema now isn't able to mark the "provider" field as required, due to SchemaRS lacking [support for internally tagged enums](GREsau/schemars#39). I'm thinking this isn't actually a huge deal since end users don't edit storage mappings in catalog specs anyway. So I'm inclined to leave that as it is for right now.
- Loading branch information
Showing
8 changed files
with
380 additions
and
78 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.