Add Digital Ocean spaces storage flavor

[prince-vishal]

Describe the bug
Indexes are being deleted from the metastore but delete from storage fails with the below error

2023-07-13T18:14:14.322Z ERROR quickwit_janitor::garbage_collection: Failed to delete ["01H587B1YVBTFYGX00R0PWQGS6.split", "01H587BFTT9RKSCH7HBEH5DPF3.split", "01H587BAR1R72NGXW5CP0467BM.split", "01H587AVEZ9DD64WWRHZE2C66W.split", "01H587AM881RVSFE7ZHH0TWE2X.split"] and 5 other splits. error=Some(StorageError { kind: Service, source: service error: unhandled error: unhandled error: Error { code: "NotImplemented", message: "", aws_request_id: "tx000003a1d4cb4cab98a3a-0064b03ef6-a26d3-blr1a" } (ServiceError(ServiceError { source: Unhandled(Unhandled { source: ErrorMetadata { code: Some("NotImplemented"), message: Some(""), extras: Some({"aws_request_id": "tx000003a1d4cb4cab98a3a-0064b03ef6-a26d3-blr1a"}) }, meta: ErrorMetadata { code: Some("NotImplemented"), message: Some(""), extras: Some({"aws_request_id": "tx000003a1d4cb4cab98a3a-0064b03ef6-a26d3-blr1a"}) } }), raw: Response { inner: Response { status: 501, version: HTTP/1.1, headers: {"content-length": "206", "x-amz-request-id": "tx000003a1d4cb4cab98a3a-0064b03ef6-a26d3-blr1a", "accept-ranges": "bytes", "content-type": "application/xml", "date": "Thu, 13 Jul 2023 18:14:14 GMT", "cache-control": "max-age=60", "strict-transport-security": "max-age=15552000; includeSubDomains; preload", "x-envoy-upstream-healthchecked-cluster": ""}, body: SdkBody { inner: Once(Some(b"NotImplementedtx000003a1d4cb4cab98a3a-0064b03ef6-a26d3-blr1aa26d3-blr1a-blr1-zg01")), retryable: true } }, properties: SharedPropertyBag(Mutex { data: PropertyBag { contents: ["aws_credential_types::credentials_impl::Credentials", "aws_sig_auth::middleware::Signature", "aws_credential_types::cache::SharedCredentialsCache", "aws_smithy_http::operation::Metadata", "aws_types::SigningService", "aws_smithy_types::endpoint::Endpoint", "alloc::vec::Vechttp::version::Version", "aws_http::user_agent::AwsUserAgent", "aws_types::region::Region", "aws_sdk_s3::endpoint::Params", "aws_sig_auth::signer::OperationSigningConfig", "aws_smithy_http::connection::CaptureSmithyConnection", "aws_types::region::SigningRegion"] }, poisoned: false, .. }) } })) }) index_id="brchs" num_failed_splits=5

Steps to reproduce (if applicable)
Steps to reproduce the behavior:

1. Create a index
2. Delete the same index

Expected behavior
Indexes are deleted from the DO storage too.

Configuration:

config.yaml

version: 0.6
node_id: searcher-1
listen_address: 0.0.0.0
metastore_uri: postgresql://adfd:[email protected]:25061/main
default_index_root_uri: /indexes
storage:
s3:
endpoint: https://dfad-adfadf.blr1.digitaloceanspaces.com/
force_path_style_access: true

1. Output of quickwit --version
   Quickwit v0.6.1 (798ad68 2023-06-12T06:56:37Z)

2. The index_config.yaml

version: "0.6"
index_id: "brchs"
index_uri: "s3://indexes/brchs"

doc_mapping:
mode: dynamic
field_mappings:
- name: sfhfhsfhhhh
type: text
fast: true
record: position
- name: sfhfhsfh
tokenizer: raw
record: position
type: text
fast: true
- name: fshfhsfh
tokenizer: raw
record: position
type: text
fast: true
- name: fhsf
tokenizer: raw
record: position
type: text
fast: true
- name: abdbfa
type: u64
fast: true
- name: created_at
description: "If anything exists in the DB, then add it here"
type: datetime
fast: true
- name: indexed_at
description: "Datetime when this object was indexed"
type: datetime
fast: true
tag_fields: ["sfhfhsfh", "fhsf"]
partition_key: breach_name
timestamp_field: "indexed_at"

search_settings:
default_search_fields: ["fhsf"]

indexing_settings:
commit_timeout_secs: 5
merge_policy:
type: "stable_log"
min_level_num_docs: 100000
merge_factor: 10
max_merge_factor: 12

retention:
period: "3 years"
schedule: monthly

[prince-vishal]

Setting this param in the config resolved this

disable_multi_object_delete: true

Your config.yaml should look like this

version: 0.6
node_id: searcher-1
listen_address: 0.0.0.0
metastore_uri: postgresql://dffwf:[email protected]:25061/main
default_index_root_uri: s3://indexes
storage:
s3:
endpoint: https://ewffwef-wefewf.blr1.digitaloceanspaces.com/
force_path_style_access: true
disable_multi_object_delete_requests: true

[guilload]

Hi @prince-vishal,

I'm glad you figured it out. We've never tested Quickwit on Digital Ocean before, so it's helpful for us to know which settings are working, so thank you.

We will add a Digital Ocean storage flavor shortly and document it. Again, thanks for your help!

[guilload]

A second more worrisome bug is that we were unable to delete the files, yet we deleted them from the metastore. @prince-vishal, can you confirm?

[trinity-1686a]

quickwit/quickwit/quickwit-janitor/src/garbage_collection.rs

Lines 270 to 313 in ad28cfb

	let delete_result = storage.bulk_delete(&paths).await;
	if let Some(ctx) = ctx_opt {
	ctx.record_progress();
	}
	let mut deleted_split_ids = Vec::new();
	let mut deleted_file_entries = Vec::new();
	match delete_result {
	Ok(()) => {
	for (split_id, entry) in paths_to_splits.into_values() {
	deleted_split_ids.push(split_id);
	deleted_file_entries.push(entry);
	}
	}
	Err(bulk_delete_error) => {
	let num_failed_splits =
	bulk_delete_error.failures.len() + bulk_delete_error.unattempted.len();
	let truncated_split_ids = bulk_delete_error
	.failures
	.keys()
	.chain(bulk_delete_error.unattempted.iter())
	.take(5)
	.collect::<Vec<_>>();
	error!(
	error = ?bulk_delete_error.error,
	index_id = ?index_uid.index_id(),
	num_failed_splits = num_failed_splits,
	"Failed to delete {:?} and {} other splits.",
	truncated_split_ids, num_failed_splits,
	);
	for split_path in bulk_delete_error.successes {
	let (split_id, entry) = paths_to_splits
	.remove(&split_path)
	.expect("The successful split path should be present within the lookup table.");
	deleted_split_ids.push(split_id);
	deleted_file_entries.push(entry);
	}
	}
	};

when deleting splits, we report errors about deleting splits, mark as delete the splits which were successfully deleted, and delete the index from the metastore (implicitly forgetting about any split which wasn't deleted).

[guilload]

Yes, I'm reading the same code right now.

[prince-vishal]

A second more worrisome bug is that we were unable to delete the files, yet we deleted them from the metastore. @prince-vishal, can you confirm?

Yes, it was being deleted from metastore

[guilload]

Closed via #3636 and #3632.