external index router

[var77]

• rename function external_index_receive_index_part to external_index_receive_all
• receive protocol version on the first message and compare with the one defined in client
• receive server type and check if it is router server receive new external indexing server address
• add function wait_for_data which will asynchronously check for data to be available for reading on socket. This will let us to handle interrupts while waiting for data
• now after error magic bytes, the length of the message will be returned and we will try to receive the error message from external indexing server

[greptile-apps]

PR Summary

This pull request implements significant changes to the external indexing system in Lantern, focusing on improved communication protocols and error handling for index data retrieval.

• Introduced protocol version checking and server type verification in src/hnsw/external_index_socket.c to ensure compatibility and proper routing
• Renamed external_index_receive_index_part to external_index_receive_all across multiple files, suggesting a shift to more efficient bulk data retrieval
• Added wait_for_data function in src/hnsw/external_index_socket.c for asynchronous data availability checks, improving interrupt handling
• Implemented error message retrieval from external indexing server after receiving error magic bytes in src/hnsw/external_index_socket.c
• Updated src/hnsw/external_index_socket.h with new constants for protocol version and router server type, supporting the enhanced communication system

_{4 file(s) reviewed, no comment(s)
Edit PR Review Bot Settings}

[codecov]

Codecov Report

Attention: Patch coverage is 69.00585% with 53 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
src/hnsw/external_index_socket.c	65.56%	52 Missing ⚠️
src/hnsw/build.c	94.11%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

[github-actions]

Benchmarks

metric	old	new	pct change
recall (after create)	0.953	0.953	-
recall (after insert)	0.000	0.000	-
select tps	13858.413	16748.081	+20.85%
select bulk(100) tps	38.098	37.802	-0.78%
select latency (ms)	0.894 ± 2.113𝜎	0.753 ± 1.839𝜎	-15.77%
select bulk(100) latency (ms)	816.271 ± 118.079𝜎	832.925 ± 111.395𝜎	+2.04%
create latency (ms)	403156.600	410356.892	+1.79%
insert tps	454.242	450.753	-0.77%
insert bulk(100) tps	4.792	4.767	-0.52%
insert latency (ms)	69.780 ± 15.693𝜎	70.386 ± 15.583𝜎	+0.87%
insert bulk(100) latency (ms)	6541.134 ± 137.785𝜎	6591.057 ± 103.696𝜎	+0.76%
disk usage (bytes)	8192008192.000	8192008192.000	-