feat: add tdigest data structure for statistics (#71)

This pull request implements a simplified version of [Ted Dunning's
TDigest algorithm](https://arxiv.org/pdf/1902.04023.pdf) for efficient
quantile/cdf computations.

It supports a fully parallelizable, memory-bounded computation scheme,
along with an easy API.

Simplified for two reasons:
- The linear interpolation is only done for the quantile, not the cdf.
- Linear interpolation in general could be done more efficiently if we
leverage unit-weighted centroids at the edges, as described in the
paper.
 
Both are marked as TODOs in the code for future work.

Most open-source implementations found online had bugs, were incomplete,
or were too complex (i.e., **poorly written**).

Includes unit tests on uniform and weighted distributions.

Next steps: Implementing the HyperLogLog algorithm for NDistinct.

Cargo.toml

@@ -6,5 +6,6 @@ members = [
     "optd-datafusion-repr",
     "optd-sqlplannertest",
     "optd-adaptive-demo",
+    "gungnir",
 ]
 resolver = "2"

README.md

@@ -37,6 +37,8 @@ The documentation is available in the mdbook format in the [docs](docs) director
 * `optd-datafusion-repr`: Representation of Apache Arrow Datafusion plan nodes in optd.
 * `optd-adaptive-demo`: Demo of adaptive optimization capabilities of optd. More information available in the [docs](docs/).
 * `optd-sqlplannertest`: Planner test of optd based on [risinglightdb/sqlplannertest-rs](https://github.com/risinglightdb/sqlplannertest-rs).
+* `gungnir`: Scalable, memory-efficient, and parallelizable statistical methods for cardinality estimation (e.g. TDigest, HyperLogLog).
+
 
 # Related Works
 

gungnir/Cargo.toml

@@ -0,0 +1,11 @@
+[package]
+name = "gungnir"
+version = "0.1.0"
+edition = "2021"
+
+# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
+
+[dependencies]
+itertools = "0.11"
+rand = "0.8"
+crossbeam = "0.8"

gungnir/src/lib.rs

@@ -0,0 +1,3 @@
+#![allow(clippy::new_without_default)]
+
+pub mod stats;

gungnir/src/stats.rs

@@ -0,0 +1 @@
+mod tdigest;