dataio-test

dataio-core

Elastic scaling is a library that allows to control the number of resources (executors) instantiated by a Spark Structured Streaming Job in order to optimize the effective microbatch duration. The goal is to use the minimum needed resources for processing the workload, in a time that is the closest to the microbatch duration itself, adapting the number of executors to the volume of data received.