Skip to content
This repository has been archived by the owner on Feb 27, 2024. It is now read-only.
/ ds2 Public archive

DS2 is an auto-scaling controller for distributed streaming dataflows

License

Apache-2.0, MIT licenses found

Licenses found

Apache-2.0
LICENSE-APACHE
MIT
LICENSE-MIT
Notifications You must be signed in to change notification settings

strymon-system/ds2

Repository files navigation

DS2: fast, accurate, automatic scaling decisions for distributed streaming dataflows.

DS2 is a low-latency controller for dynamic scaling of streaming analytics applications. It can accurately estimate parallelism for all dataflow operators within a single scaling decision, and operates reactively online. DS2 bases scaling decisions on real-time performance traces which it collects through lightweight system-level instrumentation.

This repository contains the following DS2 components:

  • The Scaling policy implements the scaling model and estimates operator parallelism using metrics collected by the reference system instrumentation.
  • The Scaling manager periodically invokes the policy when metrics are available and sends scaling commands to the reference stream processor.
  • The Apache Flink 1.4.1 instrumentation patch contains the necessary instrumentation for Flink to be integrated with DS2.

DS2 can be integrated with any dataflow stream processor as long as it can provide the instrumentation and metrics that its scaling policy requires. For more details on required metrics and integration, please see the OSDI'18 paper.

Building DS2

These instructions assume a Unix-like system.

  1. DS2 is mainly developed in Rust. You can install Rust by following these instructions.

  2. Compile the code (dependencies will be fetched automatically):

    $ cd controller/
    $ cargo build --release --all

Executing DS2

There are two ways to execute DS2: online and offline. To run DS2 online, follow the steps below:

  1. Set up the DS2 configuration parameters in the ds2.toml file

  2. Go to controller/ and start the scaling manager:

$ cargo run --release --bin manager

On success, the scaling manager starts monitoring the specified metrics repository and performs the following actions:

  • Updates its state every time a new rate file is created by the instrumented system. The scaling manager expects exactly one rate file per operator instance and time window (epoch) following the naming convention "some_name-epoch_number.file_extension". Example rate files generated with this patch for Flink can be found here.
  • Invokes the scaling policy periodically according to the particular configuration.
  • Re-configures the streaming system automatically. To start and re-configure the running system, you must write two respective bash scripts, such as these scripts for Flink.

DS2 scaling policy can also be invoked offline on a collection of metrics generated during the execution of a dataflow. To do so, you need:

  • A file containing the configuration of the executed dataflow.
  • A log file with the collected operator rates for one or more policy intervals (epochs).

As an example, go to controller/ and run:

$ cargo run --release --bin policy -- --topo examples/offline/flink_wordcount_topology.csv --rates examples/offline/flink_rates.log --system flink

This command evaluates the scaling policy on the --topo topology for each epoch included in the --rates file assuming Flink as the streaming system.

For more information about offline execution parameters, try --help as follows:

$ cargo run --release --bin policy -- --help

Example input files for both online and offline DS2 execution are provided here, along with details on their format.

Documentation

The complete DS2 documentation can be found here.

License

DS2 is primarily distributed under the terms of both the MIT license and the Apache License (Version 2.0). See LICENSE-APACHE, and LICENSE-MIT for details.

About

DS2 is an auto-scaling controller for distributed streaming dataflows

Resources

License

Apache-2.0, MIT licenses found

Licenses found

Apache-2.0
LICENSE-APACHE
MIT
LICENSE-MIT

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published