Datavines

Data quality is used to ensure the accuracy of data in the process of integration and processing. It is also the core component of DataOps. DataVines is an easy-to-use data quality service platform that supports multiple metric.

Architecture Design

Install

Need: Maven 3.6.1 and later

$ mvn clean package -Prelease -DskipTests

Features

Data Catalog

• Obtain data source metadata regularly to construct data directory
• Regular monitoring of metadata changes
• Tag management with support for metadata

Data Quality

• Built-in 27 data quality check rules
• Support 4 data quality check rule types
  • Single Table-Column Check
  • Single Table Custom SQL check
  • Cross Table Accuracy Check
  • Two Table Value Comparison Check
• Support schedule tasks for check
• Support SLA for check result alert

Data Profile

• Support timing execution of data detection, output data profile report
• Support automatically identify column types to automatically match appropriate data profile indicators
• Support table row number trend monitoring
• Support data distribution view

Plug-in Design

The platform is based on plug-in design, and the following modules support user-defined plug-ins to expand

• Data Source: MySQL, Impala, Starocks, Doris, Presto, Trino, ClickHouse, PostgreSQL are already supported
• Check Rules: 27 check rules such as built-in null value check, non-null check, enumeration check, etc.
• Job Execution Engine: Two execution engines Spark and Local have been supported. The Spark engine currently only supports the Spark2.4 version, and the Local engine is a local execution engine developed based on JDBC, without relying on other execution engines.
• Alert Channel: Supported Email
• Error Data Storage: MySQL and local files are already supported (only Local execution engine is supported)
• Registry: Already supports MySQL, PostgreSQL and ZooKeeper

Multiple Execute Modes

• Provide Web page to configure check jobs, run jobs, view job execution logs, view error data and check results
• Support online generation job running scripts, submit jobs through datavines-submit.sh, can be used in conjunction with the scheduling system

Easy Deployment & High Availability

• Less platform dependency, easy to deploy
• Minimal only rely on MySQL to start the project and complete the check of data quality operations
• Support horizontal expansion, automatic fault tolerance
• Decentralized design, Server node supports horizontal expansion to improve performance
• Job Automatic Fault Tolerance, to ensure that jobs are not lost or repeated

Environmental Dependency

1. java runtime environment: jdk8
2. If the data volume is small, or the goal is merely for functional verification, you can use JDBC engine
3. If you want to run DataVines based on Spark, you need to ensure that your server has spark installed

Quick Start

Click Document for more information

Development

Click Document for more information

Contribution

You can submit any ideas as pull requests or as GitHub issues.

If you're new to posting issues, we ask that you read How To Ask Questions The Smart Way (This guide does not provide actual support services for this project!), How to Report Bugs Effectively prior to posting. Well written bug reports help us help you!

Thank you to all the people who already contributed to Datavines!

License

Datavines is licensed under the Apache License 2.0. Datavines relies on some third-party components, and their open source protocols are also Apache License 2.0 or compatible with Apache License 2.0. In addition, Datavines also directly references or modifies some codes in Apache DolphinScheduler, SeaTunnel and Dubbo, all of which are Apache License 2.0. Thanks for contributions to these projects.

Social Media

• WeChat Official Account (in Chinese, scan the QR code to follow)