chore: rename to raystack

.github/workflows/package.yml

@@ -45,5 +45,5 @@ jobs:
           context: .
           push: true
           tags: |
-            odpf/firehose:latest
-            odpf/firehose:${{ steps.get_version.outputs.version-without-v }}
+            raystack/firehose:latest
+            raystack/firehose:${{ steps.get_version.outputs.version-without-v }}

Dockerfile

@@ -10,4 +10,4 @@ COPY --from=GRADLE_BUILD ./jolokia-jvm-agent.jar /opt/firehose
 COPY --from=GRADLE_BUILD ./src/main/resources/log4j.xml /opt/firehose/etc/log4j.xml
 COPY --from=GRADLE_BUILD ./src/main/resources/logback.xml /opt/firehose/etc/logback.xml
 WORKDIR /opt/firehose
-CMD ["java", "-cp", "bin/*:/work-dir/*", "io.odpf.firehose.launch.Main", "-server", "-Dlogback.configurationFile=etc/firehose/logback.xml", "-Xloggc:/var/log/firehose"]
+CMD ["java", "-cp", "bin/*:/work-dir/*", "org.raystack.firehose.launch.Main", "-server", "-Dlogback.configurationFile=etc/firehose/logback.xml", "-Xloggc:/var/log/firehose"]

README.md

@@ -1,9 +1,9 @@
 # Firehose
 
-![build workflow](https://github.com/odpf/firehose/actions/workflows/build.yml/badge.svg)
-![package workflow](https://github.com/odpf/firehose/actions/workflows/package.yml/badge.svg)
+![build workflow](https://github.com/raystack/firehose/actions/workflows/build.yml/badge.svg)
+![package workflow](https://github.com/raystack/firehose/actions/workflows/package.yml/badge.svg)
 [![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg?logo=apache)](LICENSE)
-[![Version](https://img.shields.io/github/v/release/odpf/firehose?logo=semantic-release)](Version)
+[![Version](https://img.shields.io/github/v/release/raystack/firehose?logo=semantic-release)](Version)
 
 Firehose is a cloud native service for delivering real-time streaming data to destinations such as service endpoints (HTTP or GRPC) & managed databases (Postgres, InfluxDB, Redis, Elasticsearch, Prometheus and MongoDB). With Firehose, you don't need to write applications or manage resources. It can be scaled up to match the throughput of your data. If your data is present in Kafka, Firehose delivers it to the destination(SINK) that you specified.
 
@@ -25,7 +25,7 @@ Discover why users choose Firehose as their main Kafka Consumer
   - Elasticsearch
   - Redis
   - Bigquery
-  - BigTable  
+  - BigTable
   - Blob Storage/Object Storage :
     - Google Cloud Storage
 
@@ -47,28 +47,28 @@ Explore the following resources to get started with Firehose:
 
 ## Run with Docker
 
-Use the docker hub to download firehose [docker image](https://hub.docker.com/r/odpf/firehose/). You need to have docker installed in your system.
+Use the docker hub to download firehose [docker image](https://hub.docker.com/r/raystack/firehose/). You need to have docker installed in your system.
 
 ```
 # Download docker image from docker hub
-$ docker pull odpf/firehose
+$ docker pull raystack/firehose
 
 # Run the following docker command for a simple log sink.
-$ docker run -e SOURCE_KAFKA_BROKERS=127.0.0.1:6667 -e SOURCE_KAFKA_CONSUMER_GROUP_ID=kafka-consumer-group-id -e SOURCE_KAFKA_TOPIC=sample-topic -e SINK_TYPE=log -e SOURCE_KAFKA_CONSUMER_CONFIG_AUTO_OFFSET_RESET=latest -e INPUT_SCHEMA_PROTO_CLASS=com.github.firehose.sampleLogProto.SampleLogMessage -e SCHEMA_REGISTRY_STENCIL_ENABLE=true -e SCHEMA_REGISTRY_STENCIL_URLS=http://localhost:9000/artifactory/proto-descriptors/latest odpf/firehose:latest
+$ docker run -e SOURCE_KAFKA_BROKERS=127.0.0.1:6667 -e SOURCE_KAFKA_CONSUMER_GROUP_ID=kafka-consumer-group-id -e SOURCE_KAFKA_TOPIC=sample-topic -e SINK_TYPE=log -e SOURCE_KAFKA_CONSUMER_CONFIG_AUTO_OFFSET_RESET=latest -e INPUT_SCHEMA_PROTO_CLASS=com.github.firehose.sampleLogProto.SampleLogMessage -e SCHEMA_REGISTRY_STENCIL_ENABLE=true -e SCHEMA_REGISTRY_STENCIL_URLS=http://localhost:9000/artifactory/proto-descriptors/latest/raystack/firehose:latest
 ```
 
 **Note:** Make sure your protos (.jar file) are located in `work-dir`, this is required for Filter functionality to work.
 
 ## Run with Kubernetes
 
-- Create a firehose deployment using the helm chart available [here](https://github.com/odpf/charts/tree/main/stable/firehose)
+- Create a firehose deployment using the helm chart available [here](https://github.com/raystack/charts/tree/main/stable/firehose)
 - Deployment also includes telegraf container which pushes stats metrics
 
 ## Running locally
 
 ```sh
 # Clone the repo
-$ git clone https://github.com/odpf/firehose.git
+$ git clone https://github.com/raystack/firehose.git
 
 # Build the jar
 $ ./gradlew clean build
@@ -101,11 +101,11 @@ Development of Firehose happens in the open on GitHub, and we are grateful to th
 
 Read our [contributing guide](docs/docs/contribute/contribution.md) to learn about our development process, how to propose bugfixes and improvements, and how to build and test your changes to Firehose.
 
-To help you get your feet wet and get you familiar with our contribution process, we have a list of [good first issues](https://github.com/odpf/firehose/labels/good%20first%20issue) that contain bugs which have a relatively limited scope. This is a great place to get started.
+To help you get your feet wet and get you familiar with our contribution process, we have a list of [good first issues](https://github.com/raystack/firehose/labels/good%20first%20issue) that contain bugs which have a relatively limited scope. This is a great place to get started.
 
 ## Credits
 
-This project exists thanks to all the [contributors](https://github.com/odpf/firehose/graphs/contributors).
+This project exists thanks to all the [contributors](https://github.com/raystack/firehose/graphs/contributors).
 
 ## License
 

build.gradle

@@ -32,15 +32,16 @@ lombok {
     sha256 = ""
 }
 
-group 'io.odpf'
-version '0.7.4'
+group 'org.raystack'
+version '0.8.12'
 
 def projName = "firehose"
 
 sourceCompatibility = JavaVersion.VERSION_1_8
 targetCompatibility = JavaVersion.VERSION_1_8
 
 repositories {
+    mavenLocal()
     mavenCentral()
     jcenter()
     maven {
@@ -54,7 +55,7 @@ private Properties loadEnv() {
     properties
 }
 
-def mainClassName = "io.odpf.firehose.launch.Main"
+def mainClassName = "org.raystack.firehose.launch.Main"
 
 dependencies {
     implementation group: 'com.google.protobuf', name: 'protobuf-java', version: '3.1.0'
@@ -71,7 +72,7 @@ dependencies {
     implementation group: 'org.apache.commons', name: 'commons-jexl', version: '2.1'
     implementation group: 'org.apache.commons', name: 'commons-lang3', version: '3.5'
     implementation group: 'com.google.code.gson', name: 'gson', version: '2.7'
-    implementation group: 'io.odpf', name: 'stencil', version: '0.2.1' exclude group: 'org.slf4j'
+    implementation group: 'org.raystack', name: 'stencil', version: '0.4.0' exclude group: 'org.slf4j'
     implementation group: 'software.amazon.awssdk', name: 's3', version: '2.17.129'
     implementation group: 'org.influxdb', name: 'influxdb-java', version: '2.5'
     implementation group: 'com.jayway.jsonpath', name: 'json-path', version: '2.4.0'
@@ -85,7 +86,7 @@ dependencies {
         exclude group: "log4j", module: "log4j"
     }
     implementation 'io.confluent:monitoring-interceptors:3.0.0'
-    implementation "io.grpc:grpc-all:1.38.0"
+    implementation 'io.grpc:grpc-all:1.53.0'
     implementation group: 'org.jfrog.buildinfo', name: 'build-info-extractor', version: '2.6.3'
     implementation group: 'com.google.gradle', name: 'osdetector-gradle-plugin', version: '1.2.1'
     implementation group: 'org.apache.ivy', name: 'ivy', version: '2.2.0'
@@ -98,10 +99,9 @@ dependencies {
     implementation 'com.gojek.parquet:parquet-hadoop:1.11.9'
     implementation group: 'com.github.os72', name: 'protobuf-dynamic', version: '1.0.1'
     implementation platform('com.google.cloud:libraries-bom:20.5.0')
-    implementation 'com.google.cloud:google-cloud-storage:1.114.0'
-    implementation 'com.google.cloud:google-cloud-bigquery:1.115.0'
-    implementation 'org.apache.logging.log4j:log4j-core:2.17.1'
-    implementation group: 'io.odpf', name: 'depot', version: '0.3.8'
+    implementation 'com.google.cloud:google-cloud-storage:2.20.1'
+    implementation 'org.apache.logging.log4j:log4j-core:2.20.0'
+    implementation group: 'org.raystack', name: 'depot', version: '0.4.0'
     implementation group: 'com.networknt', name: 'json-schema-validator', version: '1.0.59' exclude group: 'org.slf4j'
 
     testImplementation group: 'junit', name: 'junit', version: '4.11'
@@ -146,7 +146,7 @@ test {
         events "passed", "skipped", "failed"
     }
     useJUnit {
-        excludeCategories 'io.odpf.firehose.test.categories.IntegrationTest'
+        excludeCategories 'org.raystack.firehose.test.categories.IntegrationTest'
     }
     doLast {
         delete "$projectDir/src/test/resources/__files"
@@ -158,7 +158,7 @@ clean {
 }
 jar {
     manifest {
-        attributes 'Main-Class': 'io.odpf.firehose.launch.Main'
+        attributes 'Main-Class': 'org.raystack.firehose.launch.Main'
         duplicatesStrategy = 'exclude'
         zip64 = true
     }
@@ -181,7 +181,7 @@ publishing {
     repositories {
         maven {
             name = "GitHubPackages"
-            url = "https://maven.pkg.github.com/odpf/firehose"
+            url = "https://maven.pkg.github.com/raystack/firehose"
             credentials {
                 username = System.getenv("GITHUB_ACTOR")
                 password = System.getenv("GITHUB_TOKEN")

docs/docs/advance/filters.md

@@ -6,43 +6,42 @@ Following variables need to be set to enable JSON/JEXL filters.
 
 Defines whether to use `JSON` Schema-based filters or `JEXL`-based filters or `NO_OP` \(i.e. no filtering\)
 
-* Example value: `JSON`
-* Type: `optional`
-* Default value`: NO_OP`
+- Example value: `JSON`
+- Type: `optional`
+- Default value`: NO_OP`
 
 ## `FILTER_JSON_ESB_MESSAGE_TYPE`
 
 Defines the format type of the input ESB messages, i.e. JSON/Protobuf. This field is required only for JSON filters.
 
-* Example value: `JSON`
-* Type: `optional`
+- Example value: `JSON`
+- Type: `optional`
 
 ## `FILTER_SCHEMA_PROTO_CLASS`
 
 The fully qualified name of the proto schema so that the key/message in Kafka could be parsed.
 
-* Example value: `com.gojek.esb.driverlocation.DriverLocationLogKey`
-* Type: `optional`
+- Example value: `com.raystack.esb.driverlocation.DriverLocationLogKey`
+- Type: `optional`
 
 ## `FILTER_DATA_SOURCE`
 
 `key`/`message`/`none`depending on where to apply filter
 
-* Example value: `key`
-* Type: `optional`
-* Default value`: none`
+- Example value: `key`
+- Type: `optional`
+- Default value`: none`
 
 ## `FILTER_JEXL_EXPRESSION`
 
 JEXL filter expression
 
-* Example value: `driverLocationLogKey.getVehicleType()=="BIKE"`
-* Type: `optional`
+- Example value: `driverLocationLogKey.getVehicleType()=="BIKE"`
+- Type: `optional`
 
 ## `FILTER_JSON_SCHEMA`
 
 JSON Schema string containing the filter rules to be applied.
 
-* Example value: `{"properties":{"order_number":{"const":"1253"}}}`
-* Type: `optional`
-
+- Example value: `{"properties":{"order_number":{"const":"1253"}}}`
+- Type: `optional`

docs/docs/concepts/architecture.md

@@ -40,7 +40,7 @@ _**Sink**_
 - All the existing sink types follow the same contract/lifecycle defined in `AbstractSink.java`. It consists of two stages:
   - Prepare: Transformation over-filtered messages’ list to prepare the sink-specific insert/update client requests.
   - Execute: Requests created in the Prepare stage are executed at this step and a list of failed messages is returned \(if any\) for retry.
-- Underlying implementation of AbstractSink can use implementation present in [depot](https://github.com/odpf/depot).
+- Underlying implementation of AbstractSink can use implementation present in [depot](https://github.com/raystack/depot).
 - If the batch has any failures, Firehose will retry to push the failed messages to the sink
 
 _**SinkPool**_
@@ -71,7 +71,9 @@ The final state of message can be any one of the followings after it is consumed
 One can monitor via plotting the metrics related to messages.
 
 ### Schema Handling
+
 - Incase when `INPUT_SCHEMA_DATA_TYPE is set to protobuf`
+
   - Protocol buffers are Google's language-neutral, platform-neutral, extensible mechanism for serializing structured data. Data streams on Kafka topics are bound to a protobuf schema.
   - Firehose deserializes the data consumed from the topics using the Protobuf descriptors generated out of the artifacts. The artifactory is an HTTP interface that Firehose uses to deserialize.
   - The schema handling ie., find the mapped schema for the topic, downloading the descriptors, and dynamically being notified of/updating with the latest schema is abstracted through the Stencil library.
@@ -81,10 +83,8 @@ One can monitor via plotting the metrics related to messages.
     Schema Caching, dynamic schema updates, etc. are features of the stencil client library.
 
 - Incase when `INPUT_SCHEMA_DATA_TYPE is set to json`
-    - Currently this config is only supported in Bigquery sink,
-    - For json, in bigquery sink the schema is dynamically inferred from incoming data, in future we plan to provide json schema support via stencil.
-
-
+  - Currently this config is only supported in Bigquery sink,
+  - For json, in bigquery sink the schema is dynamically inferred from incoming data, in future we plan to provide json schema support via stencil.
 
 ## Firehose Integration
 

docs/docs/concepts/monitoring.md

@@ -71,11 +71,11 @@ Lastly, set up Telegraf to send metrics to InfluxDB, following the corresponding
 
 #### Firehose deployed on Kubernetes _\*\*_
 
-1. Follow[ this guide](https://github.com/odpf/charts/tree/main/stable/firehose#readme) for deploying Firehose on a Kubernetes cluster using a Helm chart.
-2. Configure the following parameters in the default [values.yaml](https://github.com/odpf/charts/blob/main/stable/firehose/values.yaml) file and run -
+1. Follow[ this guide](https://github.com/raystack/charts/tree/main/stable/firehose#readme) for deploying Firehose on a Kubernetes cluster using a Helm chart.
+2. Configure the following parameters in the default [values.yaml](https://github.com/raystack/charts/blob/main/stable/firehose/values.yaml) file and run -
 
 ```text
-$ helm install my-release -f values.yaml odpf/firehose
+$ helm install my-release -f values.yaml raystack/firehose
 ```
 
 | Key                                              | Type   | Default                     | Description                                   |

docs/docs/concepts/overview.md

@@ -10,7 +10,7 @@ scale. This section explains the overall architecture of Firehose and describes
 ## [Monitoring Firehose with exposed metrics](monitoring.md)
 
 Always know what’s going on with your deployment with
-built-in [monitoring](https://github.com/odpf/firehose/blob/main/docs/assets/firehose-grafana-dashboard.json) of
+built-in [monitoring](https://github.com/raystack/firehose/blob/main/docs/assets/firehose-grafana-dashboard.json) of
 throughput, response times, errors and more. This section contains guides, best practices and advises related to
 managing Firehose in production.
 
@@ -27,18 +27,17 @@ Firehose provides various templating features
 
 Decorators are used for chained processing of messages.
 
-* SinkWithFailHandler
-* SinkWithRetry
-* SinkWithDlq
-* SinkFinal
+- SinkWithFailHandler
+- SinkWithRetry
+- SinkWithDlq
+- SinkFinal
 
 ## [FirehoseConsumer](consumer.md)
+
 A firehose consumer read messages from kafka, pushes those messages to sink and commits offsets back to kafka based on certain strategies.
 
 ## [Offsets](offsets.md)
 
 Offset manager is a data structure used to manage offsets asynchronously. An offset should only be committed when a
 message is processed fully. Offset manager maintains a state of all the offsets of all topic-partitions, that can be
 committed. It can also be used by sinks to manage its own offsets.
-
-

docs/docs/contribute/contribution.md

@@ -3,8 +3,8 @@
 The following is a set of guidelines for contributing to Firehose. These are mostly guidelines, not rules. Use your best judgment, and feel free to propose changes to this document in a pull request. Here are some important resources:
 
 - The [Concepts](../guides/create_firehose.md) section will explain to you about Firehose architecture,
-- Our [roadmap](https://github.com/odpf/firehose/blob/main/docs/roadmap.md) is the 10k foot view of where we're going, and
-- Github [issues](https://github.com/odpf/firehose/issues) track the ongoing and reported issues.
+- Our [roadmap](https://github.com/raystack/firehose/blob/main/docs/roadmap.md) is the 10k foot view of where we're going, and
+- Github [issues](https://github.com/raystack/firehose/issues) track the ongoing and reported issues.
 
 Development of Firehose happens in the open on GitHub, and we are grateful to the community for contributing bug fixes and improvements. Read below to learn how you can take part in improving Firehose.
 
@@ -23,14 +23,14 @@ The following parts are open for contribution:
 - Provide suggestions to make the user experience better
 - Provide suggestions to Improve the documentation
 
-To help you get your feet wet and get you familiar with our contribution process, we have a list of [good first issues](https://github.com/odpf/firehose/labels/good%20first%20issue) that contain bugs that have a relatively limited scope. This is a great place to get started.
+To help you get your feet wet and get you familiar with our contribution process, we have a list of [good first issues](https://github.com/raystack/firehose/labels/good%20first%20issue) that contain bugs that have a relatively limited scope. This is a great place to get started.
 
 ## How can I contribute?
 
 We use RFCs and GitHub issues to communicate ideas.
 
 - You can report a bug or suggest a feature enhancement or can just ask questions. Reach out on Github discussions for this purpose.
-- You are also welcome to add a new common sink in [depot](https://github.com/odpf/depot), improve monitoring and logging and improve code quality.
+- You are also welcome to add a new common sink in [depot](https://github.com/raystack/depot), improve monitoring and logging and improve code quality.
 - You can help with documenting new features or improve existing documentation.
 - You can also review and accept other contributions if you are a maintainer.
 
@@ -53,4 +53,4 @@ Please follow these practices for your change to get merged fast and smoothly:
 - If you are introducing a completely new feature or making any major changes to an existing one, we recommend starting with an RFC and get consensus on the basic design first.
 - Make sure your local build is running with all the tests and checkstyle passing.
 - If your change is related to user-facing protocols/configurations, you need to make the corresponding change in the documentation as well.
-- Docs live in the code repo under [`docs`](https://github.com/odpf/firehose/tree/7d0df99962507e6ad2147837c4536f36d52d5a48/docs/docs/README.md) so that changes to that can be done in the same PR as changes to the code.
+- Docs live in the code repo under [`docs`](https://github.com/raystack/firehose/tree/main/docs/docs/README.md) so that changes to that can be done in the same PR as changes to the code.