Fix ReaderConfig's default values and a few other fixes (#189)

* Don't assume anything * Validate parameters and normalize default values * Add start offsets enum to the types info * Update JS API docs * Remove opinionated default value for MaxWait * Use kafka-go's internal constants * Add OffsetOutOfRangeError for extreme cases - won't be documented, because it is deprecated * Remove commented code * Add example script for consuming from a consumer group * Run prettier to fix formatting * Mention test_consumer_group.js in the README * Add emitted metrics to README * Fix error message
mostafa · Jan 23, 2023 · 005db43 · 005db43
1 parent 59cd729
commit 005db43
Show file tree

Hide file tree

Showing 33 changed files with 410 additions and 153 deletions.
diff --git a/README.md b/README.md
@@ -20,7 +20,7 @@ If you want to learn more about the extension, read the [article](https://k6.io/
 - Create, list and delete [topics](https://github.com/mostafa/xk6-kafka/blob/main/scripts/test_topics.js)
 - Support for loading Avro schemas from [Schema Registry](https://github.com/mostafa/xk6-kafka/blob/main/scripts/test_avro_with_schema_registry.js)
 - Support for [byte array](https://github.com/mostafa/xk6-kafka/blob/main/scripts/test_bytes.js) for binary data (from binary protocols)
-- Support consumption from all partitions with a group ID
+- Support consumption from all partitions with a [group ID](https://github.com/mostafa/xk6-kafka/blob/main/scripts/test_consumer_group.js)
 - Support Kafka message compression: Gzip, [Snappy](https://github.com/mostafa/xk6-kafka/blob/main/scripts/test_json.js), Lz4 & Zstd
 - Support for sending messages with [no key](https://github.com/mostafa/xk6-kafka/blob/main/scripts/test_avro_no_schema_registry.js)
 - Support for k6 [thresholds](https://github.com/mostafa/xk6-kafka/blob/main/scripts/test_json.js) on custom Kafka metrics
@@ -330,6 +330,46 @@ The example scripts are available as `test_<format/feature>.js` with more code a
        vus_max........................: 50      min=50         max=50
    ```
 
+### Emitted Metrics
+
+| Metric                       | Type    | Description                                                             |
+| ---------------------------- | ------- | ----------------------------------------------------------------------- |
+| kafka_reader_dial_count      | Counter | Total number of times the reader tries to connect.                      |
+| kafka_reader_fetches_count   | Counter | Total number of times the reader fetches batches of messages.           |
+| kafka_reader_message_count   | Counter | Total number of messages consumed.                                      |
+| kafka_reader_message_bytes   | Counter | Total bytes consumed.                                                   |
+| kafka_reader_rebalance_count | Counter | Total number of rebalances of a topic in a consumer group (deprecated). |
+| kafka_reader_timeouts_count  | Counter | Total number of timeouts occurred when reading.                         |
+| kafka_reader_error_count     | Counter | Total number of errors occurred when reading.                           |
+| kafka_reader_dial_seconds    | Trend   | The time it takes to connect to the leader in a Kafka cluster.          |
+| kafka_reader_read_seconds    | Trend   | The time it takes to read a batch of message.                           |
+| kafka_reader_wait_seconds    | Trend   | Waiting time before read a batch of messages.                           |
+| kafka_reader_fetch_size      | Counter | Total messages fetched.                                                 |
+| kafka_reader_fetch_bytes     | Counter | Total bytes fetched.                                                    |
+| kafka_reader_offset          | Gauge   | Number of messages read after the given offset in a batch.              |
+| kafka_reader_lag             | Gauge   | The lag between the last message offset and the current read offset.    |
+| kafka_reader_fetch_bytes_min | Gauge   | Minimum number of bytes fetched.                                        |
+| kafka_reader_fetch_bytes_max | Gauge   | Maximum number of bytes fetched.                                        |
+| kafka_reader_fetch_wait_max  | Gauge   | The maximum time it takes to fetch a batch of messages.                 |
+| kafka_reader_queue_length    | Gauge   | The queue length while reading batch of messages.                       |
+| kafka_reader_queue_capacity  | Gauge   | The queue capacity while reading batch of messages.                     |
+| kafka_writer_write_count     | Counter | Total number of times the writer writes batches of messages.            |
+| kafka_writer_message_count   | Counter | Total number of messages produced.                                      |
+| kafka_writer_message_bytes   | Counter | Total bytes produced.                                                   |
+| kafka_writer_error_count     | Counter | Total number of errors occurred when writing.                           |
+| kafka_writer_write_seconds   | Trend   | The time it takes writing messages.                                     |
+| kafka_writer_wait_seconds    | Trend   | Waiting time before writing messages.                                   |
+| kafka_writer_retries_count   | Counter | Total number of attempts at writing messages.                           |
+| kafka_writer_batch_size      | Counter | Total batch size.                                                       |
+| kafka_writer_batch_bytes     | Counter | Total number of bytes in a batch of messages.                           |
+| kafka_writer_attempts_max    | Gauge   | Maximum number of attempts at writing messages.                         |
+| kafka_writer_batch_max       | Gauge   | Maximum batch size.                                                     |
+| kafka_writer_batch_timeout   | Gauge   | Batch timeout.                                                          |
+| kafka_writer_read_timeout    | Gauge   | Batch read timeout.                                                     |
+| kafka_writer_write_timeout   | Gauge   | Batch write timeout.                                                    |
+| kafka_writer_acks_required   | Gauge   | Required Acks.                                                          |
+| kafka_writer_async           | Rate    | Async writer.                                                           |
+
 ### FAQ
 
 1. Why do I receive `Error writing messages`?
@@ -372,6 +412,10 @@ The example scripts are available as `test_<format/feature>.js` with more code a
 
    For performance testing reasons, the `maxWait` of the `Reader` is set to 200ms. If you keep receiving this error, consider increasing it to a larger value.
 
+8. How can I consume from multiple partitions on a single topic?
+
+   You can configure your reader to consume from a (list of) topic(s) and its partitions using a consumer group. This can be achieve by setting `groupTopics`, `groupID` and a few other options for timeouts, intervals and lags. Have a look at the [`test_consumer_group.js`](https://github.com/mostafa/xk6-kafka/blob/main/scripts/test_consumer_group.js) example script.
+
 ## Contributions, Issues and Feedback
 
 I'd be thrilled to receive contributions and feedback on this project. You're always welcome to create an issue if you find one (or many). I would do my best to address the issues. Also, feel free to contribute by opening a PR with changes, and I'll do my best to review and merge it as soon as I can.

diff --git a/api-docs/docs/README.md b/api-docs/docs/README.md
@@ -17,6 +17,7 @@ This documentation refers to the development version of the xk6-kafka project, w
 - [ISOLATION_LEVEL](enums/ISOLATION_LEVEL.md)
 - [SASL_MECHANISMS](enums/SASL_MECHANISMS.md)
 - [SCHEMA_TYPES](enums/SCHEMA_TYPES.md)
+- [START_OFFSETS](enums/START_OFFSETS.md)
 - [SUBJECT_NAME_STRATEGY](enums/SUBJECT_NAME_STRATEGY.md)
 - [TLS_VERSIONS](enums/TLS_VERSIONS.md)
 

diff --git a/api-docs/docs/classes/Connection.md b/api-docs/docs/classes/Connection.md
@@ -44,7 +44,7 @@ connection.close();
 
 #### Defined in
 
-[index.d.ts:355](https://github.com/mostafa/xk6-kafka/blob/main/api-docs/index.d.ts#L355)
+[index.d.ts:361](https://github.com/mostafa/xk6-kafka/blob/main/api-docs/index.d.ts#L361)
 
 ## Methods
 
@@ -64,7 +64,7 @@ connection.close();
 
 #### Defined in
 
-[index.d.ts:381](https://github.com/mostafa/xk6-kafka/blob/main/api-docs/index.d.ts#L381)
+[index.d.ts:387](https://github.com/mostafa/xk6-kafka/blob/main/api-docs/index.d.ts#L387)
 
 ---
 
@@ -89,7 +89,7 @@ Create a new topic.
 
 #### Defined in
 
-[index.d.ts:362](https://github.com/mostafa/xk6-kafka/blob/main/api-docs/index.d.ts#L362)
+[index.d.ts:368](https://github.com/mostafa/xk6-kafka/blob/main/api-docs/index.d.ts#L368)
 
 ---
 
@@ -114,7 +114,7 @@ Delete a topic.
 
 #### Defined in
 
-[index.d.ts:369](https://github.com/mostafa/xk6-kafka/blob/main/api-docs/index.d.ts#L369)
+[index.d.ts:375](https://github.com/mostafa/xk6-kafka/blob/main/api-docs/index.d.ts#L375)
 
 ---
 
@@ -133,4 +133,4 @@ List topics.
 
 #### Defined in
 
-[index.d.ts:375](https://github.com/mostafa/xk6-kafka/blob/main/api-docs/index.d.ts#L375)
+[index.d.ts:381](https://github.com/mostafa/xk6-kafka/blob/main/api-docs/index.d.ts#L381)
diff --git a/api-docs/docs/classes/Reader.md b/api-docs/docs/classes/Reader.md
@@ -43,7 +43,7 @@ reader.close();
 
 #### Defined in
 
-[index.d.ts:314](https://github.com/mostafa/xk6-kafka/blob/main/api-docs/index.d.ts#L314)
+[index.d.ts:320](https://github.com/mostafa/xk6-kafka/blob/main/api-docs/index.d.ts#L320)
 
 ## Methods
 
@@ -63,7 +63,7 @@ reader.close();
 
 #### Defined in
 
-[index.d.ts:327](https://github.com/mostafa/xk6-kafka/blob/main/api-docs/index.d.ts#L327)
+[index.d.ts:333](https://github.com/mostafa/xk6-kafka/blob/main/api-docs/index.d.ts#L333)
 
 ---
 
@@ -88,4 +88,4 @@ Read messages from Kafka.
 
 #### Defined in
 
-[index.d.ts:321](https://github.com/mostafa/xk6-kafka/blob/main/api-docs/index.d.ts#L321)
+[index.d.ts:327](https://github.com/mostafa/xk6-kafka/blob/main/api-docs/index.d.ts#L327)
diff --git a/api-docs/docs/classes/SchemaRegistry.md b/api-docs/docs/classes/SchemaRegistry.md
@@ -79,7 +79,7 @@ writer.produce({
 
 #### Defined in
 
-[index.d.ts:443](https://github.com/mostafa/xk6-kafka/blob/main/api-docs/index.d.ts#L443)
+[index.d.ts:449](https://github.com/mostafa/xk6-kafka/blob/main/api-docs/index.d.ts#L449)
 
 ## Methods
 
@@ -104,7 +104,7 @@ Create or update a schema on Schema Registry.
 
 #### Defined in
 
-[index.d.ts:457](https://github.com/mostafa/xk6-kafka/blob/main/api-docs/index.d.ts#L457)
+[index.d.ts:463](https://github.com/mostafa/xk6-kafka/blob/main/api-docs/index.d.ts#L463)
 
 ---
 
@@ -129,7 +129,7 @@ Deserializes the given data and schema into its original form.
 
 #### Defined in
 
-[index.d.ts:478](https://github.com/mostafa/xk6-kafka/blob/main/api-docs/index.d.ts#L478)
+[index.d.ts:484](https://github.com/mostafa/xk6-kafka/blob/main/api-docs/index.d.ts#L484)
 
 ---
 
@@ -154,7 +154,7 @@ Get a schema from Schema Registry by version and subject.
 
 #### Defined in
 
-[index.d.ts:450](https://github.com/mostafa/xk6-kafka/blob/main/api-docs/index.d.ts#L450)
+[index.d.ts:456](https://github.com/mostafa/xk6-kafka/blob/main/api-docs/index.d.ts#L456)
 
 ---
 
@@ -179,7 +179,7 @@ Returns the subject name for the given SubjectNameConfig.
 
 #### Defined in
 
-[index.d.ts:464](https://github.com/mostafa/xk6-kafka/blob/main/api-docs/index.d.ts#L464)
+[index.d.ts:470](https://github.com/mostafa/xk6-kafka/blob/main/api-docs/index.d.ts#L470)
 
 ---
 
@@ -204,4 +204,4 @@ Serializes the given data and schema into a byte array.
 
 #### Defined in
 
-[index.d.ts:471](https://github.com/mostafa/xk6-kafka/blob/main/api-docs/index.d.ts#L471)
+[index.d.ts:477](https://github.com/mostafa/xk6-kafka/blob/main/api-docs/index.d.ts#L477)
diff --git a/api-docs/docs/classes/Writer.md b/api-docs/docs/classes/Writer.md
@@ -51,7 +51,7 @@ writer.close();
 
 #### Defined in
 
-[index.d.ts:272](https://github.com/mostafa/xk6-kafka/blob/main/api-docs/index.d.ts#L272)
+[index.d.ts:278](https://github.com/mostafa/xk6-kafka/blob/main/api-docs/index.d.ts#L278)
 
 ## Methods
 
@@ -71,7 +71,7 @@ writer.close();
 
 #### Defined in
 
-[index.d.ts:285](https://github.com/mostafa/xk6-kafka/blob/main/api-docs/index.d.ts#L285)
+[index.d.ts:291](https://github.com/mostafa/xk6-kafka/blob/main/api-docs/index.d.ts#L291)
 
 ---
 
@@ -96,4 +96,4 @@ Write messages to Kafka.
 
 #### Defined in
 
-[index.d.ts:279](https://github.com/mostafa/xk6-kafka/blob/main/api-docs/index.d.ts#L279)
+[index.d.ts:285](https://github.com/mostafa/xk6-kafka/blob/main/api-docs/index.d.ts#L285)
diff --git a/api-docs/docs/enums/BALANCERS.md b/api-docs/docs/enums/BALANCERS.md
@@ -18,7 +18,7 @@
 
 #### Defined in
 
-[index.d.ts:63](https://github.com/mostafa/xk6-kafka/blob/main/api-docs/index.d.ts#L63)
+[index.d.ts:69](https://github.com/mostafa/xk6-kafka/blob/main/api-docs/index.d.ts#L69)
 
 ---
 
@@ -28,7 +28,7 @@
 
 #### Defined in
 
-[index.d.ts:62](https://github.com/mostafa/xk6-kafka/blob/main/api-docs/index.d.ts#L62)
+[index.d.ts:68](https://github.com/mostafa/xk6-kafka/blob/main/api-docs/index.d.ts#L68)
 
 ---
 
@@ -38,7 +38,7 @@
 
 #### Defined in
 
-[index.d.ts:61](https://github.com/mostafa/xk6-kafka/blob/main/api-docs/index.d.ts#L61)
+[index.d.ts:67](https://github.com/mostafa/xk6-kafka/blob/main/api-docs/index.d.ts#L67)
 
 ---
 
@@ -48,7 +48,7 @@
 
 #### Defined in
 
-[index.d.ts:64](https://github.com/mostafa/xk6-kafka/blob/main/api-docs/index.d.ts#L64)
+[index.d.ts:70](https://github.com/mostafa/xk6-kafka/blob/main/api-docs/index.d.ts#L70)
 
 ---
 
@@ -58,4 +58,4 @@
 
 #### Defined in
 
-[index.d.ts:60](https://github.com/mostafa/xk6-kafka/blob/main/api-docs/index.d.ts#L60)
+[index.d.ts:66](https://github.com/mostafa/xk6-kafka/blob/main/api-docs/index.d.ts#L66)
diff --git a/api-docs/docs/enums/GROUP_BALANCERS.md b/api-docs/docs/enums/GROUP_BALANCERS.md
@@ -16,7 +16,7 @@
 
 #### Defined in
 
-[index.d.ts:71](https://github.com/mostafa/xk6-kafka/blob/main/api-docs/index.d.ts#L71)
+[index.d.ts:77](https://github.com/mostafa/xk6-kafka/blob/main/api-docs/index.d.ts#L77)
 
 ---
 
@@ -26,7 +26,7 @@
 
 #### Defined in
 
-[index.d.ts:69](https://github.com/mostafa/xk6-kafka/blob/main/api-docs/index.d.ts#L69)
+[index.d.ts:75](https://github.com/mostafa/xk6-kafka/blob/main/api-docs/index.d.ts#L75)
 
 ---
 
@@ -36,4 +36,4 @@
 
 #### Defined in
 
-[index.d.ts:70](https://github.com/mostafa/xk6-kafka/blob/main/api-docs/index.d.ts#L70)
+[index.d.ts:76](https://github.com/mostafa/xk6-kafka/blob/main/api-docs/index.d.ts#L76)
diff --git a/api-docs/docs/enums/SCHEMA_TYPES.md b/api-docs/docs/enums/SCHEMA_TYPES.md
@@ -18,7 +18,7 @@
 
 #### Defined in
 
-[index.d.ts:78](https://github.com/mostafa/xk6-kafka/blob/main/api-docs/index.d.ts#L78)
+[index.d.ts:84](https://github.com/mostafa/xk6-kafka/blob/main/api-docs/index.d.ts#L84)
 
 ---
 
@@ -28,7 +28,7 @@
 
 #### Defined in
 
-[index.d.ts:77](https://github.com/mostafa/xk6-kafka/blob/main/api-docs/index.d.ts#L77)
+[index.d.ts:83](https://github.com/mostafa/xk6-kafka/blob/main/api-docs/index.d.ts#L83)
 
 ---
 
@@ -38,7 +38,7 @@
 
 #### Defined in
 
-[index.d.ts:79](https://github.com/mostafa/xk6-kafka/blob/main/api-docs/index.d.ts#L79)
+[index.d.ts:85](https://github.com/mostafa/xk6-kafka/blob/main/api-docs/index.d.ts#L85)
 
 ---
 
@@ -48,7 +48,7 @@
 
 #### Defined in
 
-[index.d.ts:80](https://github.com/mostafa/xk6-kafka/blob/main/api-docs/index.d.ts#L80)
+[index.d.ts:86](https://github.com/mostafa/xk6-kafka/blob/main/api-docs/index.d.ts#L86)
 
 ---
 
@@ -58,4 +58,4 @@
 
 #### Defined in
 
-[index.d.ts:76](https://github.com/mostafa/xk6-kafka/blob/main/api-docs/index.d.ts#L76)
+[index.d.ts:82](https://github.com/mostafa/xk6-kafka/blob/main/api-docs/index.d.ts#L82)
diff --git a/api-docs/docs/enums/START_OFFSETS.md b/api-docs/docs/enums/START_OFFSETS.md
@@ -0,0 +1,28 @@
+# Enumeration: START_OFFSETS
+
+## Table of contents
+
+### Enumeration Members
+
+- [START_OFFSETS_FIRST_OFFSET](START_OFFSETS.md#start_offsets_first_offset)
+- [START_OFFSETS_LAST_OFFSET](START_OFFSETS.md#start_offsets_last_offset)
+
+## Enumeration Members
+
+### START_OFFSETS_FIRST_OFFSET
+
+• **START_OFFSETS_FIRST_OFFSET**
+
+#### Defined in
+
+[index.d.ts:54](https://github.com/mostafa/xk6-kafka/blob/main/api-docs/index.d.ts#L54)
+
+---
+
+### START_OFFSETS_LAST_OFFSET
+
+• **START_OFFSETS_LAST_OFFSET**
+
+#### Defined in
+
+[index.d.ts:53](https://github.com/mostafa/xk6-kafka/blob/main/api-docs/index.d.ts#L53)
diff --git a/api-docs/docs/enums/SUBJECT_NAME_STRATEGY.md b/api-docs/docs/enums/SUBJECT_NAME_STRATEGY.md
@@ -16,7 +16,7 @@
 
 #### Defined in
 
-[index.d.ts:54](https://github.com/mostafa/xk6-kafka/blob/main/api-docs/index.d.ts#L54)
+[index.d.ts:60](https://github.com/mostafa/xk6-kafka/blob/main/api-docs/index.d.ts#L60)
 
 ---
 
@@ -26,7 +26,7 @@
 
 #### Defined in
 
-[index.d.ts:53](https://github.com/mostafa/xk6-kafka/blob/main/api-docs/index.d.ts#L53)
+[index.d.ts:59](https://github.com/mostafa/xk6-kafka/blob/main/api-docs/index.d.ts#L59)
 
 ---
 
@@ -36,4 +36,4 @@
 
 #### Defined in
 
-[index.d.ts:55](https://github.com/mostafa/xk6-kafka/blob/main/api-docs/index.d.ts#L55)
+[index.d.ts:61](https://github.com/mostafa/xk6-kafka/blob/main/api-docs/index.d.ts#L61)
diff --git a/api-docs/docs/interfaces/BasicAuth.md b/api-docs/docs/interfaces/BasicAuth.md
@@ -15,7 +15,7 @@
 
 #### Defined in
 
-[index.d.ts:137](https://github.com/mostafa/xk6-kafka/blob/main/api-docs/index.d.ts#L137)
+[index.d.ts:143](https://github.com/mostafa/xk6-kafka/blob/main/api-docs/index.d.ts#L143)
 
 ---
 
@@ -25,4 +25,4 @@
 
 #### Defined in
 
-[index.d.ts:136](https://github.com/mostafa/xk6-kafka/blob/main/api-docs/index.d.ts#L136)
+[index.d.ts:142](https://github.com/mostafa/xk6-kafka/blob/main/api-docs/index.d.ts#L142)