Help with ruby test setup and configuration

[renegaderyu]

(check apply)

• read the contribution guideline
• (optional) already reported 3rd party upstream repository or mailing list if you use k8s addon or helm charts.

Steps to replicate

While trying to add and run test cases for contributions to this repo I ran bundle exec rake test but it produces errors for most test in out_opensearch_data_stream.rb like below. I suspect there is something in environment setup that was either missed or done improperly. I'd like to figure it out and get it documented to make future contributions easier.

====================================================================================================================
Error: test_custom_record_placeholder(OpenSearchOutputDataStreamTest): Fluent::ConfigError: Failed to create data stream: <foo_bar> unknown keyword: acknowledged
/Volumes/sourcecode/fluent-plugin-opensearch/lib/fluent/plugin/out_opensearch_data_stream.rb:168:in `rescue in write'
/Volumes/sourcecode/fluent-plugin-opensearch/lib/fluent/plugin/out_opensearch_data_stream.rb:165:in `write'
/Users/rmiguel/.gem/ruby/2.5.0/gems/fluentd-1.16.1/lib/fluent/plugin/output.rb:1225:in `try_flush'
/Users/rmiguel/.gem/ruby/2.5.0/gems/fluentd-1.16.1/lib/fluent/plugin/output.rb:1538:in `flush_thread_run'
/Users/rmiguel/.gem/ruby/2.5.0/gems/fluentd-1.16.1/lib/fluent/plugin/output.rb:510:in `block (2 levels) in start'
/Users/rmiguel/.gem/ruby/2.5.0/gems/fluentd-1.16.1/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
====================================================================================================================
E

Expected Behavior or What you need to ask

I'd like to be able to test locally

Using Fluentd and OpenSearch plugin versions

Gemfile.lock

PATH
  remote: .
  specs:
    fluent-plugin-opensearch (1.1.3)
      aws-sdk-core (~> 3)
      excon
      faraday (>= 2.0.0)
      faraday-excon (>= 2.0.0)
      faraday_middleware-aws-sigv4 (~> 1.0.1)
      fluentd (>= 0.14.22)
      opensearch-ruby

GEM
  remote: https://rubygems.org/
  specs:
    addressable (2.8.4)
      public_suffix (>= 2.0.2, < 6.0)
    aws-eventstream (1.2.0)
    aws-partitions (1.785.0)
    aws-sdk-core (3.177.0)
      aws-eventstream (~> 1, >= 1.0.2)
      aws-partitions (~> 1, >= 1.651.0)
      aws-sigv4 (~> 1.5)
      jmespath (~> 1, >= 1.6.1)
    aws-sigv4 (1.6.0)
      aws-eventstream (~> 1, >= 1.0.2)
    concurrent-ruby (1.2.2)
    cool.io (1.7.1)
    crack (0.4.5)
      rexml
    docile (1.4.0)
    excon (0.100.0)
    faraday (2.7.10)
      faraday-net_http (>= 2.0, < 3.1)
      ruby2_keywords (>= 0.0.4)
    faraday-excon (2.1.0)
      excon (>= 0.27.4)
      faraday (~> 2.0)
    faraday-net_http (3.0.2)
    faraday_middleware-aws-sigv4 (1.0.1)
      aws-sigv4 (~> 1.0)
      faraday (>= 2.0, < 3)
    flexmock (2.3.6)
    fluentd (1.16.1)
      bundler
      cool.io (>= 1.4.5, < 2.0.0)
      http_parser.rb (>= 0.5.1, < 0.9.0)
      msgpack (>= 1.3.1, < 2.0.0)
      serverengine (>= 2.3.2, < 3.0.0)
      sigdump (~> 0.2.2)
      strptime (>= 0.2.4, < 1.0.0)
      tzinfo (>= 1.0, < 3.0)
      tzinfo-data (~> 1.0)
      webrick (~> 1.4)
      yajl-ruby (~> 1.0)
    hashdiff (1.0.1)
    http_parser.rb (0.8.0)
    io-console (0.6.0)
    irb (1.6.3)
      reline (>= 0.3.0)
    jmespath (1.6.2)
    minitest (5.18.1)
    msgpack (1.7.1)
    multi_json (1.15.0)
    oj (3.14.2)
    opensearch-api (2.2.0)
      multi_json
    opensearch-ruby (2.1.0)
      opensearch-api (~> 2.1)
      opensearch-transport (~> 2.0)
    opensearch-transport (2.1.0)
      faraday (>= 1.0, < 3)
      multi_json
    power_assert (2.0.3)
    public_suffix (5.0.1)
    rake (13.0.6)
    reline (0.3.6)
      io-console (~> 0.5)
    rexml (3.2.5)
    ruby2_keywords (0.0.5)
    serverengine (2.3.2)
      sigdump (~> 0.2.2)
    sigdump (0.2.5)
    simplecov (0.22.0)
      docile (~> 1.1)
      simplecov-html (~> 0.11)
      simplecov_json_formatter (~> 0.1)
    simplecov-html (0.12.3)
    simplecov-lcov (0.8.0)
    simplecov_json_formatter (0.1.4)
    strptime (0.2.5)
    test-unit (3.3.9)
      power_assert
    tzinfo (2.0.6)
      concurrent-ruby (~> 1.0)
    tzinfo-data (1.2023.3)
      tzinfo (>= 1.0.0)
    webmock (3.18.1)
      addressable (>= 2.8.0)
      crack (>= 0.3.2)
      hashdiff (>= 0.4.0, < 2.0.0)
    webrick (1.7.0)
    yajl-ruby (1.4.3)

PLATFORMS
  arm64-darwin-22

DEPENDENCIES
  flexmock (~> 2.0)
  fluent-plugin-opensearch!
  irb
  minitest (~> 5.8)
  oj
  rake
  simplecov
  simplecov-lcov
  strptime
  test-unit (~> 3.3.0)
  webmock (~> 3)
  webrick (~> 1.7.0)

BUNDLED WITH
   2.3.26

[renegaderyu]

So I thought this was environment config (it may still be). But I realized that if I make the changes below that I can test locally and do not receive the errors mentioned above

diff --git a/test/plugin/test_out_opensearch_data_stream.rb b/test/plugin/test_out_opensearch_data_stream.rb
index 872bf75..86e30ab 100644
--- a/test/plugin/test_out_opensearch_data_stream.rb
+++ b/test/plugin/test_out_opensearch_data_stream.rb
@@ -57,24 +57,24 @@ class OpenSearchOutputDataStreamTest < Test::Unit::TestCase
     {'@timestamp' => Time.now.iso8601, 'message' => 'Sample record'}
   end
 
-  RESPONSE_ACKNOWLEDGED = {"acknowledged": true}
+  RESPONSE_ACKNOWLEDGED = '{"acknowledged": true}'
   DUPLICATED_DATA_STREAM_EXCEPTION = {"error": {}, "status": 400}
   NONEXISTENT_DATA_STREAM_EXCEPTION = {"error": {}, "status": 404}
 
   def stub_index_template(name="foo_tpl", url="http://localhost:9200")
-    stub_request(:put, "#{url}/_index_template/#{name}").to_return(:status => [200, RESPONSE_ACKNOWLEDGED])
+    stub_request(:put, "#{url}/_index_template/#{name}").to_return(status: 200, body: RESPONSE_ACKNOWLEDGED)
   end
 
   def stub_data_stream(name="foo", url="localhost:9200")
-    stub_request(:put, "#{url}/_data_stream/#{name}").to_return(:status => [200, RESPONSE_ACKNOWLEDGED])
+    stub_request(:put, "#{url}/_data_stream/#{name}").to_return(status: 200, body: RESPONSE_ACKNOWLEDGED)
   end
 
   def stub_existent_data_stream?(name="foo", url="localhost:9200")
-    stub_request(:get, "#{url}/_data_stream/#{name}").to_return(:status => [200, RESPONSE_ACKNOWLEDGED])
+    stub_request(:get, "#{url}/_data_stream/#{name}").to_return(status: 200, body: RESPONSE_ACKNOWLEDGED)
   end
 
   def stub_existent_template?(name="foo_tpl", url="localhost:9200")
-    stub_request(:get, "#{url}/_index_template/#{name}").to_return(:status => [200, RESPONSE_ACKNOWLEDGED])
+    stub_request(:get, "#{url}/_index_template/#{name}").to_return(status: 200, body: RESPONSE_ACKNOWLEDGED)
   end
 
   def stub_nonexistent_data_stream?(name="foo", url="localhost:9200")

[24367dfa]

i managed to get the test setup running in both ubuntu and wsl2 on windows following the instructions given in the readme.

[cosmo0920]

Already resolved at master.