Smithy4s Kinesis Client #36

etspaceman · 2023-02-05T02:22:36Z

Changes Introduced

Introduces a Smithy4s Kinesis Client implementation, bundled with logs and localstack support. Important decisions:

Slf4j was not included as the default logger for this module, as we would like to eventually cross-build this for scalajs/native. Instead, a NoOp logger is the default and users can provide their own implementations as needed. This matches the approach that Http4s is using in their upcoming changes.
Scoverage was removed as there were issues with running coverage tests with smithy4sGenerate. There is little actual benefit to scoverage, so it feels safe to remove
sbt-projectmatrix was included to help reason between the scala version differences, as Smithy4s does not offer a 2.12 version of the AWS modules
Using scalafmt instead of scalafix to organize imports
sbt-typelevel's MergifyPlugin was included to help manage that file

Applicable linked issues

#17 #16

Checklist (check all that apply)

This change maintains backwards compatibility
I have introduced tests for all new features and changes
I have added documentation covering all new features and changes
This pull-request is ready for review

etspaceman · 2023-02-05T03:03:18Z

@kubukoz per our convo, I tried setting up a client using smithy4s, but ran into a few weird bugs with the build. Mostly, I've been dealing with using a subset of scala versions in the smithy project as smithy4s is only publishing Scala 2.13/Scala 3 artifacts, so I am poorly attempting to use sbt-projectmatrix as well. But also, I ran into this when compiling on scala 2.13 versions:

[error] (smithy4s-client / Compile / smithy4sCodegen) java.lang.ClassCastException: class software.amazon.smithy.rulesengine.traits.ContextParamTrait cannot be cast to class software.amazon.smithy.rulesengine.traits.ContextParamTrait (software.amazon.smithy.rulesengine.traits.ContextParamTrait is in unnamed module of loader java.net.URLClassLoader @17cc56dd; software.amazon.smithy.rulesengine.traits.ContextParamTrait is in unnamed module of loader java.net.URLClassLoader @756a6cba)

kubukoz · 2023-02-05T16:17:18Z

project/Kinesis4CatsPlugin.scala

+          )
+      })
+  }
+
  final implicit class Kinesi4catsProjectOps(private val p: Project)


fyi: typo in class name

kubukoz · 2023-02-05T16:18:51Z

Never seen that before I think... perhaps @Baccata will have more insight. I know there was talk of removing the rules engine traits from the model before loading, and in general smithy4s will eventually make it easier to use AWS specs.

disneystreaming/smithy4s#674 (comment) and that general thread may have more useful context

kubukoz · 2023-02-05T16:54:22Z

smithy4s-client / Compile / smithy4sCodegen

I ran that in sbt as smithy4s-client/compile, and it worked. CI seems to be failing because the yml generated doesn't match the one checked in, so it's hard for me to see the issue in CI either.

Additionally it'd be great to minimize any issue you encounter with smithy4s, as your build is pretty big and there could be multiple things at play

kubukoz · 2023-02-05T17:21:37Z

if you do manage to hit this again, can you try with a local snapshot of smithy4s from the series/0.17 branch?

etspaceman · 2023-02-05T19:37:28Z

Thanks for looking into it @kubukoz ! I'll be out of pocket today but I'll see if I can get this PR cleaned up and reproduce the same. And yes, I'll try the new version as well!

Baccata · 2023-02-06T11:54:41Z

@etspaceman don't bother trying a snapshot, it does fix the bug you're currently encountering but another unrelated bug was introduce which leads the generated code for the smithy.rules to be generated twice. I'm in the process of fixing it

Baccata · 2023-02-06T14:06:20Z

Here's a fix to the newly introduced bug I'm talking about : disneystreaming/smithy4s#789

etspaceman · 2023-02-06T16:19:24Z

OK PR is cleaned up with clean reproduce, you should be able to get this via doing sbt smithy4s-client3/compile

[error] stack trace is suppressed; run last smithy4s-client3 / Compile / smithy4sCodegen for the full output
[error] (smithy4s-client3 / Compile / smithy4sCodegen) java.lang.ClassCastException: class software.amazon.smithy.rulesengine.traits.ContextParamTrait cannot be cast to class software.amazon.smithy.rulesengine.traits.ContextParamTrait (software.amazon.smithy.rulesengine.traits.ContextParamTrait is in unnamed module of loader java.net.URLClassLoader @485438eb; software.amazon.smithy.rulesengine.traits.ContextParamTrait is in unnamed module of loader java.net.URLClassLoader @1f8c81d0)

kubukoz · 2023-02-06T18:19:42Z

I can't see it with just client3, but I can see it with this sequence:

smithy4s-client/compile
smithy4s-client3/compile

Updating to a local snapshot of disneystreaming/smithy4s#789 seems to make it work.

etspaceman · 2023-02-06T19:39:00Z

@kubukoz It does make this particular error go away but then a number of compiler errors come on due to the other bug that @Baccata found.

It does seem to be working from the PR branch in @Baccata 's proposal! So that's great 😄

etspaceman · 2023-02-06T21:57:28Z

.scalafix.conf

@@ -1,4 +1,4 @@
-rules = [DisableSyntax, NoValInForComprehension, ProcedureSyntax, OrganizeImports]
+rules = [DisableSyntax, NoValInForComprehension, OrganizeImports]


sbt-projectmatrix added the impact of having scalafmt/scalafix operations run for each project inside of the matrix, including scala 3. These updates allow us to run it on scala 3.

etspaceman · 2023-02-06T21:58:40Z

build.sbt

+      "com.amazonaws.kinesis"
+    )
+  )
+  .jvmPlatform(last2ScalaVersions)


Brought in sbt-projectmatrix to work through having a subset of scala versions supported here (Smithy4s AWS does not have a 2.12 offering)

etspaceman · 2023-02-06T21:59:01Z

project/DockerComposePlugin.scala

@@ -140,7 +140,7 @@ object DockerComposePlugin extends AutoPlugin {
      configuration: Configuration,
      build: Boolean,
      projects: List[Project]
-  ): Seq[Setting[_]] =
+  ): Seq[Setting[_]] = inConfig(configuration)(


Fixes the integration tests always targeting the functional test file...

etspaceman · 2023-02-06T22:00:30Z

project/Kinesis4CatsPlugin.scala

@@ -186,6 +186,17 @@ object Kinesis4CatsPlugin extends AutoPlugin {
    Test / testOptions ++= {
      List(Tests.Argument(MUnitFramework, "+l"))
    },
+    scalacOptions ++= {


typelevel/sbt-typelevel#464

etspaceman · 2023-02-06T22:01:32Z

...alstack/src/main/scala/kinesis4cats/smithy4s/client/localstack/LocalstackKinesisClient.scala

+import kinesis4cats.smithy4s.client.KinesisClient
+import kinesis4cats.smithy4s.client.middleware._
+
+object LocalstackKinesisClient {


The implementation here is almost identical to the KinesisClient, except it adds an additional middleware to proxy the request, as well as uses some mock credentials.

etspaceman · 2023-02-06T22:02:44Z

smithy4s-client/src/main/scala/kinesis4cats/smithy4s/client/KinesisClient.scala

+      val http4sResponseLogEncoder: LogEncoder[Response[F]]
+  )
+
+  def apply[F[_]](


Very simple implementation. Basically the provided generated client from Smithy4s with some middleware to add standard logging.

etspaceman · 2023-02-06T22:04:10Z

smithy4s-client/src/main/scala/kinesis4cats/smithy4s/client/middleware/RequestLogger.scala

+
+import kinesis4cats.logging.LogContext
+
+object RequestLogger {


Mostly copied this from http4s. The http4s implementation records the requests and responses as a part of the message. This one will encode them as a part of the context, per the structured logging desires.

etspaceman · 2023-02-10T22:47:21Z

preprocessors/src/main/scala/preprocessors/KinesisSpecTransformer.scala

+        `trait`.toShapeId() == LengthTrait.ID &&
+        shape.toShapeId() == metricsNameListShapeId
+      ) {
+        LengthTrait.builder().min(0).max(7).build()


Usually this is 1-7. Set to 0-7

etspaceman · 2023-02-10T22:47:35Z

preprocessors/src/main/scala/preprocessors/KinesisSpecTransformer.scala

+    IntegerShape
+      .builder()
+      .id(
+        ShapeId.fromParts("com.amazonaws.kinesis", "NonNegativeIntegerObject")


Counterpart to PositiveIntegerObject

etspaceman · 2023-02-12T01:10:18Z

...client-transformers/src/main/scala/kinesis4cats/smithy4s/client/KinesisSpecTransformer.scala

+        `trait`.toShapeId() == LengthTrait.ID &&
+        shape.toShapeId() == metricsNameListShapeId
+      ) {
+        LengthTrait.builder().min(0).max(7).build()


Empty responses are possible, this seems to be a bug w/ the smithy file

etspaceman · 2023-02-12T01:10:40Z

...client-transformers/src/main/scala/kinesis4cats/smithy4s/client/KinesisSpecTransformer.scala

+          `trait`
+            .asInstanceOf[DocumentationTrait] // scalafix:ok
+            .getValue()
+            .replace("ListStreamConsumersInput$NextToken", "NextToken")


The dollar sign is not valid for scaladoc, so we replace it here.

etspaceman · 2023-02-12T01:10:59Z

...client-transformers/src/main/scala/kinesis4cats/smithy4s/client/KinesisSpecTransformer.scala

+          `trait`
+            .asInstanceOf[DocumentationTrait] // scalafix:ok
+            .getValue()
+            .replace("ListShardsInput$NextToken", "NextToken")


https://github.com/etspaceman/kinesis4cats/pull/36/files#r1103718254

etspaceman · 2023-02-12T01:11:32Z

...client-transformers/src/main/scala/kinesis4cats/smithy4s/client/KinesisSpecTransformer.scala

+            if (memberName == "FailedRecordCount")
+              MemberShape
+                .builder()
+                .target(nonNegativeIntegerObjectShape.getId())


FailedRecordCount can be 0. This is a bug with smithy

etspaceman added 3 commits February 4, 2023 20:22

Update to ProjectMatrix

3924f80

Add crossScalaVersions

98db831

Add scalacOptions manually, generate workflows

7644a51

kubukoz reviewed Feb 5, 2023

View reviewed changes

etspaceman added 3 commits February 6, 2023 09:16

Merge remote-tracking branch 'origin/main' into smithy4sclient

3829370

Fix scalaVersion derivation

50576b9

Cleanup PR

2361949

etspaceman added 5 commits February 6, 2023 10:55

Clean up build

3cb8201

Delete ci.yml

cddb952

Delete clean.yml

a50981b

Ignore .sbt files

66d97da

Fix class name

cc11871

Workaround for kinesis-mock bugs, fixed localstack integration

120aeca

etspaceman commented Feb 6, 2023

View reviewed changes

WIP on using kinesis spec with transformers

4431a22

etspaceman commented Feb 10, 2023

View reviewed changes

etspaceman added 11 commits February 10, 2023 16:55

Cleanup

fb4cdfe

More cleanup

2d56736

Merge remote-tracking branch 'origin/main' into smithy4sclient

d200414

Fixes

71d7199

Remove scala 2.12, project matrix

9f0721a

Mergify plugin

315a933

fix build for some reason

3714d04

Remove scoverage, bring back scala 2.12/matrix

d244cb5

Fix for mergify labels

94cbdd5

Fixes

4493101

remove compat dep

a052373

etspaceman commented Feb 12, 2023

View reviewed changes

Fix mima

4041601

etspaceman added the enhancement New feature or request label Feb 12, 2023

etspaceman merged commit 79170fa into main Feb 12, 2023

etspaceman deleted the smithy4sclient branch February 12, 2023 02:55

mergify bot added kcl shared shared-ciris shared-localstack smithy4s-client smithy4s-client-localstack smithy4s-client-logging-circe smithy4s-client-transformers labels Feb 12, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Smithy4s Kinesis Client #36

Smithy4s Kinesis Client #36

etspaceman commented Feb 5, 2023 •

edited

Loading

etspaceman commented Feb 5, 2023

kubukoz Feb 5, 2023

kubukoz commented Feb 5, 2023

kubukoz commented Feb 5, 2023

kubukoz commented Feb 5, 2023

etspaceman commented Feb 5, 2023 •

edited

Loading

Baccata commented Feb 6, 2023

Baccata commented Feb 6, 2023

etspaceman commented Feb 6, 2023 •

edited

Loading

kubukoz commented Feb 6, 2023

etspaceman commented Feb 6, 2023

etspaceman Feb 6, 2023

etspaceman Feb 6, 2023

etspaceman Feb 6, 2023

etspaceman Feb 6, 2023

etspaceman Feb 6, 2023

etspaceman Feb 6, 2023

etspaceman Feb 6, 2023

etspaceman Feb 10, 2023

etspaceman Feb 10, 2023

etspaceman Feb 12, 2023

etspaceman Feb 12, 2023

etspaceman Feb 12, 2023

etspaceman Feb 12, 2023

		@@ -1,4 +1,4 @@
		rules = [DisableSyntax, NoValInForComprehension, ProcedureSyntax, OrganizeImports]
		rules = [DisableSyntax, NoValInForComprehension, OrganizeImports]


		import kinesis4cats.logging.LogContext

		object RequestLogger {

Smithy4s Kinesis Client #36

Smithy4s Kinesis Client #36

Conversation

etspaceman commented Feb 5, 2023 • edited Loading

Changes Introduced

Applicable linked issues

Checklist (check all that apply)

etspaceman commented Feb 5, 2023

Choose a reason for hiding this comment

kubukoz commented Feb 5, 2023

kubukoz commented Feb 5, 2023

kubukoz commented Feb 5, 2023

etspaceman commented Feb 5, 2023 • edited Loading

Baccata commented Feb 6, 2023

Baccata commented Feb 6, 2023

etspaceman commented Feb 6, 2023 • edited Loading

kubukoz commented Feb 6, 2023

etspaceman commented Feb 6, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

etspaceman commented Feb 5, 2023 •

edited

Loading

etspaceman commented Feb 5, 2023 •

edited

Loading

etspaceman commented Feb 6, 2023 •

edited

Loading