Logs stop appearing in cloudwatch

[mukshe01]

Hi Team,

We are running a quarkus app with quarkus-logging-cloudwatch extention to push logs to cloudwatch. we are running the app with default quarkus-logging-cloudwatch extention configs,

https://github.com/quarkiverse/quarkus-logging-cloudwatch
quarkus-logging-cloudwatch extention version: 5.4.0
our quarkus platform version: 2.13.0.Final

After some time the app stops pushing logs to cloudwatch, we see these errors in the app logs, would you let us how to fix this issue,

2024-06-03T08:43:51.133976517Z stdout F {"timestamp":"2024-06-03T08:43:51.133Z","sequence":1773,"loggerClassName":"org.apache.commons.logging.impl.JBossLog","loggerName":"org.apache.http.headers","level":"
DEBUG","message":"http-outgoing-0 << HTTP/1.1 400 Bad Request","threadName":"pool-12-thread-1","threadId":32,"mdc":{},"ndc":"","hostName":"rest-api-qa-86b7799858-68skq","processName":"quarkus-run.jar","pro
cessId":1}
2024-06-03T08:43:51.133988847Z stdout F {"timestamp":"2024-06-03T08:43:51.133Z","sequence":1774,"loggerClassName":"org.apache.commons.logging.impl.JBossLog","loggerName":"org.apache.http.headers","level":"
DEBUG","message":"http-outgoing-0 << x-amzn-RequestId: 125decf5-51f6-46cd-be57-af4919a3abff","threadName":"pool-12-thread-1","threadId":32,"mdc":{},"ndc":"","hostName":"rest-api-qa-86b7799858-68skq","proce
ssName":"quarkus-run.jar","processId":1}
2024-06-03T08:43:51.133991568Z stdout F {"timestamp":"2024-06-03T08:43:51.133Z","sequence":1775,"loggerClassName":"org.apache.commons.logging.impl.JBossLog","loggerName":"org.apache.http.headers","level":"
DEBUG","message":"http-outgoing-0 << Content-Type: application/x-amz-json-1.1","threadName":"pool-12-thread-1","threadId":32,"mdc":{},"ndc":"","hostName":"rest-api-qa-86b7799858-68skq","processName":"quark
us-run.jar","processId":1}
2024-06-03T08:43:51.133998188Z stdout F {"timestamp":"2024-06-03T08:43:51.133Z","sequence":1776,"loggerClassName":"org.apache.commons.logging.impl.JBossLog","loggerName":"org.apache.http.headers","level":"
DEBUG","message":"http-outgoing-0 << Content-Length: 107","threadName":"pool-12-thread-1","threadId":32,"mdc":{},"ndc":"","hostName":"rest-api-qa-86b7799858-68skq","processName":"quarkus-run.jar","processI
d":1}
2024-06-03T08:43:51.134000398Z stdout F {"timestamp":"2024-06-03T08:43:51.133Z","sequence":1777,"loggerClassName":"org.apache.commons.logging.impl.JBossLog","loggerName":"org.apache.http.headers","level":"
DEBUG","message":"http-outgoing-0 << Date: Mon, 03 Jun 2024 08:43:50 GMT","threadName":"pool-12-thread-1","threadId":32,"mdc":{},"ndc":"","hostName":"rest-api-qa-86b7799858-68skq","processName":"quarkus-ru
n.jar","processId":1}
2024-06-03T08:43:51.134064129Z stdout F {"timestamp":"2024-06-03T08:43:51.133Z","sequence":1778,"loggerClassName":"org.apache.commons.logging.impl.JBossLog","loggerName":"org.apache.http.headers","level":"
DEBUG","message":"http-outgoing-0 << Connection: close","threadName":"pool-12-thread-1","threadId":32,"mdc":{},"ndc":"","hostName":"rest-api-qa-86b7799858-68skq","processName":"quarkus-run.jar","processId"
:1}
2024-06-03T08:43:51.1345481Z stdout F {"timestamp":"2024-06-03T08:43:51.134Z","sequence":1779,"loggerClassName":"org.slf4j.impl.Slf4jLogger","loggerName":"software.amazon.awssdk.requestId","level":"DEBUG",
"message":"Received failed response: 400, Request ID: 125decf5-51f6-46cd-be57-af4919a3abff, Extended Request ID: not available","threadName":"pool-12-thread-1","threadId":32,"mdc":{},"ndc":"","hostName":"r
est-api-qa-86b7799858-68skq","processName":"quarkus-run.jar","processId":1}
2024-06-03T08:43:51.134570519Z stdout F {"timestamp":"2024-06-03T08:43:51.134Z","sequence":1780,"loggerClassName":"org.slf4j.impl.Slf4jLogger","loggerName":"software.amazon.awssdk.request","level":"DEBUG",
"message":"Received failed response: 400, Request ID: 125decf5-51f6-46cd-be57-af4919a3abff, Extended Request ID: not available","threadName":"pool-12-thread-1","threadId":32,"mdc":{},"ndc":"","hostName":"r
est-api-qa-86b7799858-68skq","processName":"quarkus-run.jar","processId":1}
2024-06-03T08:43:51.135589531Z stdout F {"timestamp":"2024-06-03T08:43:51.135Z","sequence":1781,"loggerClassName":"org.apache.commons.logging.impl.JBossLog","loggerName":"org.apache.http.wire","level":"DEB
UG","message":"http-outgoing-0 << "{"__type":"InvalidParameterException","message":"Upload too large: 1064089 bytes exceeds limit of 1048576"}"","threadName":"pool-12-thread-1","threadId":32,"mdc
":{},"ndc":"","hostName":"rest-api-qa-86b7799858-68skq","processName":"quarkus-run.jar","processId":1}
2024-06-03T08:43:51.135661173Z stdout F {"timestamp":"2024-06-03T08:43:51.135Z","sequence":1782,"loggerClassName":"org.apache.commons.logging.impl.JBossLog","loggerName":"org.apache.http.impl.conn.DefaultM
anagedHttpClientConnection","level":"DEBUG","message":"http-outgoing-0: Close connection","threadName":"pool-12-thread-1","threadId":32,"mdc":{},"ndc":"","hostName":"rest-api-qa-86b7799858-68skq","processN
ame":"quarkus-run.jar","processId":1}
2024-06-03T08:43:51.135938688Z stdout F {"timestamp":"2024-06-03T08:43:51.135Z","sequence":1783,"loggerClassName":"org.slf4j.impl.Slf4jLogger","loggerName":"software.amazon.awssdk.http.apache.internal.net.
SdkSslSocket","level":"DEBUG","message":"closing logs.us-east-1.amazonaws.com/3.236.94.242:443","threadName":"pool-12-thread-1","threadId":32,"mdc":{},"ndc":"","hostName":"rest-api-qa-86b7799858-68skq","pr
ocessName":"quarkus-run.jar","processId":1}
2024-06-03T08:43:51.136365877Z stdout F {"timestamp":"2024-06-03T08:43:51.136Z","sequence":1784,"loggerClassName":"org.apache.commons.logging.impl.JBossLog","loggerName":"org.apache.http.impl.execchain.Mai
nClientExec","level":"DEBUG","message":"Connection discarded","threadName":"pool-12-thread-1","threadId":32,"mdc":{},"ndc":"","hostName":"rest-api-qa-86b7799858-68skq","processName":"quarkus-run.jar","proc
essId":1}

i have attached entire logfile.

Regards
Shekhar

app_logfile.gz

[gwenneg]

Hey @mukshe01. There's something interesting in the errors you posted:

Upload too large: 1064089 bytes exceeds limit of 1048576

The extension sends a batch of log entries to CloudWatch. If the log entries are big enough, the size of a batch sometimes exceeds the maximum limit allowed by CloudWatch. Setting quarkus.log.cloudwatch.batch-size to a value lower than 10000 (default value) in your application.properties file might help fix your issue.

[mukshe01]

Hi @gwenneg ,

Thank you for your response, we have tried updating batch size from 10000 to 5000 then to 1000, we still see the issue.

Below are the environment variables we are passing to our app, are these environment variables look correct?,

QUARKUS_LOG_CLOUDWATCH_LOG_GROUP:                                /aws/containerinsights/one-source-qa-n5p1P1d1/application-new
     QUARKUS_LOG_CLOUDWATCH_LOG_STREAM_NAME:                          participant1-qa.rest-api-cw
     QUARKUS_LOG_CLOUDWATCH_BATCH_SIZE:                               1000
     QUARKUS_LOG_CLOUDWATCH_MAX_QUEUE_SIZE:                           50000000
     QUARKUS_LOG_CLOUDWATCH_BATCH_PERIOD:                             2s

we are still seeing the error

","message":"Received failed response: 400, Request ID: 945f0bd9-0fbe-419d-8892-f2e549d93a9b, Extended Request ID: not available","threadName":"pool-12-thread-1","threadId":32,"mdc":{},"ndc":"","hostName":
"rest-api-qa-5c5cc46dc-qlhbj","processName":"quarkus-run.jar","processId":1}
2024-06-04T13:36:45.503614412Z stdout F {"timestamp":"2024-06-04T13:36:45.503Z","sequence":1708,"loggerClassName":"org.slf4j.impl.Slf4jLogger","loggerName":"software.amazon.awssdk.request","level":"DEBUG",
"message":"Received failed response: 400, Request ID: 945f0bd9-0fbe-419d-8892-f2e549d93a9b, Extended Request ID: not available","threadName":"pool-12-thread-1","threadId":32,"mdc":{},"ndc":"","hostName":"r
est-api-qa-5c5cc46dc-qlhbj","processName":"quarkus-run.jar","processId":1}
2024-06-04T13:36:45.504961956Z stdout F {"timestamp":"2024-06-04T13:36:45.504Z","sequence":1709,"loggerClassName":"org.apache.commons.logging.impl.JBossLog","loggerName":"org.apache.http.wire","level":"DEB
UG","message":"http-outgoing-0 << \"{\"__type\":\"InvalidParameterException\",\"message\":\"Upload too large: 1063020 bytes exceeds limit of 1048576\"}\"","threadName":"pool-12-thread-1","threadId":32,"mdc
":{},"ndc":"","hostName":"rest-api-qa-5c5cc46dc-qlhbj","processName":"quarkus-run.jar","processId":1}
2024-06-04T13:36:45.505110638Z stdout F {"timestamp":"2024-06-04T13:36:45.505Z","sequence":1710,"loggerClassName":"org.apache.commons.logging.impl.JBossLog","loggerName":"org.apache.http.impl.conn.DefaultM
anagedHttpClientConnection","level":"DEBUG","message":"http-outgoing-0: Close connection","threadName":"pool-12-thread-1","threadId":32,"mdc":{},"ndc":"","hostName":"rest-api-qa-5c5cc46dc-qlhbj","processNa
me":"quarkus-run.jar","processId":1}
2024-06-04T13:36:45.505404201Z stdout F {"timestamp":"2024-06-04T13:36:45.505Z","sequence":1711,"loggerClassName":"org.slf4j.impl.Slf4jLogger","loggerName":"software.amazon.awssdk.http.apache.internal.net.
SdkSslSocket","level":"DEBUG","message":"closing logs.us-east-1.amazonaws.com/3.236.94.234:443","threadName":"pool-12-thread-1","threadId":32,"mdc":{},"ndc":"","hostName":"rest-api-qa-5c5cc46dc-qlhbj","pro
cessName":"quarkus-run.jar","processId":1}
2024-06-04T13:36:45.505830914Z stdout F {"timestamp":"2024-06-04T13:36:45.505Z","sequence":1712,"loggerClassName":"org.apache.commons.logging.impl.JBossLog","loggerName":"org.apache.http.impl.execchain.Mai
nClientExec","level":"DEBUG","message":"Connection discarded","threadName":"pool-12-thread-1","threadId":32,"mdc":{},"ndc":"","hostName":"rest-api-qa-5c5cc46dc-qlhbj","processName":"quarkus-run.jar","proce
ssId":1}
2024-06-04T13:36:45.505886748Z stdout F {"timestamp":"2024-06-04T13:36:45.505Z","sequence":1713,"loggerClassName":"org.apache.commons.logging.impl.JBossLog","loggerName":"org.apache.http.impl.conn.PoolingH
ttpClientConnectionManager","level":"DEBUG","message":"Connection released: [id: 0][route: {s}->https://logs.us-east-1.amazonaws.com:443][total available: 0; route allocated: 0 of 50; total allocated: 0 of
 50]","threadName":"pool-12-thread-1","threadId":32,"mdc":{},"ndc":"","hostName":"rest-api-qa-5c5cc46dc-qlhbj","processName":"quarkus-run.jar","processId":1}

I have uploaded entire log, would you suggest how to fix this.

cloudwatch_extention_logs_20240604.log.gz

Regards
Shekhar