Running the launcher replication-orchestrator failed, WorkerException
#39512
Labels
area/platform
issues related to the platform
community
team/platform-move
type/bug
Something isn't working
Comments
jeffsdata
added
area/platform
|
Tried a different connector - GA4 to CSV - and that didn't change anything. |
|
@jeffsdata, did you upgrade from a previous version? This error previously occurred when the |
marcosmarxm
changed the title
WorkerException
|
This wasn't an upgrade scenario. Another team member created the environment, so I don't know exactly what they did -- but my assumption is he ran a fresh install using the most up-to-date helm chart provided (since that is the app version installed). I'll check with them to find out if they did anything weird, though. |
|
Turns out, this was just a problem with the Helm chart version. They updated to the most recent version, redeployed, and that seems to have fixed this. |
|
This issue has come back for me. I'm working with our internal team, but just wanted to update that it worked for about 3 weeks, and then the last 3 days it's just been throwing the original error -- no logs, just: |
|
@marcosmarxm same for me, happens with |
|
We ended up clearing our log files in "airbyte storage" and that got our jobs running again. It was about half full - we'd allocated 1GB and it had about 475MB worth of logs. |
|
@jeffsdata thanks for updating. unfortunately it didn't work for us @marcosmarxm any idea what could cause this? might be related to temporal state somehow? |
|
We're facing the same issue here. We migrated from a previous version (v0.50.35) to the latest (v0.63.14) and now none of our connections work. In addition to the logs posted by @jeffsdata, we're also getting this message:
|
|
+1 |
|
Same issue here. It starts even on a new install, after any subsequent Full Exception: I see a 401 Authorization issues somewhere in the massive stack trace. EDIT: the only way I can get things working again is if I do a |
|
for us the solution was to increase worker replicas to be |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Helm Chart Version
Chart Version: 0.163.0 App Version: 0.63.0
What step the error happened?
During the Sync
Relevant information
We just installed Airbyte into a new environment. I am unable to successfully run any syncs - I've set up three different connections and none of them work.
A warning comes almost immediately:
Warning from replication: Something went wrong during replication Copy text message='io.temporal.serviceclient.CheckedExceptionWrapper: io.airbyte.workers.exception.WorkerException: Running the launcher replication-orchestrator failed', type='java.lang.RuntimeException', nonRetryable=falseSometimes, seemingly randomly, the sync is able to get data (and it'll say "1,000,000 records extracted | 700,000 records loaded") but it never actually creates the final tables. However, most of the attempts have no logs at all. Also just a note... this error is an "ERROR" in an old environment (app version 0.55.0) and a warning in this new environment (app version 0.63.0) - but both environments have this error.
I read some of the past commentary about this, and I think we're going to try setting
CONTAINER_ORCHESTRATOR_ENABLED=false. Just figured I'd record this problem.Relevant log output
The text was updated successfully, but these errors were encountered: