Schedulis中执行hsql报错org.apache.linkis.orchestrator.ecm.exception.ECMPluginErrorException: errCode: 12003 ,desc: gateway02:9101_0 Failed to async get EngineNode AMErrorException: errCode: 210003 ,desc: Not exists EngineConn(不存在的引擎) ,ip: gateway ,port: 9101 ,serviceKind: linkis-cg-linkismanager ,ip: gateway02 ,port: 9104 ,serviceKind: linkis-cg-entrance

[tezzl1]

在dss中运行hsql正常，在Schedulis调度sh正常，但是调度hsql报错
2024-03-28 14:19:27.664 [INFO ] [qtp1915435938-28 ] o.a.l.e.r.EntranceRestfulApi (151) [submit] [JobId-] - Begin to get an execID
2024-03-28 14:19:27.722 [INFO ] [qtp1915435938-28 ] o.a.l.e.c.CSEntranceHelper$ (151) [resetCreator] [JobId-] - reset creator from schedulis to Schedulis
2024-03-28 14:19:27.817 [INFO ] [qtp1915435938-28 ] o.a.l.e.s.DefaultEntranceServer (80) [execute] [JobId-] - received a request,convert JobRequest{id=117, reqId='null', submitUser='hadoop', executeUser='hadoop', labels=[[key: codeType, value: {"codeType":"hql"}, str: hql], [key: engineType, value: {"engineType":"hive","version":"3.1.3"}, str: hive-3.1.3], [key: userCreator, value: {"creator":"Schedulis","user":"hadoop"}, str: hadoop-Schedulis]], params={configuration={runtime={nodeName=hql_8350, workspace={"workspaceId":228,"workspaceName":"test2","cookies":{"linkis_user_session_ticket_id_v1":"M7UZXQP9Ld0FhgeoLoskCyyGLgIzksAN2HLiVea4FcQ=","dataworkcloud_inner_request":"true","workspaceName":"test2","workspaceId":"228"},"dssUrl":"http://gateway02:29001"}, wds.dss.workflow.submit.user=superadmin, contextID={"type":"HAWorkFlowContextID","value":"{"instance":null,"backupInstance":null,"user":"hadoop","workspace":"test2","project":"test123","flow":"t32","contextId":"8-8--cs_1_devcs_1_dev17","version":"v000001","env":"BDAP_PROD"}"}, nodeType=hql, labels={"route":"dev"}}}, variable={user.to.proxy=hadoop, run_today_h=2024032814, run_date=20240327}, run_date=20240327, labels={userCreator=hadoop-schedulis, codeType=hql, engineType=hive-3.1.3, labels={"route":"dev"}}}, status=Inited}
2024-03-28 14:19:27.823 [INFO ] [qtp1915435938-28 ] o.a.l.e.c.CSEntranceHelper$ (180) [addCSVariable] [JobId-117] - parse variable nodeName: hql_8350
2024-03-28 14:19:27.826 [INFO ] [qtp1915435938-28 ] o.a.l.c.c.Configuration$ (94) [getGateWayURL] [JobId-117] - gatewayUrl is http://127.0.0.1:29001
2024-03-28 14:19:27.838 [INFO ] [Linkis-Default-Scheduler-Thread-15 ] o.a.l.c.c.Configuration$ (94) [getGateWayURL] [JobId-] - gatewayUrl is http://127.0.0.1:29001
2024-03-28 14:19:27.902 [INFO ] [Linkis-Default-Scheduler-Thread-15 ] o.a.l.h.d.DWSHttpClient (160) [addAttempt$1] [JobId-] - invoke http://127.0.0.1:29001/api/rest_j/v1/contextservice/heartbeat taken: 60 ms.
2024-03-28 14:19:28.068 [INFO ] [qtp1915435938-28 ] o.a.l.h.d.DWSHttpClient (160) [addAttempt$1] [JobId-117] - invoke http://127.0.0.1:29001/api/rest_j/v1/contextservice/searchContextValue taken: 221 ms.
2024-03-28 14:19:28.079 [INFO ] [qtp1915435938-28 ] o.a.l.c.c.s.CSVariableService (65) [getUpstreamVariables] [JobId-117] - contextID: 8-8--cs_1_devcs_1_dev17 and nodeName: hql_8350 succeed to getUpstreamVariables size 1
2024-03-28 14:19:28.081 [INFO ] [qtp1915435938-28 ] o.a.l.e.c.CSEntranceHelper$ (200) [addCSVariable] [JobId-117] - parse variable end nodeName: hql_8350
2024-03-28 14:19:28.081 [INFO ] [qtp1915435938-28 ] o.a.l.e.c.CSEntranceHelper$ (123) [initNodeCSInfo] [JobId-117] - init node(hql_8350) cs info
2024-03-28 14:19:28.107 [INFO ] [qtp1915435938-28 ] o.a.l.h.d.DWSHttpClient (160) [addAttempt$1] [JobId-117] - invoke http://127.0.0.1:29001/api/rest_j/v1/contextservice/removeAllValueByKeyPrefixAndContextType taken: 23 ms.
2024-03-28 14:19:28.109 [INFO ] [qtp1915435938-28 ] o.a.l.c.c.s.CSNodeServiceImpl (59) [initNodeCSInfo] [JobId-117] - contextIDStr: {"type":"HAWorkFlowContextID","value":"{"instance":null,"backupInstance":null,"user":"hadoop","workspace":"test2","project":"test123","flow":"t32","contextId":"8-8--cs_1_devcs_1_dev17","version":"v000001","env":"BDAP_PROD"}"} and nodeName: hql_8350 init cs info
2024-03-28 14:19:28.901 [WARN ] [qtp1915435938-28 ] o.a.h.u.NativeCodeLoader (60) [] [JobId-117] - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2024-03-28 14:19:29.206 [INFO ] [qtp1915435938-28 ] o.a.l.e.i.i.TenantLabelSetUtils$ (55) [$anonfun$load$1] [JobId-117] - load tenant data user hadoop creator schedulis data
2024-03-28 14:19:29.220 [WARN ] [qtp1915435938-28 ] o.a.l.e.i.i.TenantLabelSetUtils$ (64) [load] [JobId-117] - TenantCache data loading failed , plaese check warn log
2024-03-28 14:19:29.220 [INFO ] [qtp1915435938-28 ] o.a.l.e.i.i.TenantLabelSetUtils$ (55) [$anonfun$load$1] [JobId-117] - load tenant data user * creator schedulis data
2024-03-28 14:19:29.229 [WARN ] [qtp1915435938-28 ] o.a.l.e.i.i.TenantLabelSetUtils$ (64) [load] [JobId-117] - TenantCache data loading failed , plaese check warn log
2024-03-28 14:19:29.229 [INFO ] [qtp1915435938-28 ] o.a.l.e.i.i.TenantLabelSetUtils$ (55) [$anonfun$load$1] [JobId-117] - load tenant data user hadoop creator * data
2024-03-28 14:19:29.237 [WARN ] [qtp1915435938-28 ] o.a.l.e.i.i.TenantLabelSetUtils$ (64) [load] [JobId-117] - TenantCache data loading failed , plaese check warn log
2024-03-28 14:19:29.237 [INFO ] [qtp1915435938-28 ] o.a.l.e.i.i.TenantLabelSetUtils$ (103) [checkTenantLabel] [JobId-117] - get cache tenant:,jobRequest:117
2024-03-28 14:19:29.292 [INFO ] [qtp1915435938-28 ] o.a.l.s.f.FileSystem (102) [getParentPath] [JobId-117] - Get Parent Path:/tmp/linkis/log/2024-03-28/Schedulis/hadoop
2024-03-28 14:19:29.363 [INFO ] [qtp1915435938-28 ] o.a.l.s.u.FileSystemUtils (94) [createNewFileWithFileSystem] [JobId-117] - doesn't need to call setOwner
2024-03-28 14:19:29.363 [INFO ] [qtp1915435938-28 ] o.a.l.e.l.CacheLogManager (99) [createLogWriter] [JobId-117] - job 117 create cacheLogWriter
2024-03-28 14:19:29.372 [INFO ] [qtp1915435938-28 ] o.a.l.e.s.EntranceGroupFactory (112) [getOrCreateGroup] [JobId-117] - Getting user configurations for Schedulis_hadoop_hive userCreatorLabel: hadoop-Schedulis, engineTypeLabel:hive-3.1.3.
2024-03-28 14:19:29.435 [INFO ] [qtp1915435938-28 ] o.a.l.e.s.EntranceGroupFactory (178) [getUserMaxRunningJobs] [JobId-117] - There are 0 offlining instance.
2024-03-28 14:19:29.435 [INFO ] [qtp1915435938-28 ] o.a.l.e.s.EntranceGroupFactory (138) [getOrCreateGroup] [JobId-117] - Got user configurations: groupName=Schedulis_hadoop_hive, maxRunningJobs=20, initCapacity=100, maxCapacity=2000.
2024-03-28 14:19:29.439 [INFO ] [Schedulis_hadoop_hiveConsumerThread ] o.a.l.s.q.f.FIFOUserConsumer (84) [run] [JobId-] - Schedulis_hadoop_hiveConsumer thread started!
2024-03-28 14:19:29.441 [INFO ] [qtp1915435938-28 ] o.a.l.e.s.DefaultEntranceServer (161) [$anonfun$execute$5] [JobId-117] - 2024-03-28 14:19:29.019 INFO Job with jobId : 117 and execID : Schedulis_hadoop_hive_0 submitted
2024-03-28 14:19:29.443 [INFO ] [qtp1915435938-28 ] o.a.l.s.u.ModuleUserUtils (68) [getProxyUserEntity] [JobId-] - user hadoop proxy to null operation submit jobReqId: 117
2024-03-28 14:19:29.444 [INFO ] [Schedulis_hadoop_hiveConsumerThread ] o.a.l.e.j.EntranceExecutionJob (97) [transition] [JobId-] - jobGroupId:117 execID:Schedulis_hadoop_hive_0 change status Inited => Scheduled.
2024-03-28 14:19:29.450 [INFO ] [qtp1915435938-28 ] o.a.l.e.r.EntranceRestfulApi (193) [submit] [JobId-] - End to get an an execID: exec_id018026linkis-cg-entrancegateway02:9104Schedulis_hadoop_hive_0, taskID: 117
2024-03-28 14:19:29.479 [INFO ] [linkis-cg-entrance-EventListenerBus-Thread-0] o.a.l.e.e.EntranceEventListenerBus (299) [run] [JobId-] - linkis-cg-entrance-EventListenerBus-Thread-0 begin.
2024-03-28 14:19:29.484 [INFO ] [EntranceJobScheduler-ThreadPool-2 ] o.a.l.e.j.EntranceExecutionJob (97) [transition] [JobId-] - jobGroupId:117 execID:Schedulis_hadoop_hive_0 change status Scheduled => Running.
2024-03-28 14:19:29.503 [INFO ] [qtp1915435938-22 ] o.a.l.s.u.ModuleUserUtils (68) [getProxyUserEntity] [JobId-] - user hadoop proxy to null operation progress realId: Schedulis_hadoop_hive_0
2024-03-28 14:19:29.538 [INFO ] [EntranceJobScheduler-ThreadPool-2 ] o.a.l.o.Orchestrator$ (65) [getOrchestrator] [JobId-117] - Use org.apache.linkis.orchestrator.Orchestrator$$$Lambda$1363/762407719 to instance a new orchestrator.
2024-03-28 14:19:29.614 [INFO ] [EntranceJobScheduler-ThreadPool-2 ] o.a.l.o.c.i.SessionStateImpl$$anon$1 (51) [convert] [JobId-117] - Finished to convert JobReq(117) to AstJob(TaskID_117_otJobId_astJob_0).
2024-03-28 14:19:29.631 [INFO ] [EntranceJobScheduler-ThreadPool-2 ] o.a.l.o.l.OrchestratorAsyncListenerBus (40) [addListener] [JobId-117] - Orchestrator-Listener-Asyn-Thread-ListenerBus add a new listener => class org.apache.linkis.orchestrator.code.plans.execution.CodeReheaterNotifyTaskConsumer
2024-03-28 14:19:29.631 [INFO ] [EntranceJobScheduler-ThreadPool-2 ] o.a.l.o.l.OrchestratorSyncListenerBus (40) [addListener] [JobId-117] - org.apache.linkis.orchestrator.listener.OrchestratorSyncListenerBus@5e33423d add a new listener => class org.apache.linkis.orchestrator.execution.impl.ExecutionImpl
2024-03-28 14:19:29.631 [INFO ] [EntranceJobScheduler-ThreadPool-2 ] o.a.l.o.l.OrchestratorSyncListenerBus (40) [addListener] [JobId-117] - org.apache.linkis.orchestrator.listener.OrchestratorSyncListenerBus@5e33423d add a new listener => class org.apache.linkis.orchestrator.strategy.async.AsyncTaskManager
2024-03-28 14:19:29.631 [INFO ] [EntranceJobScheduler-ThreadPool-2 ] o.a.l.o.e.i.ExecutionImpl (47) [start] [JobId-117] - execution start
2024-03-28 14:19:29.632 [INFO ] [EntranceJobScheduler-ThreadPool-2 ] o.a.l.o.c.p.e.CodeReheaterNotifyTaskConsumer (33) [start] [JobId-117] - start consumer CodeReheaterNotifyTaskConsumer success
2024-03-28 14:19:29.633 [INFO ] [EntranceJobScheduler-ThreadPool-2 ] o.a.l.o.c.i.SessionStateImpl (135) [getExecution] [JobId-117] - Finished to create execution org.apache.linkis.orchestrator.execution.impl.ExecutionImpl@5673ba81
2024-03-28 14:19:29.669 [INFO ] [EntranceJobScheduler-ThreadPool-2 ] o.a.l.o.s.a.AsyncTaskManager (89) [putExecTask] [JobId-117] - submit execTask TaskID_117_otJobId_astJob_0_job_0 to taskManager get executionTask execution_0
2024-03-28 14:19:29.671 [INFO ] [Orchestrator-Listener-Asyn-Thread-Thread-0] o.a.l.o.l.OrchestratorAsyncListenerBus (299) [run] [JobId-] - Orchestrator-Listener-Asyn-Thread-Thread-0 begin.
2024-03-28 14:19:29.672 [INFO ] [EntranceJobScheduler-ThreadPool-2 ] o.a.l.e.e.DefaultEntranceExecutor (283) [$anonfun$callExecute$1] [JobId-117] - JobRequest (117) was submitted to Orchestrator.
2024-03-28 14:19:29.675 [INFO ] [EntranceJobScheduler-ThreadPool-2 ] o.a.l.o.l.OrchestratorAsyncListenerBus (40) [addListener] [JobId-117] - Orchestrator-Listener-Asyn-Thread-ListenerBus add a new listener => class org.apache.linkis.orchestrator.computation.operation.log.LogOperation
2024-03-28 14:19:29.677 [INFO ] [EntranceJobScheduler-ThreadPool-2 ] o.a.l.o.l.OrchestratorAsyncListenerBus (40) [addListener] [JobId-117] - Orchestrator-Listener-Asyn-Thread-ListenerBus add a new listener => class org.apache.linkis.orchestrator.computation.operation.progress.DefaultProgressOperation
2024-03-28 14:19:29.680 [INFO ] [EntranceJobScheduler-ThreadPool-2 ] o.a.l.e.e.DefaultEntranceExecutor (313) [$anonfun$callExecute$1] [JobId-117] - For job 117 and orchestrator task id 117 to create EngineExecuteAsyncReturn
2024-03-28 14:19:29.694 [INFO ] [CodeReheaterNotifyTaskConsumer ] o.a.l.o.s.a.AsyncTaskManager (197) [$anonfun$taskRunnableTasks$2] [JobId-] - user key hadoop-Schedulis,hive-3.1.3, executionTaskId execution_0 to addNumber: 1
2024-03-28 14:19:29.697 [INFO ] [BaseTaskScheduler-Thread-1 ] o.a.l.o.s.a.AsyncExecTaskRunnerImpl (69) [$anonfun$run$1] [JobId-117] - ExecTaskRunner Submit execTask(TaskID_117_otJobId_astJob_0_codeExec_0) to running
2024-03-28 14:19:29.697 [INFO ] [BaseTaskScheduler-Thread-1 ] o.a.l.o.c.p.CodeLogicalUnitExecTask (85) [execute] [JobId-117] - Start to execute CodeLogicalUnitExecTask(TaskID_117_otJobId_astJob_0_codeExec_0).
2024-03-28 14:19:29.700 [INFO ] [BaseTaskScheduler-Thread-1 ] o.a.l.o.e.ComputationEngineConnManager (194) [addMark] [JobId-117] - add mark mark_0
2024-03-28 14:19:29.700 [INFO ] [BaseTaskScheduler-Thread-1 ] o.a.l.o.c.e.DefaultCodeExecTaskExecutorManager (127) [createExecutor] [JobId-117] - create Executor for execId TaskID_117_otJobId_astJob_0_codeExec_0 mark id is mark_0, user hadoop
2024-03-28 14:19:29.701 [INFO ] [BaseTaskScheduler-Thread-1 ] o.a.l.o.e.ComputationEngineConnManager (125) [getAvailableEngineConnExecutor] [JobId-117] - mark mark_0 start to getAvailableEngineConnExecutor
2024-03-28 14:19:29.742 [INFO ] [BaseTaskScheduler-Thread-1 ] c.n.c.ChainedDynamicProperty (115) [checkAndFlip] [JobId-117] - Flipping property: linkis-cg-linkismanager.ribbon.ActiveConnectionsLimit to use NEXT property: niws.loadbalancer.availabilityFilteringRule.activeConnectionsLimit = 2147483647
2024-03-28 14:19:29.743 [INFO ] [BaseTaskScheduler-Thread-1 ] c.n.u.c.ShutdownEnabledTimer (58) [] [JobId-117] - Shutdown hook installed for: NFLoadBalancer-PingTimer-linkis-cg-linkismanager
2024-03-28 14:19:29.743 [INFO ] [BaseTaskScheduler-Thread-1 ] c.n.l.BaseLoadBalancer (197) [initWithConfig] [JobId-117] - Client: linkis-cg-linkismanager instantiated a LoadBalancer: DynamicServerListLoadBalancer:{NFLoadBalancer:name=linkis-cg-linkismanager,current list of Servers=[],Load balancer stats=Zone stats: {},Server stats: []}ServerList:null
2024-03-28 14:19:29.745 [INFO ] [BaseTaskScheduler-Thread-1 ] c.n.c.ChainedDynamicProperty (115) [checkAndFlip] [JobId-117] - Flipping property: linkis-cg-linkismanager.ribbon.ActiveConnectionsLimit to use NEXT property: niws.loadbalancer.availabilityFilteringRule.activeConnectionsLimit = 2147483647
2024-03-28 14:19:29.837 [INFO ] [BaseTaskScheduler-Thread-1 ] o.a.l.o.e.ComputationEngineConnManager (156) [getEngineNodeAskManager] [JobId-117] - mark_0 received EngineAskAsyncResponse id: gateway02:9101_0 serviceInstance: ServiceInstance(linkis-cg-linkismanager, gateway02:9101)
2024-03-28 14:19:30.235 [INFO ] [RPC-Receiver-Asyn-Thread-Thread-0 ] o.a.l.r.AsynRPCMessageBus (299) [run] [JobId-] - RPC-Receiver-Asyn-Thread-Thread-0 begin.
2024-03-28 14:19:30.239 [INFO ] [RPC-Receiver-Asyn-Thread-Thread-0 ] o.a.l.r.MessageReceiver (35) [receive] [JobId-] - From caller ServiceInstance(linkis-cg-linkismanager, gateway02:9101) get async message
2024-03-28 14:19:30.239 [INFO ] [RPC-Receiver-Asyn-Thread-Thread-0 ] o.a.l.r.MessageReceiver (40) [receiveAndReply] [JobId-] - From caller ServiceInstance(linkis-cg-linkismanager, gateway02:9101) get sync message
2024-03-28 14:19:30.245 [INFO ] [RPC-Receiver-Asyn-Thread-Thread-0 ] o.a.l.r.m.p.DefaultServiceParser (79) [getServiceMethod] [JobId-] - public void org.apache.linkis.orchestrator.computation.service.ComputationTaskExecutionReceiver.taskProgressReceiver(org.apache.linkis.manager.common.protocol.resource.ResponseTaskRunningInfo,org.apache.linkis.rpc.Sender) parameter:[taskProgressWithResource, sender]
2024-03-28 14:19:30.245 [INFO ] [RPC-Receiver-Asyn-Thread-Thread-0 ] o.a.l.r.m.p.DefaultServiceParser (79) [getServiceMethod] [JobId-] - public void org.apache.linkis.orchestrator.computation.service.ComputationTaskExecutionReceiver.taskStatusReceiver(org.apache.linkis.governance.common.protocol.task.ResponseTaskStatus,org.apache.linkis.rpc.Sender) parameter:[taskStatus, sender]
2024-03-28 14:19:30.245 [INFO ] [RPC-Receiver-Asyn-Thread-Thread-0 ] o.a.l.r.m.p.DefaultServiceParser (79) [getServiceMethod] [JobId-] - public void org.apache.linkis.orchestrator.computation.service.ComputationTaskExecutionReceiver.taskResultSizeReceiver(org.apache.linkis.governance.common.protocol.task.ResponseTaskResultSize,org.apache.linkis.rpc.Sender) parameter:[taskResultSize, sender]
2024-03-28 14:19:30.245 [INFO ] [RPC-Receiver-Asyn-Thread-Thread-0 ] o.a.l.r.m.p.DefaultServiceParser (79) [getServiceMethod] [JobId-] - public void org.apache.linkis.orchestrator.computation.service.ComputationTaskExecutionReceiver.taskResultSetReceiver(org.apache.linkis.governance.common.protocol.task.ResponseTaskResultSet,org.apache.linkis.rpc.Sender) parameter:[taskResultSet, sender]
2024-03-28 14:19:30.245 [INFO ] [RPC-Receiver-Asyn-Thread-Thread-0 ] o.a.l.r.m.p.DefaultServiceParser (79) [getServiceMethod] [JobId-] - public void org.apache.linkis.orchestrator.computation.service.ComputationTaskExecutionReceiver.taskErrorReceiver(org.apache.linkis.governance.common.protocol.task.ResponseTaskError,org.apache.linkis.rpc.Sender) parameter:[responseTaskError, sender]
2024-03-28 14:19:30.245 [INFO ] [RPC-Receiver-Asyn-Thread-Thread-0 ] o.a.l.r.m.p.DefaultServiceParser (79) [getServiceMethod] [JobId-] - public void org.apache.linkis.orchestrator.computation.service.ComputationTaskExecutionReceiver.taskLogReceiver(org.apache.linkis.governance.common.protocol.task.ResponseTaskLog,org.apache.linkis.rpc.Sender) parameter:[taskLog, sender]
2024-03-28 14:19:30.247 [INFO ] [RPC-Receiver-Asyn-Thread-Thread-0 ] o.a.l.r.m.p.DefaultServiceParser (79) [getServiceMethod] [JobId-] - public void org.apache.linkis.orchestrator.ecm.service.impl.DefaultEngineAsyncResponseService.onSuccess(org.apache.linkis.manager.common.protocol.engine.EngineCreateSuccess,org.apache.linkis.rpc.Sender) parameter:[engineCreateSuccess, sender]
2024-03-28 14:19:30.252 [INFO ] [RPC-Receiver-Asyn-Thread-Thread-0 ] o.a.l.r.m.p.DefaultServiceParser (79) [getServiceMethod] [JobId-] - public void org.apache.linkis.orchestrator.ecm.service.impl.DefaultEngineAsyncResponseService.onError(org.apache.linkis.manager.common.protocol.engine.EngineCreateError,org.apache.linkis.rpc.Sender) parameter:[engineCreateError, sender]
2024-03-28 14:19:30.255 [INFO ] [RPC-Receiver-Asyn-Thread-Thread-0 ] o.a.l.o.e.s.i.DefaultEngineAsyncResponseService (54) [onError] [JobId-] - Failed to create engine gateway02:9101_0, can retry false
2024-03-28 14:19:30.440 [ERROR] [BaseTaskScheduler-Thread-1 ] o.a.l.o.e.ComputationEngineConnManager (173) [getEngineNodeAskManager] [JobId-117] - mark_0 async id: gateway02:9101_0 Failed to async get EngineNode, AMErrorException: errCode: 210003 ,desc: Not exists EngineConn(不存在的引擎) ,ip: gateway02 ,port: 9101 ,serviceKind: linkis-cg-linkismanager
2024-03-28 14:19:30.440 [WARN ] [BaseTaskScheduler-Thread-1 ] o.a.l.o.e.ComputationEngineConnManager (115) [askEngineConnExecutor] [JobId-117] - mark_0 Failed to askEngineAskRequest time taken (734 ms)
2024-03-28 14:19:30.441 [ERROR] [BaseTaskScheduler-Thread-1 ] o.a.l.o.s.a.AsyncExecTaskRunnerImpl (86) [$anonfun$run$2] [JobId-117] - Failed to execute task TaskID_117_otJobId_astJob_0_codeExec_0 org.apache.linkis.orchestrator.ecm.exception.ECMPluginErrorException: errCode: 12003 ,desc: gateway02:9101_0 Failed 2 to async get EngineNode AMErrorException: errCode: 210003 ,desc: Not exists EngineConn(不存在的引擎) ,ip: gateway02 ,port: 9101 ,serviceKind: linkis-cg-linkismanager ,ip: gateway02 ,port: 9104 ,serviceKind: linkis-cg-entrance
at org.apache.linkis.orchestrator.ecm.ComputationEngineConnManager.getEngineNodeAskManager(ComputationEngineConnManager.scala:183) ~[linkis-computation-orchestrator-1.4.0.jar:1.4.0]
at org.apache.linkis.orchestrator.ecm.ComputationEngineConnManager.askEngineConnExecutor(ComputationEngineConnManager.scala:89) ~[linkis-computation-orchestrator-1.4.0.jar:1.4.0]
at org.apache.linkis.orchestrator.ecm.AbstractEngineConnManager.getAvailableEngineConnExecutor(EngineConnManager.scala:132) ~[linkis-computation-orchestrator-1.4.0.jar:1.4.0]
at org.apache.linkis.orchestrator.computation.execute.DefaultCodeExecTaskExecutorManager.createExecutor(DefaultCodeExecTaskExecutorManager.scala:135) ~[linkis-computation-orchestrator-1.4.0.jar:1.4.0]
at org.apache.linkis.orchestrator.computation.execute.DefaultCodeExecTaskExecutorManager.askExecutor(DefaultCodeExecTaskExecutorManager.scala:101) ~[linkis-computation-orchestrator-1.4.0.jar:1.4.0]
at org.apache.linkis.orchestrator.computation.physical.CodeLogicalUnitExecTask.$anonfun$execute$1(CodeLogicalUnitExecTask.scala:92) ~[linkis-computation-orchestrator-1.4.0.jar:1.4.0]
at org.apache.linkis.common.utils.Utils$.tryCatch(Utils.scala:49) ~[linkis-common-1.4.0.jar:1.4.0]
at org.apache.linkis.orchestrator.computation.physical.CodeLogicalUnitExecTask.execute(CodeLogicalUnitExecTask.scala:92) ~[linkis-computation-orchestrator-1.4.0.jar:1.4.0]
at org.apache.linkis.orchestrator.strategy.async.AsyncExecTaskRunnerImpl.$anonfun$run$1(AsyncExecTaskRunnerImpl.scala:70) ~[linkis-orchestrator-core-1.4.0.jar:1.4.0]
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) ~[scala-library-2.12.17.jar:?]
at org.apache.linkis.common.utils.Utils$.tryCatch(Utils.scala:49) ~[linkis-common-1.4.0.jar:1.4.0]
at org.apache.linkis.orchestrator.strategy.async.AsyncExecTaskRunnerImpl.run(AsyncExecTaskRunnerImpl.scala:85) ~[linkis-orchestrator-core-1.4.0.jar:1.4.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_181]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_181]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_181]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_181]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_181]

2024-03-28 14:19:30.448 [INFO ] [BaseTaskScheduler-Thread-1 ] o.a.l.o.s.a.AsyncExecTaskRunnerImpl (105) [transientStatus] [JobId-117] - TaskID_117_otJobId_astJob_0_codeExec_0 change status Inited => Failed.
2024-03-28 14:19:30.456 [INFO ] [BaseTaskScheduler-Thread-1 ] o.a.l.o.s.a.AsyncTaskManager (250) [addCompletedTask] [JobId-117] - TaskID_117_otJobId_astJob_0_codeExec_0 task completed, now remove from taskManager
2024-03-28 14:19:30.458 [INFO ] [BaseTaskScheduler-Thread-1 ] o.a.l.o.s.a.AsyncTaskManager (285) [$anonfun$addCompletedTask$1] [JobId-117] - executionTask(execution_0) no task running, user key hadoop-Schedulis,hive-3.1.3, minusNumber: 0
2024-03-28 14:19:30.464 [INFO ] [CodeReheaterNotifyTaskConsumer ] o.a.l.o.c.c.r.PruneTaskRetryTransform (52) [$anonfun$apply$1] [JobId-] - task:TaskID_117_otJobId_astJob_0_job_0 has 1 child tasks which execute failed, some of them may be retried
2024-03-28 14:19:30.465 [INFO ] [CodeReheaterNotifyTaskConsumer ] o.a.l.o.s.a.AsyncTaskManager (197) [$anonfun$taskRunnableTasks$2] [JobId-] - user key hadoop-Schedulis,hive-3.1.3, executionTaskId execution_0 to addNumber: 1
2024-03-28 14:19:30.465 [INFO ] [BaseTaskScheduler-Thread-2 ] o.a.l.o.s.a.AsyncExecTaskRunnerImpl (69) [$anonfun$run$1] [JobId-117] - ExecTaskRunner Submit execTask(TaskID_117_otJobId_astJob_0_job_0) to running
2024-03-28 14:19:30.468 [ERROR] [BaseTaskScheduler-Thread-2 ] o.a.l.o.s.GatherStrategyJobExecTask (82) [parseChildrenErrorInfo] [JobId-117] - There are Tasks execution failure of stage TaskID_117_otJobId_astJob_0_job_0, now mark ExecutionTask as failed
2024-03-28 14:19:30.468 [INFO ] [BaseTaskScheduler-Thread-2 ] o.a.l.o.s.a.AsyncTaskManager (407) [onRootTaskResponseEvent] [JobId-117] - received rootTaskResponseEvent TaskID_117_otJobId_astJob_0_job_0
2024-03-28 14:19:30.469 [INFO ] [BaseTaskScheduler-Thread-2 ] o.a.l.o.s.a.AsyncTaskManager (384) [clearExecutionTask] [JobId-117] - executionTask(execution_0) finished user key hadoop-Schedulis,hive-3.1.3
2024-03-28 14:19:30.469 [INFO ] [BaseTaskScheduler-Thread-2 ] o.a.l.o.s.a.AsyncTaskManager (402) [clearExecutionTask] [JobId-117] - executionTask(execution_0) finished user key hadoop-Schedulis,hive-3.1.3, minusNumber: 0
2024-03-28 14:19:30.470 [INFO ] [BaseTaskScheduler-Thread-2 ] o.a.l.o.e.i.BaseExecutionTask (78) [transientStatus] [JobId-117] - execution_0 change status Inited => Failed.
2024-03-28 14:19:30.471 [INFO ] [BaseTaskScheduler-Thread-2 ] o.a.l.o.e.i.ExecutionImpl (86) [apply] [JobId-117] - TaskID_117_otJobId_astJob_0_job_0 completed, Now to remove from execTaskToExecutionTasks
2024-03-28 14:19:30.478 [INFO ] [BaseTaskScheduler-Thread-2 ] o.a.l.e.j.EntranceExecutionJob (97) [transition] [JobId-117] - jobGroupId:117 execID:Schedulis_hadoop_hive_0 change status Running => Failed.