Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[GOBBLIN-1910]Refactor dag manager to reduce bulkiness for adhoc REST calls for launch, resume and kill #3776

Open
wants to merge 7 commits into
base: master
Choose a base branch
from
Open

[GOBBLIN-1910]Refactor dag manager to reduce bulkiness for adhoc REST calls for launch, resume and kill #3776

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
b80ecd1
initial commit with high-level design for refactoring DagManager
Aug 31, 2023
ab16ac2
initial commit for refactoring dag manager handling adhoc REST calls …
Sep 14, 2023
3c15b90
address PR comments and add advance and clean up dag tasks
Sep 18, 2023
262998b
modify to provide impl. for kill dag action type only
Sep 20, 2023
d3000ae
address PR comments and add retries for processing Dag via DagProc
Sep 26, 2023
74442ea
add apis to dag management state store and update logic for flow/job …
Sep 27, 2023
e675e50
add back restspec and snapshot json
Sep 27, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions gobblin-api/src/main/java/org/apache/gobblin/service/ServiceConfigKeys.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,7 @@ public class ServiceConfigKeys {
public static final boolean DEFAULT_GOBBLIN_SERVICE_DAG_MANAGER_ENABLED = false;
public static final String GOBBLIN_SERVICE_JOB_STATUS_MONITOR_ENABLED_KEY = GOBBLIN_SERVICE_PREFIX + "jobStatusMonitor.enabled";
public static final String GOBBLIN_SERVICE_WARM_STANDBY_ENABLED_KEY = GOBBLIN_SERVICE_PREFIX + "warmStandby.enabled";
public static final String GOBBLIN_SERVICE_DAG_PROCESSING_ENGINE_ENABLED_KEY = GOBBLIN_SERVICE_PREFIX + "dagProcessingEngine.enabled";
public static final String GOBBLIN_SERVICE_MULTI_ACTIVE_SCHEDULER_ENABLED_KEY = GOBBLIN_SERVICE_PREFIX + "multiActiveScheduler.enabled";
// If true, will mark up/down d2 servers on leadership so that all requests will be routed to the leader node
public static final String GOBBLIN_SERVICE_D2_ONLY_ANNOUNCE_LEADER = GOBBLIN_SERVICE_PREFIX + "d2.onlyAnnounceLeader";
Expand Down
350 changes: 0 additions & 350 deletions ...ce/gobblin-rest-api/src/main/snapshot/org.apache.gobblin.rest.jobExecutions.snapshot.json
View file
Open in desktop
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this meant to be a part of ur change? was this done or included by mistake?

This file was deleted.

4 changes: 4 additions & 0 deletions ...ce/src/main/java/org/apache/gobblin/service/modules/core/GobblinServiceConfiguration.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,9 @@ public class GobblinServiceConfiguration {
@Getter
private final boolean onlyAnnounceLeader;

@Getter
private final boolean isDagProcessingEngineEnabled;

@Getter
private final Config innerConfig;

Expand Down Expand Up @@ -124,5 +127,6 @@ public GobblinServiceConfiguration(String serviceName, String serviceId, Config
this.isTopologySpecFactoryEnabled =
ConfigUtils.getBoolean(config, ServiceConfigKeys.GOBBLIN_SERVICE_TOPOLOGY_SPEC_FACTORY_ENABLED_KEY, true);
this.onlyAnnounceLeader = ConfigUtils.getBoolean(config, ServiceConfigKeys.GOBBLIN_SERVICE_D2_ONLY_ANNOUNCE_LEADER, false);
this.isDagProcessingEngineEnabled = ConfigUtils.getBoolean(config, ServiceConfigKeys.GOBBLIN_SERVICE_DAG_PROCESSING_ENGINE_ENABLED_KEY, false);
}
}
Loading