[GOBBLIN-2170] Define Gobblin-on-Temporal dynamic ScalingDirectives with parser and FsScalingDirectiveSource

[phet]

Dear Gobblin maintainers,

Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!

JIRA

• My PR addresses the following Gobblin JIRA issues and references them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"
  • https://issues.apache.org/jira/browse/GOBBLIN-2170

Description

• Here are some details about my PR, including screenshots (if applicable):

Enable Gobblin-on-Temporal to dynamically scale up/down the number of workers and also to reconfigure worker containers (including resource resizing). Support a heterogeneous workforce of multiple simultaneous worker container configurations.

The ScalingDirective, WorkforcePlan, and WorkforceStaffing abstractions will allow implementing with the current GoT-atop-YARN deployment, while also being general enough for containerization beyond YARN, such as with k8s.

The ScalingDirectiveParser supports a simple and compact human-readable syntax for Dynamic Scaling Directives that is "FS-safe", so each DSD may appear as a filename w/o excessive encoding. This enables the FsScalingDirectiveSource to function as a source of Dynamic Scaling Directives by reading entries from a HDFS directory ordered by file modtime.

Suggested review sequence:

• ScalingDirective => WorkforceStaffing => WorkforcePlan (with WorkforceStaffingTest and WorkforcePlanTest along the way)
• ScalingDirectiveParser (with ScalingDirectiveParserTest)
• FsScalingDirectiveSource (with FsScalingDirectiveSourceTest)

Tests

• My PR adds the following unit tests OR does not need testing for this extremely good reason:

unit tests included

Commits

• My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":
  1. Subject is separated from body by a blank line
  2. Subject is limited to 50 characters
  3. Subject does not end with a period
  4. Subject uses the imperative mood ("add", not "adding")
  5. Body wraps at 72 characters
  6. Body explains "what" and "why", not "how"

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 50.51%. Comparing base (edaf474) to head (a31969e).

Additional details and impacted files

@@             Coverage Diff              @@
##             master    #4068      +/-   ##
============================================
- Coverage     51.47%   50.51%   -0.97%     
+ Complexity     7546     5964    -1582     
============================================
  Files          1386     1079     -307     
  Lines         52105    41523   -10582     
  Branches       5727     4658    -1069     
============================================
- Hits          26821    20975    -5846     
+ Misses        22990    18760    -4230     
+ Partials       2294     1788     -506     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.