Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement Workflows in COAL-SDS #9

Open
lewismc opened this issue Feb 27, 2018 · 4 comments
Open

Implement Workflows in COAL-SDS #9

lewismc opened this issue Feb 27, 2018 · 4 comments
Assignees
Labels
enhancement help wanted workflowmgr OODT Workflow Management Service
Milestone

Comments

@lewismc
Copy link
Member

lewismc commented Feb 27, 2018

For full end-to-end automation, we need to eventually implement OODT's Workflow Management capabilities.

@lewismc
Copy link
Member Author

lewismc commented Sep 14, 2018

@alexwyi19 @ahuerta33 @DenimMazuki @kristencheung can you please assign yourselves to whatever issue we had previous discussed? Thanks

@lewismc lewismc added the workflowmgr OODT Workflow Management Service label Sep 14, 2018
@lewismc
Copy link
Member Author

lewismc commented Nov 8, 2018

./crawler_launcher --printSupportedCrawlerActions

https://github.com/capstone-coal/coal-sds/wiki/Automation-of-Data-Acquisition-and-Crawling
Flags to add are

-wm,    --workflowMgrUrl <url> which is most likely http://localhost:9001

Additionally you need to add a post ingestion actionId

e.g. 
  Action:
    Id: TriggerPostIngestWorkflow
    Description: Triggers workflow event with the name [ProductType]Ingest
    Phases: [postIngestSuccess]

This should appended to

--actionIds DeleteDataFile

@lewismc
Copy link
Member Author

lewismc commented Nov 8, 2018

The workflow documentation for OODT resides at http://oodt.apache.org/site_docs/cas-workflow/ you can see the bottom left=hand-side for the navigation links to
http://oodt.apache.org/site_docs/cas-workflow/development/developer.html
http://oodt.apache.org/site_docs/cas-workflow/user/basic.html
http://oodt.apache.org/site_docs/cas-workflow/user/advanced.html
I would read them through thoroughly and then begin working within the workflow component for coal-sds.

@lewismc
Copy link
Member Author

lewismc commented Dec 4, 2018

I've committed a lot of code to cover this issue e.g. fc1418d
There is more work to do be to automate the initiation of the workflows directly after the crawler_launcher has successfully ingested the AVIRIS products.
Additionally, the new workflow policy needs to be documented such that workflows are easily understood. This will however be covered in #13

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement help wanted workflowmgr OODT Workflow Management Service
Projects
None yet
Development

No branches or pull requests

4 participants