[Tracking issue] Dapr Workflows Stable

[olitomlinson]

This issue will be the source of truth for collating all tasks what must be complete in order for Workflows to progress to Stable, for the 1.15 release.

Dapr Runtime / durabletask-go

• [P0] Scheduler moving to being the default, tracking issue here
  • Migration - State -> Scheduler reminders
• [P0] GetWorkflowStateAsync throws an exception when instanceid is not found #8085
• [P0] Sidecar connection issues with placement service with on-disk raft logs #7749
• [P0] E2E testing of concurrent workflow execution across multiple app replicas #6889
• [P0] Potential workflow -sidecar memory leak #7777
• [P0] GetWorkflowStateAsync "error from internal actor: no such instance exists" #7824
• [P0] [Proposal] Remove usage of DaprClient APIs for wokflow dotnet-sdk#1205
• [P0] [Workflows] Missing eventSent functionality in wfengine #8243
• [P1] Enable distributed trace context to flow to external services that are invoked by a workflow activity #7927
• [P1] [Workflows] Ability to limit the size of inputs and outputs for workflows and activities #6014
• [P1] Creating a child workflow with an ID used by another active workflow should properly fail #7633
• [P2] Proposal to use reminders for internal workflow actors #7934
• [P2] Dapr Workflows loop forever when app is disconnected from sidecar #7706
• [P2] [Workflow] Bleeding out internal details of workflow #8088

Contrib / State Store

• [P0] [AzureSQL] Workflow activity with a large amount of data processing cause error "actor: failed to unmarshal history event" #7570

Quickstarts

• Its Not clear which is the most up-to-date-dotnet Workflow Sample dotnet-sdk#1234

Perf

• [ ]

Docs

• [Workflow] Workflow actor state should be saved with a TTL of -1 #6475

SDKs

.NET SDK

• [P0] Remove obsolete workflow methods from DaprClient dotnet-sdk#1344
• [P0] [Workflow] GRPC connection to workflow runtime doesn't self-heal when app restarts dotnet-sdk#1293
• [P0] [Workflows] Race condition in purge can result in infinite retry loop #8086
• [P0] [Workflows] Returning an entity that was added to an EF context causes the activity to retry. dotnet-sdk#1216
• [P0] Workflow status displays as Unknown after being paused using  dotnet-sdk#1215
• [P0] Add more workflow samples dotnet-sdk#1203
• [P0] Enable specifying Grpc options like " MaxReceiveMessageSize" in WorkflowServiceCollectionExtensions dotnet-sdk#1163 (also requires runtime work)
• [P1] [Workflow] Workflow hangs when running an activity that can't fulfill DI requirements dotnet-sdk#1118
• [P1] Support workflow ID reuse policy dotnet-sdk#1220
• [P1] WorkflowActivityContext cannot be mocked dotnet-sdk#1200
• [P1] Added option to override GrpcChannelOptions when adding DaprWorkflow (#7218) dotnet-sdk#1244 (PR that needs to be reviewed/merged)
• [P2] [Workflows] Expose CancellationToken as part of RunAsync dotnet-sdk#1225

Java SDK

• [P????] [Workflows] Missing sendEvent feature java-sdk#1156
• [P????] [Workflows] Rename callSubWorkflow to callChildWorkflow java-sdk#1155
• [P0] Java Workflow client naming inconsistent with other Workflow SDKs.  java-sdk#1012
• [P1] Support Cascade Terminate/Purge support for Workflow java-sdk#991
• [P1] Support workflow ID reuse policy java-sdk#980
• [P1] Allow register a workflow using String instead of only class canonical name java-sdk#966
• [P1] Use generic and the serializer interface to return the actual object for workflow activity java-sdk#905

JavaScript SDK / durabletask_js - thank you @famarting

• [Workflows] Rename callSubWorkflow to callChildWorkflow js-sdk#643
• grpc worker stream connection should be retried microsoft/durabletask-js#64 (requires counterpart change in dapr sdk)
• WorkflowRuntime.stop() throws Stream error Error: 1 CANCELLED: Cancelled on client microsoft/durabletask-js#47 (requires counterpart change in dapr sdk)
• Support orchestration ID reuse policy microsoft/durabletask-js#30 (requires counterpart change in dapr sdk)
• Durable retry policies support microsoft/durabletask-js#13 (requires counterpart change in dapr sdk)
• workflows: support set custom status js-sdk#638
• grpc: configure client keep alive for grpc connections js-sdk#639

Go SDK - thank you @famarting

• Show how to pass Workflow Activity retry options using go-sdk go-sdk#611
• Workflow - Implement custom retry policies go-sdk#541
• Remove obsolete workflow methods go-sdk#634
• Support workflow ID reuse policy go-sdk#493
• Cascade Terminate/Purge support for Workflow go-sdk#497
• workflows: support set custom status go-sdk#635
• grpc: configure client keep alive for grpc connections go-sdk#636
• Support activity retry policies microsoft/durabletask-go#83

Python SDK - thank you @famarting

• [WORKFLOW SDK FEATURE REQUEST] Support workflow ID reuse policy python-sdk#655
• [WORKFLOW SDK FEATURE REQUEST] Remove obsolete dapr client workflow methods python-sdk#738
• [WORKFLOW SDK FEATURE REQUEST] support set custom status python-sdk#739
• [WORKFLOW SDK FEATURE REQUEST] configure client keep alive for grpc connections python-sdk#740
• [WORKFLOW SDK FEATURE REQUEST] Purge API python-sdk#711
• [WORKFLOW SDK FEATURE REQUEST] Missing send_event feature python-sdk#748
• Missing implementation for send_event microsoft/durabletask-python#32

Deferred / Not essential for Stable

• Setup GitHub token permission in Dapr workflows #3417
• [CosmosDb] [Workflow] Workflows get stuck due to max operation items and max size limitations in certain state stores #7321

[arturotrenard]

Hello, this keeps happening. #7824
@olitomlinson

[mikeee]

@dapr/approvers-dotnet-sdk @dapr/maintainers-dotnet-sdk - Are all of these SDK issues proportionate, necessary, appropriate and is there bandwidth to get them completed for workflows to be stable?

[olitomlinson]

@mikeee I wasn't clear btw, I fully expect the above list to be triaged for suitability by the maintainers for what is in and out of scope for 1.15 :)

[mikeee]

@mikeee I wasn't clear btw, I fully expect the above list to be triaged for suitability by the maintainers for what is in and out of scope for 1.15 :)

Thought so, I'll await the dotnet-sdk maintainers to triage the SDK items

[cgillum]

I'll be helping out with the triage of these items over the next few days.

[cgillum]

@olitomlinson I think #3417 might be referring to GitHub actions workflows and not Dapr Workflow. If you agree, then we can probably take it off this list.

[cicoyle]

I went ahead and added [P_] next to the above issues for us to better see what work is a P0 for this release

[olitomlinson]

@cgillum this should probably go on the list too right? (And fixes for all other language SDKS)

#7218
/ dapr/dotnet-sdk#1244

[famarting]

@olitomlinson @cgillum I'll be going through the go, python and javascript SDKs to review the different changes we need pursuing SDK parity and workflows to become stable

So far this is a list of issue for the go sdk that I believe should be tracked here:

---

Oli : Moved items into main post

[famarting]

Here are the issues for the python SDK, notice the issues that are closed should be re-opened

---

Oli : Moved items into main post

[famarting]

And the js sdk issues, the durabletask-js issues would need their counterpart issues created in the js-sdk repo:

---

Oli : Moved items into main post

[WhitWaldo]

@olitomlinson The top P0 item in the .NET SDK referencing dapr/dotnet-sdk#1344 can be marked off as completed.

So can the P0 item referencing an Unknown status referencing dapr/dotnet-sdk#1215

[cgillum]

Added prioritized list of Java SDK issues to the OP.

[WhitWaldo]

@olitomlinson I'd like to propose adding #7321 to the stabilization list as it's a named issue on the Workflow Limitations, but the issue was closed due to staleness.

[olitomlinson]

I've discovered that the workflow SDKs are inconsistent in the naming of callChildWorkflow / callSubWorkflow

• dotnet, go, and python use callChildWorkflow
• java and javascript use callSubWorkflow

I believe the SDKs should strive for consistency where sensible, and should strive to be consistent with the docs which only refers to child workflows

I've raised tickets to bring the Java and JS SDKs inline with the others :

• [Workflows] Rename callSubWorkflow to callChildWorkflow java-sdk#1155
• [Workflows] Rename callSubWorkflow to callChildWorkflow js-sdk#643

I believe these would both be considered P0 as we want to go into the Stable release with method names that won't change.

Thoughts @yaron2 @artursouza ?

[olitomlinson]

@olitomlinson I'd like to propose adding #7321 to the stabilization list as it's a named issue on the Workflow Limitations, but the issue was closed due to staleness.

I agree. I've reopened this and added this to the main list.

@cgillum what priority should be attached to this item? Are we intending to ship with CosmosDb support in 1.15?

[cgillum]

@olitomlinson per an earlier discussion with @yaron2, I think the best path forward is to have a list of supported state stores for workflow, similar to actors (it would necessarily be a subset of actor state stores). Cosmos DB and similar state stores would not be on that list due to the issue you linked to. Unfortunately, I think there isn't enough time to work through those issues before Stable.

[WhitWaldo]

@cgillum Is there an issue somewhere that tracks progress of the NoSQL support and what's lacking?

[cgillum]

@WhitWaldo I don't believe we have an issue tracking progress of this work. However, I do have an old PR that does the foundational work in the durabletask-go dependency to enable it: microsoft/durabletask-go#50 ("state update chunking"). The main problem has been finding the time to validate and stabilize it.