[toc]
FLOWER is an Acronym for:
FlowEngine For Real Time processing needs
Flower attempts to bridge the gap between code-it-all approach to back-end development versus externalisation of business logic from core execution via DSL.
It is a general purpose, polyglot, configuration driven, DSL based business flow engine designed to democratise backend development.
Caution:
This aims to reduce the number of developers required to run a business, not to increase it to create a tech empire.
From the last decade the cult of "api-development" has become synonymous with the following:
- Exposing data over web ( CRUD )
- Aggregating multiple other API calls or data sources ( Aggregation )
- Writing next to trivial logic ( apparently called business logic )
- All of these w/o any inherent capability of parallelism, asynchrony and timeouts and retries ( performance & resiliency )
In the end business and tech needs diverged : where it is like a self fulfilling circle. This throttles business development, this must be stopped.
The previous paragraph suggests ( many ) wrong ideas about the nature of the Tech. All of these have to be modelled such that these get done by few, while being used by many others who are not even developers.
Software development must be democratised beyond developers. If something was possible with 100 developers, then those MUST be solved with 10, if not less.
Goal of any development must start with profit, and end with increasing margin. This cannot happen with an empire full of developers doing development for development's sake. A crack team of 10 should be more than capable of handling a top team of 100 developers.
Excel Macros are such an example, so are SQL Engines.
And we should stop thinking about "long run". In the long run, the business will be more than dead.
What we are suggesting is the very anti thesis of what the "Industry Experts" promote. Perhaps they know better, perhaps they mean well. That is not to say that we are willing to sacrifice Quality - which means to us at-least : "write once, forget".
This does not mean never to look at the source code, it is about once it is written, there would be no need to look at it again and again because of maintenance. Can this be done in principle? Yes. Read on.
There will be parts of a flow which would almost never change, and there will be parts which will be frequently changing. First creates the core engine, second one creates the malleable business logic.
Code in some form is unavoidable, no matter how differently one represents it. Consider the goal of query. In 1960s it was COBOL and then SQL became a stable standard. With the advent of scale people almost stopped using SQL - instead started using SQL-ish dialect which are DSLs.
Inventing DSL is key. So DSL it is. Kubernetes/Ansible popularised YAML and hence Yaml form is chosen for representation.
- Business Logic outside Code Repos
- Configurability with Turing Completeness
- Polyglot Environment
- Cost Reduction in Development
One paragraph suffices for each of them.
Key issue of loss of dev time today begins with custom logic residing inside code. This must be separated. logic from business people should be kept separated from core logic to do anything. To achieve this, functional style helps while Declarative style does it better.
A business need at most must be computable, hence describable in a Turing Complete language. Given the separation of logic - one must build a configuration based engine. This prompts creation of a DSL that is configuration based, but Turing Complete as well.
Such a DSL is not new. The T-SQL completion of SQL as well as Gradle is Turing Complete. Incidentally both are declarative in nature:
apply plugin: 'java'
repositories {
jcenter()
}
dependencies {
compile 'org.slf4j:slf4j-api:1.7.12'
testCompile 'junit:junit:4.12'
}
jar {
manifest {
attributes 'Main-Class': 'com.example.main.Application'
}
}
It is inconceivable why business logic never became declarative in last 10 years which saw incredibly processor speed improvements.
The DSL in use must have scripting and language injection support.
Given any problem there are sub problems and within them one language may trump others in terms of applicability into the domain.
Hence the system must support a fairly large class of languages in which people can type in unavoidable code for business.
Such code must be kept few and far between. For all practical purpose, they should be indistinguishable from writing declarative expressions.
Henceforth, one does not need to find a specific set of developers : let's call them business developers with specified tech.
Previous section raises the bar for Engine Development and reduces the development bar for Business Development .
Imagine SQL. The people building SQL would be very less, 10 or less even. Folks who would be using SQL to get something done would be in millions.
One does not need to reach that much margin, but a ratio of 1 core Engineer per 10~50 business developer is good enough. Most of the cases they can be contractual employees.
Cost reduction immediately follows.
Any application consists of many user flows, all of them are workflow. Hence API's depict entry points for such flows.
Given a business requirement if one can create a configuration based flow out of it, then run it via a real time workflow engine - that would in effect be equivalent to an an API call.
The executable unit for any computation is a flow. Any time an user is doing anything [s]he is in an user flow. Flower runs such a flow.
These are the atomic units of a flow.
A node must have a
- body to be executed,
- a guard condition - failing which the node will not be executed, and
- a list of dependant nodes, after successful completion of them ONLY the node would be queued to be executed
A node is an abstraction of a pure function.
The structure of dependency between nodes creates a directed graph structure. Thus given a graph structure, multiple flows can be embedded in it, defining a set of PATH culminating in a single destination node in the graph.
This graph depicts the inherent dependency relation between each nodes.
Now we can define a flow, formally. It is a pair, the graph G, and the destination node N.
$$
F = <G,N>
$$
What does this man? This mean, if one were to execute a node, one has to successfully execute all immediate dependent nodes of the node. This now becomes a recursive algorithm.
Thus, there are many ordering of nodes possible to eventually execute the destination node
All of them are flows, and all of them are mathematically equivalent, while computationally might not be.
Hence, the flow
In other words, an actual executing flow is one of the topological sorts of the subgraph of
A flow is a composition of pure functions.
See :
- https://en.wikipedia.org/wiki/Pure_function
- https://en.wikipedia.org/wiki/Function_composition_(computer_science)
name: 'example graph'
params: # parameters for the graph flow
ai : int # int type
bi : int
nodes:
a:
body: ai # return value of node is return value of body
b:
body: bi # gets returned as body
c:
body: a + b # adds two return values
depends: # node c depends on a,b
- a
- b
d:
body: c ** 2
depends:
- c
This produce the following control flow
graph:
graph TD;
s(("start")) --> a;
s --> b;
a --> c["c := a + b"];
b --> c;
c --> d["d := c ** 2"];
d --> e(("end"));
c --> e;
Which is the precise flow from start
to the end
.
In this example a,b,c,d
are nodes.
The destination node is c
, and there are two flows that merges into it, a->c
and b->c
.
We can create a flow that ends in c
and that can be done by:
Map<String,Object> params = new HashMap<>();
params.put("a", 10 );
params.put("a", 20 );
String path = .... ; // path to the workflow
String node = "c" ; // node to reach
DependencyWorkFlow workFlow = MapDependencyWorkFlow.MANAGER.load(path);
Map<String,Object> result = MapDependencyWorkFlow.MANAGER.run(workFlow, node, params);
assertEquals(30, result.get("c"));
One possible execution order in a single processor
would be:
graph TD;
s(("start")) --> a;
a --> b;
b --> c["c := a + b"];
c --> d["d := c ** 2"];
d --> e(("end"));
c --> e;
Clearly b
could have come before a
. If we have two processors
, then we can do a bit better job of not blocking execution unless needed :
graph TD;
subgraph "Process #1"
a
c
d
end
subgraph "Process #2"
b
end
s(("start")) --> a;
s --> b;
a --> c["c := a + b"];
b --> c;
c --> d["d := c ** 2"];
d --> e(("end"));
c --> e;
It was quite possible to end in a,b,d
also.
Flower engine executes the flow based on the available processors. It has:
- Scalable parallel design, with built in node level
- guards
- timeouts
- retries
- error handling
- logging
- Relies on compositions of pure functions
- Scripting/expression support via JSR-223
Input : graph, node , parameter values.
- Get parameters and populate memory
- Put node into a queue/stack
- Get Node from queue/stack
- Given the node name
- check if all dependencies are satisfied
- if not, find each of the unsatisfied dependencies and repeat from step-2
- if yes, then execute body.
- If the current node is the input node and we did execute - HALT.
- Else - continue
- Memory : Mutable Map is being used. Each node creates a variable of the same name where their output is stored.
- Execution : Multithreaded - gets executed in a thread-pool, thus parallel processing is default.
- Timeout : Execution of individual nodes as well as the whole flow is time bound
For expression and programming support JSR-223 languages are used. The default is JavaScript - and it use ZoomBA to handle many internal operations.
Comes in 3 varieties.
# this is a basic node
i_am_a_node: # that is my name
timeout : 30 # times out after 30 msec
when : true # guard block
body : 42 # this is how you specify the body function
depends: # my dependencies
- another_node
- more_node
No of milli seconds for the node to execute before it times out. See more : https://en.wikipedia.org/wiki/Timeout_(computing)
Defines the Guard block. A node will only execute when the guard condition is true. Failing this will ignore the node, and will not process any of its dependants. This may result in flow failure.
See :
- https://en.wikipedia.org/wiki/Guard_(computer_science)
- https://en.wikipedia.org/wiki/Event_condition_action
Body of the computation. The result of this expression/script will be stored as the result for the node in the compute memory. See: https://en.wikipedia.org/wiki/Event_condition_action
Set of dependencies for the node, w/o whom the node execution would be meaningless. This is how the dependency graph gets created.
See:
This is used to make web calls, ( HTTP
).
get_all_comments:
https: # protocol
url: "#{base}/comments"
verb: get
For these nodes http
and https
protocols are supported.
url
defines the url for the web call while verb
defines the HTTP
verb to use.
See:
- https://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol
- https://docs.oracle.com/en/java/javase/12/docs/api/java.net.http/java/net/http/HttpClient.html
One can read more about the object mapper - it deserves it's own manual - you can find it here: ObjectMapper.md
Typical use case of the mapper is shown:
# Get all user ids who chatted a lot on comments
---
name: 'gather_chatty_users'
engine: zmb
params:
LARGE_WORDS : int
constants:
base : "jsonplaceholder.typicode.com"
nodes:
get_all_comments:
https:
url: "#{base}/comments"
verb: get
select_large_post_ids_by_mapper:
transform:
apply: "id_collector"
from: "@_/mapper.yaml"
depends:
- get_all_comments # this is the source for data for the transform
This shows that transform
block defines the ObjectMapper
. There are two fields:
This defines from
which external file we should load the mapper. We do not support inline mapping as of now.
On that external file, there will be many many mappings. apply
applies the map with the given name.
Now, into the file mapper.yaml
it is defined as follows:
id_collector:
_each: "#/."
_when: |
size( tokens( $.body , '\\w+' ) ) > _$.LARGE_WORDS
"*" : "#postId"
Detail structure of the mapping is scope of the object mapper proper. Important to note that the memory map of the current workflow is available as the variable _$
, and thus the mapper can ( by closure ) access the param
: LARGE_WORDS
.
Data source for the transform will be the first dependency node. In the example - get_all_comments
is the source of data for the transformation. Clearly one can have multiple dependencies, be careful to specify the data source node as the first one.
This node used to distribute a sub-graph ( a node ) for parallel execution.
distribute:
fork :
node: dummy_node
var : some_id
unique : false
depends:
- gen_fork
dummy_node:
body: >
some_id ** 2
As one can see the term fork
distinguishes a fork node.
See:
- https://en.wikipedia.org/wiki/Fork–exec
- https://en.wikipedia.org/wiki/MapReduce
- https://mathworld.wolfram.com/Fork.html
Only one dependency, the node it depends upon must produce a collection of sorts to distribute the work.
In this regard dependency is the data source
for the fork node, and the mapper
in the map-reduce
paradigm.
See:
That is the target reachable node, the exec
node for the fork. Engine will isolate the subgraph to reach into the node
and then will run the sub-workflow in a separate isolated environment for each item in the data source.
The context variable name - which will be used to store the item coming from data source. Imagine the following code :
dataSource = [1,2,3,4,5]
for ( x : dataSource ){ doSomething() }
The name of the variable x
is defined in var. As we can see the node dummy_node
use some_id
to get the data passed on by the data source.
If set to true
collects the result of this fork operation in a Set
. Default is false
so it stores it in a List
.
See : https://www.w3schools.com/sql/sql_distinct.asp
This node defines the quantifiers
- both existential
and universal
. From predicate logic we have these two :
This is existential quantification, there is some x in collection X such that the predicate P() is true. This with a chain can be used to implement control flow over multiple options. At the same time :
Can be used to select
a subset X from the Y where P(x) is true. This is the generic collection condition. These makes the flower engine Turing Complete and thus capable of universal computation w/o even a Turing complete expression engine.
This can be used as the following example shows:
name: 'test-any'
engine: zmb
params:
x : int
nodes:
switch_over_x:
any: # anything matches first will be used
- a
- b
- c
all_over_x:
all: # everything matching would be used
- a
- b
- c
a:
when: x < 10
body: "'a'"
b:
when: x < 20
body: "'b'"
c:
when: x < 30
body: "'c'"
It is either of any
or all
.
In case of any
the first node that matches the condition is triggered as a sub workflow. Careful, if the dependency of such a node is not satisfied, it would trigger a failure.
The result of the node execution is the result of the selected node.
all
selects all nodes which are evaluated via when
condition as true and then run them all in parallel - and stores the outcome in a map - for each node execution. That is returned as result.
when
of the node gets used as the condition. Unfortunately the condition will be evaluated at least 2 times - one for the condition check and another for node execution.
See more on Quantification
:
https://en.wikipedia.org/wiki/Universal_quantification
https://en.wikipedia.org/wiki/Existential_quantification
https://www.w3schools.com/sql/sql_select.asp
A retry is an attribute of any node
. This is how one can use retry:
name: 'simple-flow-with-node-retry'
engine: zmb
constants:
base : "some.random.server"
params:
fail_unto : int
nodes:
possible_fail:
when: "@_/pre.zm"
body: "@_/fail_unto.zm"
retry: # this node can be retried
strategy: counter # simple counter
max: 3 # 4th time would be a failure
interval: 10 # delaying for 10 ms
outcome:
body: possible_fail
depends:
- possible_fail
As one can see, there are two parameters along with the strategy type:
max
: defines the maximum no of retries before failureinterval
: interval between two successive tries
Caution : All retries run within the stipulated timeout, hence, if timeout exceeds retries will not work. Retries will only work iff the timeout is not over.
There are 4 strategies available:
This is not even a strategy, it suggests no retry.
Essentially keeps a counter of failures. Successive calls are separated by constant interval spacing.
retry: # counter retry
strategy: counter
max : 3
interval : 10
Like a counter, has a counter of failures. Successive calls are separated by random interval spacing - not exceeding :
And not lower than:
The actual gap between successive calls will be distributed uniformly between these two numbers.
retry: # random retry
strategy: random
max : 3
interval : 10
Like a counter, has a counter of failures. Successive calls are separated by exponentially larger interval spacing :
Where interval
that is passed,
retry: # Exponential Back-Off retry
strategy: exp
max : 3
interval : 10
Given JSR-223 is included, all JVM scripting languages are default supported, default provided are javascript
and ZoomBA zmb
.
One can use various script engine in various nodes although that would be a terrible experience.
The engine support is described as follows:
engine: zmb
Default support are zmb
for ZoomBA
, js
, javascript
for JavaScript.
It is recommended to use ZoomBA, because it was developed for business development ( see wiki in references ).
See:
- https://en.wikipedia.org/wiki/Scripting_for_the_Java_Platform
- https://stackoverflow.com/questions/11838369/where-can-i-find-a-list-of-available-jsr-223-scripting-languages
One can use inline script, or can reference a script file.
For example this is how one can reference the file large_enough.zm
stored in the same folder as that of the yaml file.
body: "@_/large_enough.zm"
Notice the trick @_/
before the file name. This gets replaced by the folder
of where the yaml file is situated.
This is very useful to copy the whole workflow source code as is.
A typical Yaml Structure for a Graph is shown here:
name: 'gather_chatty_users'
engine: zmb
timeout : 100
params:
LARGE_WORDS : int
constants:
base : "jsonplaceholder.typicode.com"
nodes:
get_all_comments:
https:
url: "#{base}/comments"
verb: get
select_large_post_ids:
body: "@_/large_enough.zm"
depends:
- get_all_comments
As we can see name
is the name of the workflow.
engine
is the default script engine to be used, if no one else specifies anything else. timeout
has already been discussed.
constants
is a special map which stores all things constant that can be used to dereference later.
They are available to every node and script as values.
This defines the parameters for the flow. Engine checks if each parameter is filled in before the engine runs a flow. Same is applicable for each node.
See the tests to understand more.
- Rule Engines ( https://en.wikipedia.org/wiki/Business_rules_engine )
- Workflow Engines ( https://en.wikipedia.org/wiki/Workflow_engine )
- Low Code ( https://en.wikipedia.org/wiki/Low-code_development_platform )
- DSL ( https://en.wikipedia.org/wiki/Domain-specific_language )
- ZoomBA ( https://gitlab.com/non.est.sacra/zoomba/ )
Here is my own presentation on DSL :
https://www.slideshare.net/nogamondal/formal-methods-in-qa-automation-using-dsl
Unfortunately, the tech clan is dead against using any of them which does not allow them to write code, and clutter the whole business logic out of it. Hence, while some of these sounds like great ideas, they are very less samples of it in production.
This work is under Apache 2.0. Here is from where one get a copy: https://www.apache.org/licenses/LICENSE-2.0.txt