sync 02

worker state and controller

In this case, the state backend is run in a dedicated worker but accessed from a controller of the drivers.

      {worker}                    {worker}                            {worker}
+---------+-------+             +----------+                     +-------+----------+
|         |       |   (drives)  |          |      (drives)       |       |          |
|  driver | state |<------------|  engine  +-------------------->| state |  driver  |
|         | contr |             |          |                     | contr |          |
|         |       |             |          |                     |       |          |
+---------+-------+             +----------+                     +-------+----------+
              |                                                      |
              |                                                      |
  {worker}    |                                                      |
+----------+  |                                                      |
|          |  |                   (drives)                           |
|  state   +<-+------------------------------------------------------+
|  driver  |
|          |
+----------+

Use case

• In the engine:

left.search(searchConditions) # Async
right.search(searchConditions) # Async

• In the controller (so in the driver worker):

self.state.search(searchConditions) # Async
messages = self.driver.search_sync(searchConditions) # Sync
stateMessages = self.state.getSearchResult_sync() # Sync

# 2-way merge algo between stateMessages and messages.

# Returns only the messages that changed since last sync.
return mergedMessages # To the engine.

• In the engine:

leftMessages = left.getSearchResult_sync() # Sync
rightMessages = right.getSearchResult_sync() # Sync

# 2-way merge between both sides. Would only be usefull to discard identical
# changes made on both sides.

# Sample: apply the changes on the left side.
left.update(rightMessages) # Async

• In the controller (so in the driver worker):

for message in mergedMessages:
  success = self.driver.updateMessage(message)
  if success is True:
    stateSuccess = self.state.updateMessage(message) # Async
    if stateSuccess is not True:
      # Bad. We might take action here like rollback the changes in the driver
      # or tag the email in the state # to avoid further unaligned state at "low
      # performance penalty".

Pros

• The sync engine and controllers hold simple 2-way merge algos.
• Fits well in a fully async environment.
• Recover on errors at low performance penalty.
• Globally performance friendly.

Cons

• The state backend will be a bottleneck for concurrent read access from controllers. Using two different backends would help performance.

Possible improvements

The state will get concurrent reads at discover time. However, if we know which state database to load early, we would load it in RAM before we get read requests.