sync 08

two log state workers and controllers

Modified version of 07.

In this case, the log state backends are run in a dedicated worker and accessed from a controller of the drivers. They store logs on disk.

       {worker}                     {worker}                          {worker}
 +---------+-------+               +----------+                   +-------+----------+
 |         |       |               |          |                   |       |          |
 |  left   | state |   (drives)    |          |    (drives)       | state |  right   |
 |  driver | contr |<--------------|  engine  +------------------>| contr |  driver  |
 |         |       |               |          |                   |       |          |
 +---------+-------+               +----------+                   +-------+----------+
             |   |                                                  |   |
             |   |                                                  |   |
             |   |                                                  |   |
     (write) |   |        (read)                 (read)             |   | (write)
             |   +----------------------+---------------------------+   |
             |                          |                               |
             v                          |                               v
       +----------+                     v                        +----------+
       |          |                +----------+                  |          |
       |  right   |   (read/play)  |          |  (read/play)     |  left    |
       |   log    |<---------------|  state   |----------------->|   log    |
       |  state   |                |  driver  |                  |  state   |
       |          |                |          |                  |          |
       +----------+                +----------+                  +----------+
         {worker}                    {worker}                      {worker}

Use case

• In the engine:

left.search(searchConditions) # Async
right.search(searchConditions) # Async

• In the controller (so in the driver worker):

self.state.search(searchConditions) # Async
messages = self.driver.search_sync(searchConditions) # Sync
stateMessages = self.state.getSearchResult_sync() # Sync

# 2-way merge algo between stateMessages and messages.

# Returns only the messages that changed since last sync.
return mergedMessages # To the engine.

• In the engine:

leftMessages = left.getSearchResult_sync() # Sync
rightMessages = right.getSearchResult_sync() # Sync

# 2-way merge between both sides. Would only be usefull to discard identical
# changes made on both sides.

# Sample: apply the changes on the left side.
left.update(rightMessages) # Async

• In the controller (so in the driver worker):

for message in mergedMessages:
  success = self.driver.updateMessage(message)
  if success is True:
    stateSuccess = self.state.updateMessage(message)
    if stateSuccess is not True:
      # Bad. We might take action here like possibly tag the email on disk
      # to avoid further unaligned state at "low performance penalty".

Pros

• The sync engine and controllers hold simple 2-way merge algos.
• Fits well in a fully async environment.
• Recover on errors at low performance penalty.
• Globally performance friendly.

Cons

• The full merging code algo is dispatched. Is it really a con?
• The engine needs a way to update the chain of controllers.
• Log state format should be well thought (use pickle)?

Compared to 07

Pros

• Only one storage to record the previous state.
• No possible mis-alignment between right and left state storages.
• Can support legacy offlineimap local cache.

Cons

• Logs must be played at either pre-sync or post-sync time.
• Requires one more worker.
• Requires more channels.