Data format design
The current plan is to have a small “phone sync” cache that is used for most bi-directional communication between the phone and the server. This will allow us to support offline operation, customize background operation, and reduce user perceived latency.
We will then transfer this data to the long term storage/analysis cache. The data formats in the two caches need not be the same, and in fact, we expect them to be significantly different. In particular, the data in the cache is optimized for flexibility, while the long-term storage is optimized for ease of analysis.
Here, we decide the data format for the user cache (flow 1 above). The flow is bi-directional, and so can again be split into phone -> server and server -> phone components. For now, we will endeavor to separate the data in the two components so that there is no overlap. This will help us in resolving conflicts because there will be none. We will revisit and extend this design if and when we find a data element that needs to be modified from both the server and the phone sides.
There are several existing solutions for syncing data between the phone and the server. However, most of them are paid solutions, except for Apache Couchbase. Apache Couchbase appears to require access to non-standard ports on the server in order for the sync to complete successfully, which may be an issue for IT departments or IRBs, specially while dealing with sensitive user data. http://blog.couchbase.com/iptables-firewall-settings-couchbase-db-and-couchbase-mobile-syncgateway
In order to provide a balance between ease of use and flexibility, we plan to include our own built-in implementation which will work out of the box. However, the usercache will be accessed via a standard abstraction layer that can support the use of other providers in the future.