Tag Archives: Sync

Sync: Operational Transformation vs. Conflict-Free Replication Data Types (CRDTs)

I need a solution for data sync/replication of offline data that doesn’t require my team to read whitepapers and understand theoretical mathematics.

There is an argument going on right now as to whether Operational Transformations (OT) or Conflict-Free Replication Data Types  (CRDTs) are the way to go here.  Both technologies are intended to solve the thorny problem of handling (or removing the potential) of conflicts when multiple parties are working on the same data without direct awareness of the efforts of another party or parties (perhaps because of temporal or location differences).

Maturity

I really like the idea of CRDTs, but there isn’t really a practical (or at least popular) implementation of full document CRDTs (think JSON) that I know of right now.  There also seems to be the (old) problem where the CRDTs are replicating correctly, but we are asking them to do the wrong thing… To be a little more clear, we are having difficulty getting the intent of the users expressed in the data structure that prevents conflicts.  We can get eventual consistency between the two bodies of data, but what would the two (or more) parties have created if they did it together side-by-side?

This is a problem that has been explored a little more in the world of Operational Transformation so the solutions (that I am aware of) are a little more mature.

Sync via peer to peer?

The primary downside of OT (that I can see) is that there really needs to be a single source of truth (think server) with Operational Transformations whereas CRDTs allow full mesh or peer-to-peer (P2P) sync.

Because P2P communication is almost as difficult right now as sync itself, it may just be practical to work with OT.

Some libraries to look at

I have been playing a bit with sharedb.  This seems to be the best OT work going on in JavaScript right now.  That said, there isn’t a huge community around the library and the owner’s (though amazing and brilliant people with other real jobs to keep down) do not appear to be super responsive to pull requests and issues.

If you are looking at doing the P2P thing, it seems like Scuttlebutt is a protocol/replication technology that is getting a bit of traction.  I believe it is inherently duplex though… so YMMV.  Here is a JavaScript implementation that might interest you.