39151090.consensus-and-hard-forkery.html

<p>Here goes the theory:</p><p>Consensus is the agreement of a large set of actors over a fact holding true. It can be defined as <a href="https://en.wikipedia.org/wiki/Common_knowledge">common knowledge</a> of common agreement. Common knowledge exists when everyone knows that X happened, and everyone knows that everyone knows that X happened, and everyone knows that ..ad infintum. Common agreement exists when everyone agrees to accept X as a <em>correct fact</em>. Blockchains enable the steady creation of facts X, the establishment of common agreement over these facts, and the establishment of common knowledge over these agreements.</p><p>A blockchain is a glorified database. Blockchain consensus involves large groups of actors coming together to agree what goes into the database, and what updates happened on it. Blockchain consensus is established by two means:</p><ul><li><p>economic consensus - 1 dollar 1 vote</p></li><li><p>social consensus - 1 social entity ?? 1 vote</p></li></ul><p>Defining consensus requires defining actors in the system. Since we do not natively have consensus over who distinct persons (human beings) are at a global scale, we cannot establish consensus among all humans directly. Part of the appeal of blockchains is that users do not need to reveal their identities globally in order to operate within the system.</p><h3>Economic consensus (aka consensus among block producers)</h3><p>Therefore, for day to day operation, consensus is established between blobs of capital rather than humans. Each dollar or bunch of dollars is defined as a unique actor, and consensus is said to exist when a majority of dollars agree to what happened in the system. This applies to both proof of work and proof of stake.</p><p>(There are important similarities and differences in terms of risk tolerance of those dollars, memetics, profitability, capex versus opex, but we will not get into that in this article)</p><p>Economic consensus is a simple yet elegant way of arriving at consensus essential for a fast-moving system such a blockchain where agreements between actors need to be made on a second-to-second basis. A parliament for instance, would not need such efficiency.</p><h3>Iteration of knowledge</h3><p>Typically common knowledge in its strictest sense is not arrived at in practice, but nth order knowledge is (knowledge of knowledge of knowledge .. n times).</p><p>Consider a proof of work system.</p><p>Miner A mines a block ‘a’ on top of the longest chain (which for now, let’s simplistically assume has common acceptance).</p><ul><li><p>Facts created: a</p></li><li><p>Agreements created: A approves a</p></li><li><p>Knowledge of agreements created:</p></li></ul><p>Miner B mines a block ‘b’ on top of ‘a’</p><ul><li><p>Facts created: b</p></li><li><p>Agreements created: B approves b and a</p></li><li><p>Knowledge of agreements created: B knows that A approves a</p></li></ul><p>Miner C mines ‘c’ on top of ‘b’</p><ul><li><p>Facts created: c</p></li><li><p>Agreements created: C approves c, b and a</p></li><li><p>Knowledge of agreements created: 1st order: C knows that A approves a. C knows that B approves b and a. 2nd order: C knows that B knows A approves a.</p></li></ul><p>Miner D mines ‘d’ on top of ‘c’</p><ul><li><p>Facts created: d</p></li><li><p>Agreements created: D approves d, c, b and a</p></li><li><p>Knowledge of agreements created: 1st order: D knows that C approved c, b and a. D knows that B approves b and a. D knows that A approves a. 2nd order: D knows that C knows that A approves a. D knows that C knows that B approves b and a. 3rd order: D knows that C knows that B knows that A approves a.</p></li></ul><p>After the creation of some number of blocks n, the actor creating the nth block will gain knowledge of common agreement. For instance if n is 12, the 12th actor L knows that the 11th actor approves the first 11 blocks (a through l) and the 10th actor approves the first 10 (a through j) and so on. Therefore the 12th actor L knows that all 11 actors before them have approved the first block a.</p><p>In proof of work, this actor makes a reasonable assumption that 11 actors mining 11 blocks covers <em>almost everybody </em>and that it is economically irrational for any other actor to not want to also agree to these blocks (probabilistic finality). Actor L cannot establish certainty over everyone agreeing to even block a. In proof of stake, since it is pre-defined which actors act in which sequence, actor L can arrive at the agreement that everyone in the system agrees, assuming the pre-defined order in fact is common knowledge of common agreement (which it never is, it only is n-th order for even larger n).</p><p>This still doesn’t arrive at common knowledge over common agreement (i.e. consensus). Actor A for instance, does not yet know that all others actors have approved their block a, and actors A does not know that actor B knows that all other actors have approved the block a. Arriving at common knowledge over common agreement takes many more iterations, especially when the system can also add or delete actors. However, arriving at common agreements provides a natural shelling point to also attain common knowledge of those common agreements.</p><h3>When economic consensus fails?</h3><p>The above system makes a bunch of assumptions.</p><ul><li><p>It assumes that all actors in the above system are both able and willing to communicate with each other. (No network partition)</p></li><li><p>It assumes that all actors are willing to come to consensus over facts. (No chain halt)</p></li><li><p>It assumes that all actors are willing to come to consensus over any fact a produced by any actor A without discrimination. (No censorship)</p></li><li><p>It assumes that all actors are willing to come to consensus on only those facts that follow some rules for the database defined by the social consensus layer (No invalid state transition)</p></li><li><p>It assumes that all actors aren’t willing to undo facts they’ve already come to consensus on in favour of new facts. (No chain rewrite.) This is typically split into two categories - short reorgs and deep reorgs.</p></li></ul><p>There is also a meta-assumption of block producers (economic layer) being willing and able to communicate with nodes (social layer), that we will get to later.</p><p>One assumes that the economic consensus layer will never want to break down their consensus because of their own economic rationality, however there are various reasons why economic rationality may be insufficient. Maybe a government orders a block producer to censor another block producer using threat of violence, maybe the block producers realise that the social layer is too weak to keep them in check, maybe some block producers are not motivated purely by economic incentives, maybe block producers are able to gain net short market exposure to the chain they run on making it economically rational to behave nefariously.</p><h3>Social consensus (aka consensus among nodes)</h3><p>The social consensus layer exists to check against the failure modes of economic consensus. More specifically it is used to:</p><ul><li><p>establish consensus on what the rules are for the blockchain are in the first place. This is non-trivial. However, once you do, dealing with invalid state transitions is trivial.</p></li><li><p>establish consensus on what the last checkpoint state of the blockchain is. This is non-trivial. However, once you do, dealing with deep reorgs is trivial. (Some bitcoin folks reject the notion of any checkpoint after genesis, this is also a valid set of rules however it also permits continuous rewrites of the chain upto genesis)</p></li><li><p>re-establish consensus in case of any of the other failure modes of economic consensus: network partitition, chain halt, censorship of blocks, short reorgs. This is hard. A lot of these failure modes and their recovery mechanisms are understudied.</p></li></ul><p>All these forms of recovery are called hard forks.</p><p>Social consensus is the layer upon which economic consensus is created in the first place. For instance, if you are running a node for a blockchain, you likely queried a few more people for documentation on it to know what its rules are, you queried a few people to download the source code for the node and you queried a few more people to know that the blockchain you are running is in fact a <em>popular</em> one that many people you know are also using. (Assuming ofcourse, you are running a node, which a lot of people aren’t)</p><p>Social consensus too operates as both knowledge, agreements and knowledge of agreements. However who is a distinct actor when it comes to social consensus is more vague.</p><h3>Local versus global</h3><p>Let’s assume one person one vote holds true for social consensus. It is indeed possible to establish knowledge of persons at a local scale - each person knows atleast a few hundred other human beings personally - and trusts a subset of them to various degrees. No social interaction could function without this minimum knowledge. It is also possible to establish various degrees of trust in a small finite set of online entities you haven’t physically met. Knowledge of large number of persons at a global scale is more tricky, and exactly one of the problems blockchains try to avoid. Social media websites can provide you knowledge of persons at a global scale, so can a national database. Using either requires trust of a different form and degree than that required by a blockchain. There has currently been insufficient work on how local communication can be used to establish consensus at a global scale. p2p messaging networks that don’t require us to trust twitter would be very helpful.</p><h3>When are opinions expressed on the social layer?</h3><p>For checkpoints and syncing, opinions are expressed as and when new nodes need checkpointed states to sync from. Ideally opinions should be publicly expressed the moment checkpoints are created, and there should be ways to access numerous users’ checkpoint over the internet and choose whose to trust in order to sync.</p><p>In all other scenarios, social consensus is expressed in the form of a hard fork. Be it changing the rules of the blockchain or defending against a failure mode of the economic layer (block producers). Detecting of scenarios that require forks is non-trivial, so is establishing consensus over which forks are popular.</p><p>I am personally of the strong opinion that there should be more analysis on the detection and recovery modes for these. I also feel there should exist a “constitution” - essentially a set of meta rules and agreements that people have already come to consensus on even before a fork actually occurs, in order to agree on forks faster. Coming to consensus on forks after both forks are live is harder and comes with pressure to resolve as quickly as possible. It also involves a lot of communication - both socially and via markets - that can be chaotic and require time to be expressed. Reducing the time required for this expression is critical to ensure no bad-faith actor is able to exit the system with illicit profits. Reducing the time required to resolve is also necessary to minimise the harm that everyone faces due to on-chain activity (social or economic) breaking down for a period of time.</p><h3>Is social consensus really one person one vote?</h3><p>Knowledge and opinions are typically expressed on a person-to-person basis. However the expression of these knowledge and agreements on the blockchain are not defined by one person one vote, because the blockchain has no direct notion of a person. Persons can use their opinions to impact token prices, transaction activity and on-chain activity beyond simple transactions (such as in DeFi). This is the extent to which they can express their opinion on the blockchain itself. Besides this, they can communicate within the social layer.</p><p><code>The social layer is bound by economic rationality just as the economic layer is.</code></p><p>They may also have ideological preferences that motivate them to various degrees.</p><p>However this economic rationality assumption means users will more often than not, act <em>against</em> what they ideologically believe in if there’s a clash between the two motivations. The rest of this article tries to understand what the economically rational actions are in different kinds of blockchains and assets.</p><h3>Hard forks for SoV assets</h3><p>For blockchains whose asset prices are determined primarily due to their memetic store-of-value nature, clear network effects exist. This has been seen various times in the past, when BCH forked off BTC or ETC forked off ETH. Forks have typically had winners and losers as expressed by market cap. Winners have on average made more wealth in dollar terms than losers. This means everyone has an immediate economic incentive to hold more of the asset of the winning chain, <em>irrespective of what the winning chain is.</em></p><p>This violates the naive assumption that hard forks are non-violent in the sense that minorities can peacefully fork off from majorities without suffering repercussions. It also means that developers include core devs cannot ignore prices and large token holders. They are not only “not an adverse actor” but the very actors who help define social consensus in the first place. Atleast as long as chains are valued purely on SoV-based memetics.</p><h3>Markets and SoV forks</h3><p>Since everybody has an immediate incentive to remain on the winning chain, this also means that if someone in fact has assets on both forks, their economically rational decision is to continue holding their assets on both forks, until they know who the winner will be. Therefore it is the most ideologically motivated and risk-loving users who will first express their opinions on the market by selling assets of the fork they dislike and buying assets of the fork they like. Wealthy users are typically more risk-loving, since their living conditions are not as determined by their wealth - this is why log utility is sometimes assumed. It is very possible that due to pareto distribution of wealth, the fork that is initially valued more by a few wealthy actors becomes the winning fork simply because the most profitable thing for everyone else to do is to continue following the price momentum generated by these actors. Even if it is what they ideologically dislike, as long as they are more economically rational than ideological.</p><p>Once someone has sold their assets on one fork it is not guaranteed they will be able to buy it back at a similar price. Therefore, once something has expressed their market opinion on a fork, it is economically rational for them to also try and push others to use this fork via the social layer. Again, this could even be a fork they ideologically dislike or change their mind on. Changing your mind later is costly.</p><p>This discussion assumes the existence of neutral market infrastructure with sufficient liquidity that does not attempt to express its own opinion. Often this won’t be the case, as exchanges too are profit-motivated actors that may benefit more from certain chains winning over others. They can censor or delay deposits, trades and withdrawals, they can show incorrect trade information such as depth and price, they can straight up steal your assets from you and trade them however they like, barring legal restrictions. Decentralised trading infrastructure will likely not be as active since the existence of DeFi infrastructure is fork-dependent, whereas you want fork-independent infrastructure to express market opinion in favour of a fork.</p><h3>Forks and DeFi chains</h3><p>With the existence of ethereum we now have blockchains that are valuable for more than just currencies that are valued as stores of value. Whether ETH the asset gains any value from DeFi yet (as opposed to SoV and narratives) is an open question, however there is definitely the possibility that DeFi infrastructure will play a significant role in defining winning hard forks.</p><p>DeFi inherently involves honest majority assumptions of small groups of actors that blockchains do not make. Blockchains assume the existence of a social layer that can recover if the “1 dollar 1 vote” layer fails, DeFi assumes no such thing. The main honest majority assumptions DeFi makes are: Chainlink serving price feeds and doing so correctly (honest majority of ~21 nodes that own LINK), MakerDAO serving price feeds and doing so correctly (honest majority of some oracle feed operators governed by MKR holders), Circle backing and redeeming USDC (honest majority of one actor) and Tether backing and redeeming USDT (honest majority of one).</p><p>If critical DeFi infrastructure signals that it will only run on forks that follow certain rules, there is natural pressure for those forks to win, assuming users really want DeFi infrastructure on the winning fork. Oracles and stablecoin backers are sources of trust, and creating new sources of trust to act as replacements on the other fork is non-trivial. This also applies to middleware infrastructure such as indexers, frontends and subgraphs, however these may still be somewhat replaceable.</p><p>This in turn could mean that the hard fork process of ethereum is currently captured by a very small number of entities - perhaps to the extent that nodes are unable to fork against the wishes of these entities, or reject the forks of these entities (without risking serious economic loss).</p><p><code>If this is true, it would mean our entire theory of blockchains is voided.</code></p><p>We could attempt creating new mechanisms for social coordination over how decentralised oracles run and signal in favour of chain forks, however I am skeptical of this effort.</p><p>If we accept that the hard fork process has been captured by these entities, some serious implications include: we will no longer really need full nodes, we can increase block size and run light clients, we’ve been unnecessarily paying billions of dollars in gas fees, we don’t need validator diversity, if these entities wish to censor a minority of users they can, if these entities wish to rob a minority of users of their assets through a fork, they can.</p><p>I’m not sure I believe in this yet, but it is a serious thought to consider, especially as use cases of ethereum become more advanced.</p><h3>Hard forks and cashflow chains</h3><p>If assets of chains are valued not as stores of value but as cashflow generating assets, then the economics of a hard fork change. Users will now want to hold the assets of the chain which is expected to have more transaction activity and therefore more profits. To be clear, I don’t think this is true for ethereum yet, firstly due to DeFi being highly reflexive on ETH and secondly because DeFi has more serious trust sources as mentioned above that have greater power.</p><p>However it is possible in the future that more blockchains are valued akin to businesses with cashflows, in that whatever actions are more profitable to the business as a whole will be more valued. This incentivises against governance minimisation and credibly neutrality, since opinionated and actively governed businesses are usually worth more. (Or atleast they’ve been worth more in the traditional world so far.) It also means forking against this governance will be hard or impossible.</p><h3>Hard forks and core developers</h3><p>Core developers ofcourse play a significant roles in signalling for and against forks. As long as development work on the base layer blockchain is incomplete, all users are dependent on core devs to complete this work (assuming they really want it). If the code base is complex and maintenance is sufficiently hard that only few teams in the world the world can do it, this is a major force for centralisation, and ensures the core devs essentially dictate all forks and the future of the chain. If however, it is possible to hire new client teams (or convince existing teams) to maintain forks, then the limiting resource to create a fork is not core developers but all the other factors mentioned in this article.</p><h3>Conclusion</h3><p>Consensus is complex. Hard forks are complex and violent, unlike theory suggests. Social layer as a defence against block producers as understudied - both from a technical and an economic perspective. Even if nodes are technically capable of arriving at social consensus and forking (which they may or may not be), they may act in favour of their economic interest rather than their ideological interest. Core developers are a centralising force that help guide social consensus.</p>