Pulse quorum L2 event confirmations

This implements a confirmation mechanism into the pulse quorum to
enhance the security of pulse blocks without introducing desyncing of
the Oxen chain to achieve consensus.

The general design implemented here is explained in the
`docs/sent-l2-confirmation/sent-confirmations.md` document (and so see
that for details rather than this commit message).

Other smaller changes included here that intersect (but aren't directly
part of the confirmation mechanism):

- Combine the virtually identical `tx_extra_ethereum_new_service_node`
  and `NewServiceNodeTx` instead a single 'eth::event::NewServiceNode`,
  and likewise for removal requests and removals.
- Update the naming of L2 events to match the more informative names in
  oxen-io/eth-sn-contracts#58
- Increase L2 refresh times to once per minute.
- Remove useless pass-through wrapper functions for txpool operations
  from blockchain.cpp, and just have tx_pool call directly into the db
  via blockchain's `db()` reference.
- Remove l2 monotonic height checks from consensus consideration; they
  are dangerous in that a malicious quorum could stall the chain by
  putting an excessive value that honest nodes couldn't follow.
- Various related code cleanup in passing.
- renamed some functions such as `get_locked_key_image_unlock_height`
  that have become rather misnamed by the eth transition.
- Simplify tx_pool sorting which was, for completely inexplicable
  reasons, a pair<tuple<A,B,C>, D> rather than a tuple<A,B,C,D>.
- Refomatting

docs/sent-l2-confirmation/sent-confirmations.md

@@ -118,6 +118,18 @@ unlock, etc.) takes effect.
   where N is the quorum round: so flags in the first backup round (the second overall quorum round
   for a block) contribute 0.5 points; flags in the 4th backup round would contribute 0.2 points, and
   99th backup quorum would contribute 0.01 points.
+- Mined blocks contribute a full-weight vote to the process.  Oxen mined blocks are only accepted on
+  the Oxen chain in one of two cases of extreme conditions: a complete network pulse failure to
+  create a block for more than 4 hours; or a drop in the number of active service nodes below the
+  threshold needed to create a pulse quorum (12).  The former case has happened just one (at the
+  initial pulse hard fork, due to a bug in the pulse activation code), and the latter has never
+  happened on mainnet (though happens from time to time on the ~20 node private testnet/devnet).
+
+  In either case, however, the mined block fallback votes are likely needed to fix the service node
+  state (e.g. to get new L2 node registrations or removals applied to the Oxen chain), and so such
+  extreme fallback cases need to have the potential to push through votes.  It is worth pointing out
+  that a mined block is not accepted and cannot be triggered by the network outside of the two
+  extremes described above.
 - A state change is finalized once the dominant + or - score is at least 5 points larger than the
   lesser score, *and* at least double the lesser score; or when 30 blocks (1 hour, typically) have
   passed since its inclusion without otherwise finalizing it (and in such a case, it is resolved as
@@ -146,6 +158,12 @@ contentious, and thus suspect; secondly service nodes typically do not indefinit
 tracking data beyond an hour or so and thus are unlikely to be able to confirm a state change once
 hours have passed since its inclusion.
 
+A failed inclusion is also not the end of the world and mainly introduces delays, not stake losses:
+a failed registration would be removable from the smart contract after a waiting period (and could
+then be resubmitted), and a failed unlock or failed removal will be noticed by Oxen nodes as a
+service node that no longer exists in the smart contract and needs to be removed, which will be
+noticed and re-submitted as removal for confirmation, even if the original removal gets denied.
+
 #### Service Node adversary example
 
 Supposing an adversary with control of an enormous one-third of the network's service nodes seeks to
@@ -253,10 +271,9 @@ contracts for more info).
 Rewards, however, are computed entirely on the OXEN side, and thus being able to advance the chain
 requires an exact consensus of what the reward is at any given time.
 
-Oxen nodes thus record the *current* L2 reward rate (queried from the contract) in each block, and
+Oxen nodes thus record the recent L2 reward rate (queried from the contract) in each block, and
 verifying this value is part of the duties of pulse quorum validators.  For all the same reasons
-discussed above, however, this means that it could be a target of abuse by a malicious pulse
-quorum.
+discussed above, however, this means that it could be a target of abuse by a malicious pulse quorum.
 
 For example, just after launch, the per-block SENT reward (distributed across all service nodes) to
 be a little bit less than 23 SENT per 2-minutes (i.e. per Oxen block).  An adversary controlling a
@@ -317,3 +334,24 @@ pool becomes larger, increases of the same size can be adjusted to more quickly.
 the increase in 13 days.  A severely withdrawn 1M pool (which would take nearly 25 years of
 withdrawals with no replenishment with the SENT launch parameters) would take around 7 months to
 fully respond to the sudden 21x size increase of the pool with these caps.
+
+## Reward rate update frequency
+
+Because the reward rate is a continuously decreasing variable, in order to properly handle pulse
+validation we would need to know the value computed for any possible recent blocks (or else pause
+pulse validation while all validators fetch the value for the leader's l2_height choice).
+
+Unlike contract events—which we can query all-at-once—*each* L2 height we want to know the reward
+for requires a separate RPC call, and thus becomes expensive in terms of the number of queries
+permitted under various RPC provider plans.
+
+To address this, we only fetch updated reward values once every 2400 blocks (approximately once
+every 10 minutes), and our canonical "proper" reward for a height is the reward rate at previous
+such fetched height before the l2 height indicated in the block itself.  For example, when
+constructing a hypothetical Oxen block with `l2_height = 23456789`, the "true" reward rate used to
+compute the reward value that gets recorded into a block would be the reward rate for L2 height
+23455200 (i.e. the last height before 23456789 divisible by 2400).
+
+Note that this does *not* mean that the block reward can only change every 10 minutes: in
+particular, because we still include an l2_reward value in every block, the block reward can still
+move slightly in accordance with the mitigations against large changes described above.

src/blockchain_db/lmdb/db_lmdb.cpp

@@ -52,6 +52,7 @@
 #include "cryptonote_core/uptime_proof.h"
 #include "epee/string_tools.h"
 #include "logging/oxen_logger.h"
+#include "oxen/log/level.hpp"
 #include "ringct/rctOps.h"
 
 using namespace crypto;

src/cryptonote_basic/cryptonote_basic.h

@@ -33,6 +33,7 @@
 #include <fmt/format.h>
 
 #include <atomic>
+#include <variant>
 #include <vector>
 
 #include "common/exception.h"
@@ -48,6 +49,7 @@
 #include "serialization/crypto.h"
 #include "serialization/variant.h"
 #include "serialization/vector.h"
+#include "serialization/vector_bool.h"
 #include "txtypes.h"
 
 namespace service_nodes {
@@ -399,7 +401,7 @@ struct block_header {
 
     bool has_pulse_header() const { return major_version >= feature::PULSE && !pulse.empty(); }
 
-    // HF19+:
+    // HF19, HF21+:
     // The height of this block.  This is not used by current Oxen code for the height before HF21+,
     // but must be present and correct in HF19 for compatibility with Oxen 10.x nodes which *did*
     // (sometimes) use it.  Before HF21 the height is always present in the miner_tx's txin_gen,
@@ -458,6 +460,12 @@ struct block_header {
     // take time to be fully reflected here, during which this value increases slowly towards the
     // appropriate value, but a compromised quorum is unable to noticeably affect the reward rate.
     uint64_t l2_reward = 0;
+
+    // vector of L2 state changes for L2 state change transactions that are still unconfirmed in
+    // this block.  true = vote for confirmation, false = vote for rejection.  Votes are sorted in
+    // blockchain order of unconfirmed transactions (i.e. by the monotonic internal transaction
+    // index).
+    std::vector<bool> l2_votes;
 };
 
 struct block : public block_header {
@@ -483,6 +491,9 @@ struct block : public block_header {
     std::optional<transaction> miner_tx;
     crypto::public_key oxen10_pulse_producer;
     std::vector<crypto::hash> tx_hashes;
+    // As of HF21 this value indicates how many leading values of `tx_hashes` are eth state change
+    // transactions; these *must* be eth state change txes, and any remainder must *not* be.
+    uint32_t tx_eth_count;
 
     // hash cache
     mutable crypto::hash hash;
@@ -515,6 +526,7 @@ void serialize_value(Archive& ar, block_header& b) {
     if (b.major_version >= feature::ETH_BLS) {
         field_varint(ar, "height", b._height);
         field_varint(ar, "l2_height", b.l2_height);
+        field(ar, "l2_votes", b.l2_votes);
     }
 }
 
@@ -544,6 +556,9 @@ void serialize_value(Archive& ar, block& b) {
     field(ar, "tx_hashes", b.tx_hashes);
     if (b.tx_hashes.size() > MAX_TX_PER_BLOCK)
         throw oxen::traced<std::invalid_argument>{"too many txs in block"};
+    if (b.major_version >= hf::hf21_eth)
+        field(ar, "tx_eth_count", b.tx_eth_count);
+
     if (b.major_version >= hf::hf16_pulse)
         field(ar, "signatures", b.signatures);
     if (b.major_version == hf::hf19_reward_batching) {

src/cryptonote_basic/cryptonote_format_utils.cpp

@@ -50,9 +50,11 @@
 #include "cryptonote_core/service_node_voting.h"
 #include "epee/string_tools.h"
 #include "epee/wipeable_string.h"
+#include "l2_tracker/events.h"
 #include "ringct/rctSigs.h"
 #include "serialization/binary_utils.h"
 #include "serialization/string.h"
+#include "serialization/vector_bool.h"
 
 using namespace crypto;
 
@@ -970,7 +972,7 @@ bool add_burned_amount_to_tx_extra(std::vector<uint8_t>& tx_extra, uint64_t burn
 //---------------------------------------------------------------
 bool add_new_service_node_to_tx_extra(
         std::vector<uint8_t>& tx_extra,
-        const tx_extra_ethereum_new_service_node& new_service_node) {
+        const eth::event::NewServiceNode& new_service_node) {
     tx_extra_field field = new_service_node;
     if (!add_tx_extra_field_to_tx_extra(tx_extra, field)) {
         log::info(logcat, "failed to serialize tx extra for new service node transaction");
@@ -981,7 +983,7 @@ bool add_new_service_node_to_tx_extra(
 //---------------------------------------------------------------
 bool add_service_node_removal_request_to_tx_extra(
         std::vector<uint8_t>& tx_extra,
-        const tx_extra_ethereum_service_node_removal_request& removal_request) {
+        const eth::event::ServiceNodeRemovalRequest& removal_request) {
     tx_extra_field field = removal_request;
     if (!add_tx_extra_field_to_tx_extra(tx_extra, field)) {
         log::info(
@@ -993,7 +995,7 @@ bool add_service_node_removal_request_to_tx_extra(
 //---------------------------------------------------------------
 bool add_service_node_removal_to_tx_extra(
         std::vector<uint8_t>& tx_extra,
-        const tx_extra_ethereum_service_node_removal& removal_data) {
+        const eth::event::ServiceNodeRemoval& removal_data) {
     tx_extra_field field = removal_data;
     if (!add_tx_extra_field_to_tx_extra(tx_extra, field)) {
         log::info(logcat, "failed to serialize tx extra for service node removal transaction");

src/cryptonote_basic/cryptonote_format_utils.h

@@ -38,6 +38,7 @@
 #include "crypto/crypto.h"
 #include "crypto/hash.h"
 #include "cryptonote_basic_impl.h"
+#include "l2_tracker/events.h"
 #include "logging/oxen_logger.h"
 #include "serialization/binary_utils.h"
 #include "serialization/json_archive.h"
@@ -190,12 +191,12 @@ bool get_encrypted_payment_id_from_tx_extra_nonce(
         const std::string& extra_nonce, crypto::hash8& payment_id);
 bool add_burned_amount_to_tx_extra(std::vector<uint8_t>& tx_extra, uint64_t burn);
 bool add_new_service_node_to_tx_extra(
-        std::vector<uint8_t>& tx_extra, const tx_extra_ethereum_new_service_node& new_service_node);
+        std::vector<uint8_t>& tx_extra, const eth::event::NewServiceNode& new_service_node);
 bool add_service_node_removal_request_to_tx_extra(
         std::vector<uint8_t>& tx_extra,
-        const tx_extra_ethereum_service_node_removal_request& removal_request);
+        const eth::event::ServiceNodeRemovalRequest& removal_request);
 bool add_service_node_removal_to_tx_extra(
-        std::vector<uint8_t>& tx_extra, const tx_extra_ethereum_service_node_removal& removal_data);
+        std::vector<uint8_t>& tx_extra, const eth::event::ServiceNodeRemoval& removal_data);
 uint64_t get_burned_amount_from_tx_extra(const std::vector<uint8_t>& tx_extra);
 bool is_out_to_acc(
         const account_keys& acc,

src/cryptonote_basic/miner.cpp

@@ -35,6 +35,7 @@
 #include <oxenc/base64.h>
 
 #include <numeric>
+#include <stdexcept>
 
 #include "common/command_line.h"
 #include "common/file.h"

src/cryptonote_basic/tx_extra.h

@@ -30,17 +30,17 @@
 
 #pragma once
 
+#include "common/exception.h"
 #include "crypto/crypto.h"
 #include "crypto/eth.h"
 #include "cryptonote_basic.h"
+#include "l2_tracker/events.h"
 #include "oxen_economy.h"
-#include "common/exception.h"
 #include "serialization/binary_archive.h"
 #include "serialization/binary_utils.h"
 #include "serialization/serialization.h"
 #include "serialization/variant.h"
 
-
 namespace cryptonote {
 
 constexpr size_t TX_EXTRA_PADDING_MAX_COUNT = 255, TX_EXTRA_NONCE_MAX_COUNT = 255;
@@ -196,7 +196,8 @@ void serialize_value(Archive& ar, tx_extra_padding& pad) {
                                    // tag part of the padding
 
     if (remaining > TX_EXTRA_PADDING_MAX_COUNT - 1)  // - 1 as above.
-        throw oxen::traced<std::invalid_argument>{"tx_extra_padding size is larger than maximum allowed"};
+        throw oxen::traced<std::invalid_argument>{
+                "tx_extra_padding size is larger than maximum allowed"};
 
     char buf[TX_EXTRA_PADDING_MAX_COUNT - 1] = {};
     ar.serialize_blob(buf, remaining);
@@ -452,7 +453,7 @@ std::vector<std::string> readable_reasons(uint16_t decomm_reasons);
 // where we want something in-between a bit field and a human-readable string.
 std::vector<std::string> coded_reasons(uint16_t decomm_reasons);
 
-// Pre-Heimdall service node deregistration data; it doesn't carry the state change (it is only
+// Old (pre-HF12) service node deregistration data; it doesn't carry the state change (it is only
 // used for deregistrations), and is stored slightly less efficiently in the tx extra data.
 struct tx_extra_service_node_deregister_old {
 #pragma pack(push, 4)
@@ -627,64 +628,6 @@ struct tx_extra_oxen_name_system {
     END_SERIALIZE()
 };
 
-struct tx_extra_ethereum_contributor {
-    eth::address address;
-    uint64_t amount;
-
-    tx_extra_ethereum_contributor() = default;
-    tx_extra_ethereum_contributor(const eth::address& addr, uint64_t amt) :
-            address(addr), amount(amt) {}
-
-    BEGIN_SERIALIZE()
-    FIELD(address)
-    FIELD(amount)
-    END_SERIALIZE()
-};
-
-struct tx_extra_ethereum_new_service_node {
-    uint8_t version = 0;
-    eth::bls_public_key bls_pubkey;
-    eth::address eth_address;
-    crypto::public_key service_node_pubkey;
-    crypto::ed25519_signature signature;
-    uint64_t fee;
-    std::vector<tx_extra_ethereum_contributor> contributors;
-
-    BEGIN_SERIALIZE()
-    FIELD(version)
-    FIELD(bls_pubkey)
-    FIELD(eth_address)
-    FIELD(service_node_pubkey)
-    FIELD(signature)
-    FIELD(fee)
-    FIELD(contributors)
-    END_SERIALIZE()
-};
-
-struct tx_extra_ethereum_service_node_removal_request {
-    uint8_t version = 0;
-    eth::bls_public_key bls_pubkey;
-
-    BEGIN_SERIALIZE()
-    FIELD(version)
-    FIELD(bls_pubkey)
-    END_SERIALIZE()
-};
-
-struct tx_extra_ethereum_service_node_removal {
-    uint8_t version = 0;
-    eth::address eth_address;
-    uint64_t amount;
-    eth::bls_public_key bls_pubkey;
-
-    BEGIN_SERIALIZE()
-    FIELD(version)
-    FIELD(eth_address)
-    FIELD(amount)
-    FIELD(bls_pubkey)
-    END_SERIALIZE()
-};
-
 // tx_extra_field format, except tx_extra_padding and tx_extra_pub_key:
 //   varint tag;
 //   varint size;
@@ -707,12 +650,12 @@ using tx_extra_field = std::variant<
         tx_extra_oxen_name_system,
         tx_extra_tx_key_image_proofs,
         tx_extra_tx_key_image_unlock,
+        eth::event::NewServiceNode,
+        eth::event::ServiceNodeRemovalRequest,
+        eth::event::ServiceNodeRemoval,
         tx_extra_burn,
         tx_extra_merge_mining_tag,
         tx_extra_mysterious_minergate,
-        tx_extra_ethereum_new_service_node,
-        tx_extra_ethereum_service_node_removal_request,
-        tx_extra_ethereum_service_node_removal,
         tx_extra_padding>;
 }  // namespace cryptonote
 
@@ -750,12 +693,9 @@ BINARY_VARIANT_TAG(
 BINARY_VARIANT_TAG(cryptonote::tx_extra_burn, cryptonote::TX_EXTRA_TAG_BURN);
 BINARY_VARIANT_TAG(
         cryptonote::tx_extra_oxen_name_system, cryptonote::TX_EXTRA_TAG_OXEN_NAME_SYSTEM);
+BINARY_VARIANT_TAG(eth::event::NewServiceNode, cryptonote::TX_EXTRA_TAG_ETHEREUM_NEW_SERVICE_NODE);
 BINARY_VARIANT_TAG(
-        cryptonote::tx_extra_ethereum_new_service_node,
-        cryptonote::TX_EXTRA_TAG_ETHEREUM_NEW_SERVICE_NODE);
-BINARY_VARIANT_TAG(
-        cryptonote::tx_extra_ethereum_service_node_removal_request,
+        eth::event::ServiceNodeRemovalRequest,
         cryptonote::TX_EXTRA_TAG_ETHEREUM_SERVICE_NODE_REMOVAL_REQUEST);
 BINARY_VARIANT_TAG(
-        cryptonote::tx_extra_ethereum_service_node_removal,
-        cryptonote::TX_EXTRA_TAG_ETHEREUM_SERVICE_NODE_REMOVAL);
+        eth::event::ServiceNodeRemoval, cryptonote::TX_EXTRA_TAG_ETHEREUM_SERVICE_NODE_REMOVAL);

src/cryptonote_basic/txtypes.h

@@ -30,6 +30,11 @@ enum class txtype : uint16_t {
     _count
 };
 
+inline constexpr bool is_l2_event_tx(txtype type) {
+    return type >= txtype::ethereum_new_service_node &&
+           type <= txtype::ethereum_service_node_removal;
+}
+
 inline constexpr std::string_view to_string(txversion v) {
     switch (v) {
         case txversion::v1: return "1"sv;