All posts

CometBFT Consensus with TEE

12 min read · ComputeFlux Team
Trust & Security

In one sentence. Every validator votes on every block, and each vote carries a freshly generated hardware certificate proving that the voter's code has not been tampered with as of this exact block.

Picture it like this. A committee vote where each member, at the moment they raise their hand, must also hold up a photo ID issued seconds ago by the building's own security system. Ordinary networks check ID once at the door. ComputeFlux re-checks it on every single vote — because the interesting attack isn't sneaking in, it's being replaced after you're already inside.

Why it matters. "Decentralized" usually just means many computers. It does not mean honest ones. Byzantine fault tolerance is the mathematics of extracting a correct answer from a group in which some members are actively working against you. Bolting hardware attestation onto each vote closes the one gap that math alone leaves open.

Know your BFT already? Skip to Vote Extensions — that's the part specific to ComputeFlux.


Consensus algorithms are well enough understood that picking one is rarely a question of correctness. It's a question of fit. The production BFT landscape offers several credible candidates: PBFT with its 30-year pedigree, HotStuff with its linear communication and pipelined design, and the Tendermint lineage that culminates in CometBFT. All of them solve the same problem — agreeing on a total order of transactions while some participants lie — but their architectural assumptions diverge sharply once the consensus engine has to run entirely inside Intel SGX's encrypted memory. ComputeFlux chose CometBFT because of how it maps onto enclave constraints, not despite them.

Why CometBFT Over the Alternatives

PBFT, as described in Castro and Liskov's 1999 paper, achieves optimal resilience (3f + 1 nodes tolerating f Byzantine faults) with O(n²) message complexity per consensus round — every node messages every other node across three phases. For a four-node deployment that's manageable, but the quadratic scaling makes it a poor foundation for network growth, and PBFT has no native support for validator rotation, a feature production deployments invariably need.

HotStuff, introduced in 2018 and adopted by Facebook's Diem, reduces message complexity to O(n) through a leader-centric design and pipelines three consensus instances for higher throughput than PBFT. But its correctness depends on aggregating n − f signatures per phase via a quorum certificate mechanism, and inside SGX, where every cryptographic operation consumes enclave memory for temporary buffers and key material, that constant factor stops being negligible. HotStuff's three-chain commit rule also requires maintaining additional state about past quorum certificates, straining the enclave's limited heap.

CometBFT sits at a pragmatic middle ground. Its predecessor Tendermint is battle-tested across Cosmos Hub, Binance Chain, and dozens of other networks that collectively secure tens of billions in value. Its O(n²) gossip-based proposal broadcast is perfectly acceptable at ComputeFlux's four-node validator set. And the ABCI (Application Blockchain Interface) separates consensus from state machine execution across a clean four-method boundary: CheckTx, PrepareProposal, ProcessProposal, FinalizeBlock.

That separation is the killer feature for a TEE deployment. Consensus logic keeps a lean memory footprint while application logic — contract execution from Article 5, billing settlement from Article 10 — runs in separately managed memory inside the same enclave process, across a well-defined interface. Being pure Go helps too: no FFI overhead, no CGo boundary crossings.

The trade-off is that CometBFT's rotating proposer election and its reliance on +2/3 voting power at each step create a performance ceiling of roughly 1,000 transactions per second on modest hardware. For a gateway settlement chain — not a general-purpose smart contract platform — this throughput is sufficient, and the simplicity is preferable.

The EPC Memory Budget: 128MB to 2048MB

As Article 1 covered, SGX's Enclave Page Cache (EPC) is the physical DRAM region reserved for enclave memory, and any page evicted from it costs roughly 8,000 cycles to fault back in. That constraint shapes CometBFT's configuration directly.

That budget has to cover a lot at once: the CometBFT state machine (block store, consensus params, peer state), the PebbleDB block store (Article 2), WASM runtime buffers (Article 5), DKG cryptographic material (Article 3), the mempool transaction cache, P2P network buffers, and the Go runtime's heap and goroutine stacks. At the original 512MB size — before commit de94c4f expanded EPC to 2048MB — the margin was thin. A burst of unconfirmed mempool transactions could trigger page faults, and the resulting latency spikes surfaced at the consensus layer as missed block deadlines.

Mempool sizing is the main relief valve for EPC pressure. ComputeFlux runs CometBFT's standard mempool configuration rather than a cap tailored to the enclave, so it's CometBFT's own size limit that holds back unbounded growth today. CheckTx helps independently: signer verification, nonce checks, and a contract dry-run all run before a transaction is even considered for the mempool, filtering out a whole class of invalid traffic before it can contribute to memory pressure.

A sustained flood that gets past that filtering could still, in principle, push a smaller enclave toward thrashing. Tightening the mempool limit specifically for the EPC-constrained deployment is a natural next step — not something already in place as a bespoke 512-entry cap.

The same budget constraint motivated choosing PebbleDB over LevelDB in the first place: PebbleDB's Go-native implementation avoids the CGo calls to C++ LevelDB that would each require an OCALL — a transition from enclave to untrusted host, visible to the OS and breaking the enclave's memory access model for the call's duration.

The 2048MB expansion effectively ends thrashing for normal workloads. Compared with the 512MB configuration it replaced, it leaves real headroom across every consumer at once: CometBFT, PebbleDB, WASM runtime buffers, DKG material, network buffers, the Go heap and goroutine stacks, and the transient allocation spikes that come with block finalization.

Vote Extensions: Embedding Attestation in the Consensus Protocol

A BFT network of TEE nodes faces a problem standard validator networks don't: proving a validator is genuinely executing within an enclave, not just presenting a public key extracted from a compromised node or generated in software. Article 1 covers how TEE Quote verification works in general — the P2P handshake layer and the certificate chain back to Intel's root CA. This article is about the piece that's specific to CometBFT: how attestation gets embedded directly into the voting protocol itself, not bolted on beside it.

A separate attestation handshake at connection establishment — exchanging Quotes over the P2P transport before consensus messages begin — has two problems specific to the consensus context. First, freshness: a Quote generated at connection time might be seconds or minutes old by the time the validator votes on a block, and an attacker could compromise an enclave after attestation but before voting while the stale Quote still appears valid. Second, the out-of-band handshake adds an extra round-trip to node discovery, increasing time-to-consensus after a network partition heals.

The Vote Extension mechanism in ABCI 2.0 solves both by embedding attestation directly into consensus messages. ExtendVote populates the vote's Extension field with a fresh TEE attestation report, its report data built from the current block height and a prefix of the block hash — binding the attestation to the specific block it's voting on, not just to "some connection established earlier." A receiving validator verifies the report's signature chain during VerifyVoteExtension and checks it against the expected code measurement. Failure means VerifyVoteExtension returns REJECT and the vote is discarded, treating the sender as Byzantine.

Worth being precise about scope. ABCI 2.0's vote extension mechanism attaches to the precommit step, not to prevotes: ExtendVote and VerifyVoteExtension run once per round, on the vote that actually decides finality. That's a narrower window than re-attesting on every message, and simpler than re-attesting separately at prevote and precommit. One fresh report per round, generated and checked as part of the step that commits the block, is enough to close the stale-attestation gap described above without doubling the work.

The Proposer Election and Its TEE-Specific Implications

CometBFT selects the block proposer through deterministic round-robin weighted by voting power. In ComputeFlux's four-node deployment with equal voting power, that's a simple circular sequence — node 0 proposes block h, node 1 proposes h+1, and so on — known to all nodes in advance.

That determinism matters for attestation. A validator receiving a proposal checks more than its cryptographic signature. It also checks that the proposer's TEE Quote, attached via Vote Extension, matches the node expected to propose at this height. If node 2 should be proposing but the proposal arrives carrying node 3's enclave measurement, the network has caught Byzantine behavior that neither signature verification nor standalone attestation would flag on its own.

Weaker proposal verification would let a compromised node with a perfectly valid enclave propose out of turn — the opening for a censorship attack, where a malicious proposer quietly excludes specific transactions. Deterministic election plus identity binding closes it. Other validators reject an out-of-turn proposal even when the proposal and the Quote are each individually valid, because the Quote belongs to the wrong node.

Consensus Liveness and Safety Under EPC Pressure

BFT consensus makes a sharp distinction between safety (no two correct nodes decide different values) and liveness (the protocol eventually produces a decision). Under normal operation CometBFT guarantees both; under memory pressure severe enough to cause EPC page faults, the line blurs.

When an enclave page faults, the SGX driver encrypts the page, writes it to unprotected memory, and records integrity metadata in a Merkle tree maintained by the Memory Encryption Engine. That's roughly 8,000 CPU cycles per fault. Let the enclave's working set exceed its EPC allocation and the system enters a thrashing regime where nearly every access faults. At 8,000 cycles on a 3GHz core, each access costs 2.7μs. A consensus round touches several hundred thousand memory operations — message deserialization, signature verification, state queries — so it can stretch from under 10ms to multiple seconds.

This lands directly on liveness. CometBFT's proposer timeout is a few seconds, and a thrashing enclave may not produce a proposal before it expires. If the next validator in the round-robin is under the same memory pressure, missed proposals cascade and the network stops producing blocks — a liveness failure with no Byzantine behavior anywhere in it.

ComputeFlux runs CometBFT's stock consensus configuration rather than an EPC-aware timeout schedule, so the mitigation comes for free: CometBFT already grows the proposal timeout on successive missed rounds. That gives a thrashing node room to stabilize while the Go garbage collector frees pages and the working set contracts. And the 2048MB EPC expansion is what turned this edge case from routine into rare.

Double Nonce Validation: Defense in Depth Against Replay

Transaction replay — submitting the same valid transaction twice — is classically defended against by nonces: each sender address maintains a monotonically incrementing counter, and a transaction is valid only if its nonce equals the address's current nonce + 1. ComputeFlux augments this single-layer check with dual-scope validation: database-level nonce continuity and intra-block nonce uniqueness.

The database-level check is standard: before processing a transaction, the validator queries the account's current nonce from state, and a nonce that isn't exactly current + 1 — too low (replay) or too high (a gap) — is rejected. This prevents replay across blocks, the most common attack vector.

The intra-block check defends against something subtler: a proposer that packages the same transaction twice inside one block, through a bug or on purpose. Both copies pass the database-level check, because state hasn't been updated mid-block. So the intra-block check keeps a set of (sender, nonce) pairs already seen in the current block and rejects the second occurrence outright. It is pure defense in depth — catching duplicates before state execution regardless of what upstream deduplication should have handled.

The concrete motivating case is cross-chain relay (Article 11): a SyncTx transaction initiates a token transfer, and if the same event gets independently observed and submitted by two relay scanners, the same SyncTx could land twice in one block. The database-level nonce check passes for the first occurrence; the intra-block deduplication catches the second. Without it, the gateway contract would credit the user twice, with no mechanism to unwind the double-spend short of a chain rollback.


Key Takeaways

  • The consensus engine was chosen for fit, not for benchmark wins. Faster designs exist. CometBFT is battle-tested across networks securing tens of billions, written in Go so it never has to cross into C code (expensive and dangerous inside an enclave), and cleanly separates "agreeing" from "executing" — which is what makes it survivable in the enclave's tight memory budget.
  • Attestation rides inside the vote, not beside it. The proof is bound to the specific block being voted on. A certificate generated when the machine connected an hour ago cannot vouch for a machine compromised five minutes ago; one generated for this block can.
  • Everyone knows whose turn it is. Because the block proposer is determined in advance, a genuine, fully attested enclave that proposes out of turn is immediately detectable — closing a censorship attack that ordinary signature checking would wave straight through.
  • Memory pressure threatens liveness, never correctness. A starved node may miss its turn and stall the network; it can never make two honest nodes disagree. Expanding the enclave's memory from 512MB to 2048MB turned this from a routine hazard into a rare one.
  • Replay is blocked twice, on purpose. Once across blocks, once within a single block. The second check exists for a real scenario, not a theoretical one: two cross-chain relay scanners can each legitimately spot the same deposit and submit it, and crediting a user twice has no clean undo.

CometBFT is where TEE attestation (Article 1) and DKG threshold signatures (Article 3) meet in practice — every block vote is both an attested claim about enclave state and, in aggregate, part of the threshold signing surface those articles describe.

That completes the foundation. Hardware you can verify, storage nothing can read, keys nobody holds, and agreement that survives liars. Part 2 turns to what all of it was built to carry: actual AI requests, actual money, actual failures at 3am.

Next — Article 5: WASM Contract Upgrades: blockchains are famously immutable. So how do you fix a bug in one without a hard fork or downtime?