Sending more than one transaction at a time is easy, right?

This blog post explores the challenges of building a backend that writes to the blockchain at scale and the need for multiple safeguards we built into thirdweb Engine.

nonce has already been used

We’ve all made the same mistake. We built an API to mint an NFT from our backend wallet, confirmed it worked in testing, and launched to production only to see frequent errors like the one above.

What is a nonce, and what went wrong?

A nonce is an internal counter of the number of transactions sent from an EVM wallet. It starts at 0 is increases by 1 after each sent transaction. Transactions are confirmed onchain in the order of the nonce, and nonces cannot be skipped. Example: If a transaction with nonce 3 is submitted, it will block until nonce 2 is submitted (or until it times out and gets dropped).

(This concept is crucial to preserve order in a distributed environment where requests may be received by different servers out of order, such as the blockchain ledger.)

Using a nonce more than once results in the error nonce has already been used. This nonce value is set to the next expected value when sending a write transaction with any web3 SDK. But our backend might attempt multiple transactions at the same time. If one transaction hasn't completed yet, the following transaction doesn't know about it and will attempt to use the same nonce, resulting one or more transactions failing.

What’s the simple fix?

Fix: Wait for one transaction to complete before sending the next one.

The naive solution avoids reusing nonces by only allowing one transaction at a time. This approach requires added infrastructure and severely limits our backend wallet to one transaction every ~2 blocks. This means one transaction every 10 seconds on chains like Base, Optimism, or Polygon.

This is won't scale for any production app. How can we send multiple transactions at once without relying on the onchain nonce?

Fix: Keep track of nonces offchain.

To keep track of in-flight transactions, we need to track nonce values ourselves in an atomic datastore like Redis. Specifically we’ll need to:

1. Sync the offchain nonce to the onchain value first.
2. Set a transaction's nonce value to the offchain nonce before sending it to RPC*.
3. Increment the nonce after a successful RPC call.

*RPC is the interface that allows our app to send requests to the blockchain. Note that sending a request to RPC does not mean the transaction is confirmed onchain. Read more about thirdweb RPC Edge.

This approach works most of the time and optimistically use the next unused nonce for each transaction without waiting for previous transactions.

But we’ve now introduced a massive maintainability headache:

Nonces must increment exactly by one. If a transaction is stuck or dropped for any reason, the blockchain never receives a transaction with that nonce. All transactions with higher nonces are blocked and will eventually time out (silently fail). All future transactions will be blocked indefinitely until the offchain nonce value is manually re-synced.

Problem: Transactions can fail for many reasons.

You’ve probably already encountered some of these. There’s numerous reasons why a transaction might fail, and we’re still finding more.

Here are common failure cases and how we might work around them.

• A transaction fails to estimate gas due to execution reverting.
  • Contract calls that are expected to revert (think of it like throwing an exception) cannot estimate gas properly.
  • Fix: Simulate the transaction first. If the simulation fails, don't increment the nonce nor send the transaction.
• The backend wallet is out of gas to send the transaction.
  • Fix: Set up alerts to top up the backend wallet when it is low on gas funds.
• The gas price set for the transaction is too low, or the network is experiencing high load (”gas spikes”).
  • Fix: Re-submit the transaction with updated aggressive gas settings. It’s crucial to re-use the same nonce to avoid sending a duplicate transaction.
  • Caveat: Avoid unbounded gas spend by only re-submitting if current gas prices are below acceptable limits.
• The blockchain or RPC is down, or our app hit RPC rate limits.
  • Fix: Re-submit the transaction if the issue is intermittent. Otherwise cancel the transaction after some deadline to unblock other transactions.
• Unexpected/silent errors, like the RPC mempool not broadcasting transactions or one of the RPC nodes dropping requests.
  • Retries and cancellations adequately handle most edge cases.
  • Add RPC failover to help mitigates RPC-specific issues.

A transaction that returns an “execution reverted” error will not cause nonce issues. It is confirmed onchain so the nonce is used by the wallet. But we'll want to minimize these cases since gas is wasted.

Phew! So my application is reliable now right? Mostly. Scalable? Not yet.

Problem: The backend will fall over at high throughput.

Read and write requests in web3 can be orders of magnitude slower than web2.

Web2 devs are familiar with < 10ms database calls and < 200ms network calls. But in web3, a transaction involves submitting an RPC request, waiting for the mempool to broadcast the transaction, and waiting for that transaction to be confirmed onchain. This flow takes at least 2 blocks or more depending on gas + other variables. That's 5+ seconds depending on the blockchain. And stuck transactions that need to be re-submitted add tens of seconds more.

Of course we want to avoid slow, blocking server requests. At 5s per request, even 20 requests per second means 100 concurrent server requests. And this value grows quickly as latency or throughput increases. High or unbounded concurrent requests lead to high resource usage, issues with preconfigured server/DB connection limits, rate limits to third-party services, and more. These are common reasons why backends experience outages during high traffic.

Solution: Control transaction concurrency with a worker queue.

Store transaction requests received by our server (fast) in a queue to be submitted to the blockchain by workers later (slow). Server requests are persisted quickly and a pool of workers process those requests (making slower calls like gas estimation and simulation) eventually. Both pieces can be scaled independently as needed.

At this point our backend is sending transactions at scale and reliably with high confidence!

…though there’s a bunch of other features we’ll want, such as:

• Webhooks when transactions are completed.
• An observability dashboard to view transactions submitted, their status, and any errors for debugging.
• The capability to manually retry/cancel transactions.
• Tracking nonces across multiple wallets and chains.

Why we built thirdweb Engine

Hopefully this post illustrates the challenges of building a production-ready web3 backend. We built Engine to tackle these problems and more, including:

• Support for KMS-secured backend wallets
• Relayers for gasless transactions
• Contract deployments on any EVM chain
• Account abstraction
• Webhooks for low gas balance and transaction events
• Contract subscriptions
• Production-ready RPC and IPFS infrastructure

Engine internally powers multiple parts of the thirdweb platform, and we’re committed to continuously making it more useful, reliable, and faster.

Engine is open-source, and it can be self-hosted, or managed by thirdweb starting at $99 per month. Read the docs to learn more.

And thirdweb is hiring!

The small team at thirdweb is on a mission to build the most intuitive and complete web3 platform. Our products empower 70,000+ developers each month including Shopify, AWS, Coinbase, Rarible, Animoca Brands, and InfiniGods.

See our open roles. We’d love to work with you!