Security Vulnerability Incident Report 12/8

We recently notified our community of a security vulnerability in the usage of two standards in an open-source library. This vulnerability impacts smart contracts across the web3 industry, including some of our pre-built smart contracts.

Since making this announcement, we have helped users — using a tool built by our team — to successfully mitigate over 9,800 smart contracts across 37 chains to prevent this exploitation. Having made this progress and after dialogue with other key participants in the ecosystem, we feel that now is the right time to share details of the vulnerability.

The Vulnerability

On November 20th, our auditing partner, 0xMacro, notified us of the vulnerability as part of a rigorous security process that involves auditing and scrutinizing our smart contract code. Our team worked tirelessly and proactively to investigate the particulars of the vulnerability to start building a safe mitigation path.

Any contract implementing ERC2271and Multicall (two third party open source standards) with a valid trusted forwarder are vulnerable to address spoofing via malicious calldata in forwarded requests. The vulnerability allows attackers to perform unauthorized actions, including but not limited to executing privileged function calls and taking control of assets from any account.

Many of our pre-built contracts implement ERC-2771, enabling transaction gas fee sponsorship. This involves the trusted forwarder contract appending the original sender's address to the transaction calldata. For contracts accepting meta transactions, rather than relying on msg.sender, they would use an alternative method to obtain the original transaction sender. When a transaction is forwarded, the _msgSender() function extracts the message sender from the last 20 bytes of the calldata. If the msg.sender is not a trusted forwarder, it returns msg.sender.

function _msgSender() internal view virtual override returns (address sender) {
    if (isTrustedForwarder(msg.sender)) {
        // The assembly code is more direct than the Solidity version using `abi.decode`.
        assembly {
            sender := shr(96, calldataload(sub(calldatasize(), 20)))
        }
    } else {
        return super._msgSender();
    }
}

Aside from meta-transactions, many pre-built contracts also utilize Multicall to improve the developer experience by allowing batch processing of multiple functions in a single transaction. This involves receiving the transaction calldata for multiple calls, iterating through them, and executing self delegate calls with the provided data.

function multicall(bytes[] calldata data) external virtual returns (bytes[] memory results) {
    results = new bytes[](data.length);
    for (uint256 i = 0; i < data.length; i++) {
        results[i] = Address.functionDelegateCall(address(this), data[i]);
    }
    return results;
}

Each delegate call to itself via multicall() is treated as a distinct external transaction containing only the relevant data for that call. It won't include any additional message sender data from a trusted forwarder. However, it preserves the call context, including the value of msg.sender.

When a trusted forwarder invokes multicall(), any delegate call it initiates using _msgSender() will assume that sender data is supplied to the call and use it accordingly.

In a scenario where an attacker manipulates a trusted forwarder to invoke a contract inheriting both of these functions, the attacker can inject extra sender data into each call within multicall(), thereby enabling them to impersonate any desired sender when making the call.

You can also read OpenZeppelin’s disclosure of the vulnerability here.

Why was this combination present in our pre-built smart contracts?

Our pre-built smart contracts are designed for broad use cases and optimized for user experience. Each contract has always allowed contract owners to sponsor gas for transactions. This allows creators to pay the gas costs for users interacting with their contracts, encouraging participation in a project.

Multicall provides a function to batch together multiple contract function calls in a single transaction call.

These features themselves don’t trigger this issue when used independently. It is only when using the two together that the vulnerability appears. Thus, the affected contracts inherited the vulnerability from the combination of these two features.

During our investigation, we identified 8 other protocols who also used this combination and alerted them to the fact that they could be impacted by this vulnerability, with the goal of supporting them in mitigating their contracts.

Contract mitigation process

Mitigating the vulnerability requires a patch in the multicall to take in consideration for any forwarder context and encode the transaction message sender address at the end of each calldata loop.

function isTrustedForwarder(address forwarder) virtual returns (bool);

function multicall(bytes[] calldata data) external returns (bytes[] memory results) {
		results = new bytes[](data.length);
		address sender = _msgSender();
		bool isForwarder = msg.sender != sender;
    for (uint256 i = 0; i < data.length; i++) {
				if (isForwarder) {
						results[i] = Address.functionDelegateCall(address(this), abi.encodePacked(data[i], sender));
        } else {
            results[i] = Address.functionDelegateCall(address(this), data[i]);
        }
    }
		return results;
}

Remediating thirdweb pre-built smart contracts for new deployments (after November 22)

To prevent the vulnerability from propagating further, we created a patch for our pre-built smart contracts that were created after November 22nd, 7pm PT. We had to do so without revealing too much information about the vulnerability, lest we give bad actors valuable information. To accomplish this, we created a patch through an indirect code path. Specifically, the patch prevents any contracts (including a forwarder) from executing the multicall function.

We decided to prevent smart contract wallets and other contracts from executing multicall in the contract to avoid any potential issues. Smart contract wallets typically already have a batch execution function that provides the same functionality. Additionally, batch execution can be integrated into the trusted forwarder contract in the future if needed.

// Multicall.sol
function multicall(bytes[] calldata data) external virtual returns (bytes[] memory results) {
    results = new bytes[](data.length);
    for (uint256 i = 0; i < data.length; i++) {
        results[i] = Address.functionDelegateCall(address(this), data[i]);
    }
    return results;
}

// Updated function in Address library (Address.sol)
function functionDelegateCall(
    address target,
    bytes memory data,
    string memory errorMessage
) internal returns (bytes memory) {
    require(isContract(target) && !isContract(msg.sender), "Address: invalid delegate call");
		...
}

Mitigation for deployed thirdweb pre-built contracts

Following ERC2771 security consideration, the trusted forwarder addresses in our ERC2771Context implementation are immutable and can only be set during deployment. We found two different mitigation strategies for our pre-built contracts.

For upgradeable dynamic pattern contracts, we mitigate the vulnerability by disabling trusted forwarders through a multicall transaction with the following steps:

1. Add a disable function to the upgradable contract that sets the storage slot value to 0
2. Execute the function to reset the storage slot values of the trusted forwarders, effectively disabling them in the contract
3. Remove the function from the contract to prevent its future use after the mitigation.

function disable(bytes32[] memory _slots) external {
		for (uint256 i = 0; i < _slots.length; i++) {
		    bytes32 slot = _slots[i];
        assembly {
		        sstore(slot, 0)
        }
    }
}

For non-upgradeable contracts, the mitigation involves locking down the contracts and migrating to a new contract. The locking process varies for each contract type but typically includes:

1. Adding [Locked] to the contract metadata name to indicate that it has been locked.
2. Disabling key contract actions such as minting, transfers, claiming, and more.
3. Revoking all roles.

After locking token type contracts, the owner can snapshot the current token holders, deploy a new contract, and distribute the tokens to the rightful owners.

Building our mitigation tool

Once we became aware of the vulnerability, our security and engineering teams, along with our security partners, worked around the clock to create an easy-to-use mitigation tool for all affected thirdweb contracts.

Our pre-built smart contracts are widely used throughout the web3 ecosystem. A significant number of our users deployed these smart contracts without providing their contact information, as our platform did not always require it. As a result, we had to provide a tool that could be easily used by thousands of users to detect and mitigate any potential vulnerabilities without needing our direct involvement.

The tool allows contract admins to either upgrade their contract (if possible) or lock and migrate their contract as detailed above. Creating a simple but powerful tool for so many different varieties of contracts required heavy-duty technical work in a condensed timeframe. As of December 8th, over 9,800 smart contracts have been mitigated using this tool.

Our incident response team was split into 2 working groups:

1. Data working group focuses on collecting, enriching, analyzing and monitoring on-chain data.
2. Mitigation working group focuses on mitigating the vulnerable contracts.

Data Collection and Enrichment for Outreach

We indexed our factory contract address to retrieve a list of deployed contracts and their corresponding proxy implementation contract addresses. However, gathering a list of impacted contracts was challenging due to our platform's support for contract deployment on any EVM network. We narrowed down the list to 26 networks based on contract deployment usage metrics. The list of networks we have data on are:

To assess data quality and data correctness, we validated our collected dataset with external data sources. However, we were not able to run validate the data for some networks due to limited infrastructure provider support.

We used data infrastructure services to bootstrap our data collection for certain networks and to identify vulnerable contracts outside of thirdweb. We included the contract address and platform name of these vulnerable contracts in our security disclosure to the library maintainer.

After finalizing the contract deployment list, we enriched all contracts with names, volumes, dollar values, and analytical information from internal and external data sources.

Using the enriched data set, we created an outreach target list to help mitigate the risk. For some impacted users, we scraped the Internet for their publicly available contact information because our tools have not always required users to provide contact information.

Snapshot

Our snapshot tool captures two types of snapshot:

1. Contract metadata state: contract metadata, permissions, token royalty information, etc.
2. Token balance snapshots: wallet balances and their respective token IDs depending on the token type (ERC20, ERC721, ERC1155).

Snapshotting is a critical element of our mitigation strategy and necessary for migrating most contract types. We used our enriched data and integrated with multiple first and third-party infrastructure providers to gather the required data.

Our tool can snapshot relevant contract states at a specified block number. In the unfortunate event of a contract exploit, we can restore the contract state to the last known good state.

Our snapshot tool had to support one of the largest NFT collections, exceeding 60 million tokens. None of the infrastructure providers could snapshot such a large collection, so we revised our approach to handle large collections.

Migration Contracts

To enable a more seamless migration experience, we built nine migration contracts that accept a previously deployed contract address in the initializer:

• DropERC1155M
• DropERC20M
• DropERC721M
• LoyaltyCardM
• OpenEditionERC721M
• SignatureDropM
• TokenERC1155M
• TokenERC20M
• TokenERC721M

Notable changes in these contracts:

1. Copied the previously configured settings to the new contract during initialization, including collection and token metadata, claim conditions, etc.
2. Added a migrate function to distribute tokens to the previous holders using provided Merkle proofs or from an authorized migrator role wallet.
3. Removed ERC2771 integration in favor of ERC4337.

The new migration contracts have been reviewed by our auditing partners and ecosystem partners.

Monitoring

We have deployed two different strategies in our monitoring to alert us when the vulnerability has been exploited. The transaction alert is stored in a database so that contract snapshotting can attempt to restore the contract to its original state just before the exploit occurs

Detection

In our analysis, we discovered that some contracts share the same underlying implementation address and contract bytecode but are not impacted by the same vulnerability due to differences in their initialization configuration, specifically, the values configured for the trusted forwarders.

To detect the vulnerability at scale across multiple EVM networks, we built a tool to detect the specific vulnerability for contracts. The tools work by using the combination of static analysis on the contract bytecode and dynamic analysis by simulating a series of transactions to assess the impact level of the vulnerability in a contract. Currently, the detection tool is only optimized for thirdweb contracts.

Trust and Safety

Trust and safety were high priorities as we built the mitigation tool, as we aimed to prevent its malicious use. To enhance the security of our tools, we require users to authenticate through 'Sign In With Ethereum (SIWE)' to verify ownership of a wallet address. All of the mitigation logic and operations are accessible only through authenticated API endpoints.

After proving ownership of the wallet, the tool provides a list of vulnerable contracts associated with the wallet, allowing the owner to view the status of their contracts and initiate mitigation.

We also worked closely with major marketplaces and block explorers to deliver an updated list of impacted smart contracts so that they could implement safety measures on their end.

Mitigation

Depending on the selected contract type, different mitigation strategies are presented to the user.

For upgradeable contracts, the tool assembles all the necessary calldata for the migration into a single multicall transaction. This allows the contract owner to perform the upgrade in a single transaction atomically without leaving a window for the contract to be in an intermediate state.

For contracts that are not upgradeable, the tool guides the contract owner through the necessary steps for mitigating their contracts. During the locking step, the tool scans for all the contract states that need to be reset to lock the contract down. It then assembles all the calldata into a single multicall transaction, enabling the contract owner to lock the contract in a single transaction. After locking the contract, the owner will be given the option to snapshot the contract or upload their own snapshot. Deploying a new migration contract requires a locked contract address and a snapshot merkle root that represents the proof of token ownership. After deploying a new contract, there are two options for token distribution: (1) Automated airdrop of all tokens or (2) Token holders can claim them through a provided claim link.

Before the public disclosure, we set up a test environment using forked networks to safely and discretely test the tool before deployment. Additionally, we onboarded a small group of partners to test the tool on their contracts.

Support

To make the mitigation process as smooth as possible, we produced written and visual guides and activated our global support team to provide around-the-clock help for smart contract owners.

Looking forward

We would like to thank our community for responding swiftly and proactively to mitigate their vulnerable smart contracts. We will continue to support all vulnerable smart contract owners as they work through the mitigation process.

We remain committed to our mission to build best-in-class developer tools for web3, and believe this is an opportunity to enhance security best practices for the whole industry. If you have any further questions about this process, please contact support@thirdweb.com.