Catapult Opt-In Migration Technical Overview

Hi Nembers!

This post should provide the community with better insights about how a Catapult Opt-In Migration is technically feasible.

As a fast forward introduction, please have a look at the following decision tree from @Jaguar0625, who posted this on Twitter here.

Catapult Migration Decision Tree

Rationale

This document analyses implications of the migration of datasets stored on one legacy blockchain network - hereafter referred to as the “NEM blockchain data” and “NEM blockchain”, also known as the “NIS blockchain” - onto a newly setup Catapult public blockchain network.

The proposed solution lays out a process that is distributed and can be audited by third parties.

The proposed migration of datasets from the NEM blockchain to a newly setup Catapult public blockchain network will store migration data on the NEM blockchain. This makes it possible for third parties to verify / audit the data that is included in the nemesis block by reading datasets on the NEM blockchain.

Additionally, the process uses features that are available on both blockchain networks without modifications, making it possible to start the migration process without delay.

The strong deadline imposed by the proposed migration strategy is clearly a disadvantage but its effect can be mitigated with the fact that unclaimed funds can be claimed at a later point in time, with a claiming period that will be extending over a few years (> 3 years).

With businesses that have implemented NEM running today, our aim was to not affect operations of our partners and supporters as well as not enforcing the migration of datasets.

Storage of Nemesis Data

The first block of the Catapult public network will hold all valid opt-in requests in the form of conventional Catapult transactions. These transactions are signed by their respective issuer(s).

In an attempt to distribute the knowledge about what will be included in the nemesis block, opt-in requests are stored on the NEM blockchain using transfer transactions with plain messages that contain JSON objects.

An example of a valid opt-in request for a namespace registration is shown in the following screenshot:

As you can see, the transaction holds quite some information in its’ message field. This is because it contains a signed Namespace Registration Transaction that can be executed as is on the newly setup Catapult public blockchain network.

In fact, these signed transactions are created with the TypeScript SDK for Catapult and stored on the NEM blockchain using NanoWallet features.

:warning: In order to permit the migration of multi-signature accounts with the same configuration as on the NEM blockchain, it is required that all involved cosignatories interact during the opt-in process.

Technical Implications

This section analyses implications of the migration of datasets stored on one legacy blockchain network - hereafter referred to as the “NEM blockchain data” and “NEM blockchain” - onto a newly setup Catapult public blockchain network.

Catapult technology offers many more features than the ones mentioned in this document. The sole purpose of explaining Catapult features in this document is to provide the necessary background information to understand specific migration decision items.

Signature Verification

Transactions on NEM & Catapult are signed cryptographically by the owner(s) of funds. This has the advantage of being verifiable at any time provided the signature and public key of the signer. In fact, a signature is deterministic and can be verified by third parties with the signer public key: the same message with the same private key always produces the same signature.

This property of being verifiable is also what allows to validate the sanity/integrity of said blockchain data. In fact, any transaction that appears in a block must have been signed by the owner(s) of the issuing account.

The Catapult protocol uses elliptic curve cryptography, more specifically it uses the twisted Edwards Curve (ed25519-donna). In fact, Catapult uses keypairs that can be represented as points on the ed25519 curve. These keypairs - a pair of private key and public key - are used to sign data and publish signatures alongside datasets on the Catapult network.

Nemesis Block Inclusion

The nemesis block in Catapult is the first block of the network. It can hold transactions that are usually signed with the nemesis account. The nemesis account is an ephemeral account that is only allowed to issue transactions included in the nemesis block, it cannot be used to issue transactions in subsequent blocks of the network.

The distribution of the network’s currency mosaic is to happen in the nemesis block as well. This is possible because the nemesis account is the owner of the network’s currency mosaic and namespace (e.g: nem.xem). In fact, the nemesis block can include a number of transfer transactions that are issued by the nemesis account and received as planned in the distribution configuration.

Based on this nemesis block configuration, we propose to include signed transaction payloads inside of the configuration file. This will permit to execute 1) multi-signature account modification transactions that are signed by cosignatories and 2) namespace registration transactions that are signed by their owner.

Catapult Multi-Signature Opt-In Feature

Catapult technology updates the multi-signature feature in several ways compared to the legacy system still in place. One of these upgrades implies that cosignatory accounts have to send a co-signature for the multi-signature setup - the initial multi-signature modification transaction.

:warning: This feature is not to be confused with the so-called opt-in migration. In fact, the Catapult Multisig Opt-In Feature is a protocol feature that was introduced with catapult-server.

:warning: Catapult also introduces limits in both, the number of cosignatories an account can have and the number of accounts of which one can be a cosignatory.

Dataset Migration Strategies

In the following table, you will find the analysed dataset migration strategies for the migration of legacy datasets.

Strategy Description
Opt-Out Migration Automatic datasets migration
Opt-Out & Opt-In Migration Automatic accounts migration
Manual namespaces migration
Manual multi-signature migration
Opt-In Migration Manual accounts migration
Manual namespaces migration
Manual multi-signature migration

Opt-Out Migration

In the event of a so-called Opt-Out Migration strategy, following are the technical implications of migrating legacy datasets:

  • All accounts (multisig & non-multisig) are migrated without the interaction of the owner (end-user).
  • All namespaces are migrated without the interaction of the owner (end-user).
  • The nemesis block would hold transactions that are not signed. This means that it is not possible to verify the block/transactions cryptographically.
  • :warning: Multi-signature modification transactions are not signed, hence cryptographically non-verifiable, in the nemesis block.
  • :warning: Namespace registration transactions are not signed, hence cryptographically non-verifiable, in the nemesis block.

Opt-Out & Opt-In Migration

In the event of a so-called Opt-Out & Opt-In Migration strategy, following are the technical implications of migrating legacy datasets:

  • All non-multisig accounts, except those that opt-out, are migrated without the interaction of the owner (end-user).
    • The funds of opted-out non-multisig accounts will be held in a reserve fund (unclaimed tokens).
  • Multi-signature accounts that send an opt-in request are migrated with interaction of the owners (end-users).
    • The funds of multisig accounts that have not opted-in pre-launch will be held in a reserve fund (unclaimed tokens).
  • Namespaces for which an opt-in request was sent are migrated with interaction of the owners (end-users).
  • :white_check_mark: Multi-signature modification transactions are signed, hence cryptographically verifiable, in the nemesis block.
  • :white_check_mark: Namespace registration transactions are signed, hence cryptographically verifiable, in the nemesis block.

Opt-In Migration

In the event of a so-called Opt-In Migration strategy, following are the technical implications of migrating legacy datasets:

  • Non-multisig accounts that send an opt-in request are migrated with interaction of the owners (end-users).
    • The funds of non-multisig accounts that have not opted-in pre-launch will be held in a reserve fund (unclaimed tokens).
  • Multi-signature accounts that send an opt-in request are migrated with interaction of the owners (end-users).
    • The funds of multisig accounts that have not opted-in pre-launch will be held in a reserve fund (unclaimed tokens).
  • Namespaces for which an opt-in request was sent are migrated with interaction of the owners (end-users).
  • :white_check_mark: Multi-signature modification transactions are signed, hence cryptographically verifiable, in the nemesis block.
  • :white_check_mark: Namespace registration transactions are signed, hence cryptographically verifiable, in the nemesis block.

Catapult Opt-In Migration

As announced in the first migration committee update, the migration committee recommended an Opt-In Migration with a Token Allocation process - it is also possible to use the opt-out strategy and the migration committee is inspecting its execution in parallel. This section will lay out the details of the technical execution of the Catapult Opt-In Migration.

Problem Statements

  • Multi-signature accounts can only be migrated given their cosignatories’ consent.
  • Namespace registration transactions must be signed by the namespace owners.

Proposed Solution

As it is possible to sign Catapult transactions offline, a NanoWallet Opt-In Plugin can be implemented to execute the creation of signed Catapult transactions. Multiple transaction types will come into play as to cover more datasets to be ported over.

Signed Catapult Transactions will be stored on the NEM blockchain by means of transfer transaction messages containing pre-defined data transfer objects (DTO). In particular, these DTOs will hold all the information required to execute the transactions on the newly setup Catapult public network.

Pre-defined data transfer objects (DTO) that will be involved in this Opt-In Migration include signed Catapult transactions with:

a) Namespace registration transactions with a duration of one year.
b) Multi-signature account modification transactions.

The migration process should be reproducible outside of NanoWallet given that some end-users manage their XEM with different client applications. This will be addressed with the release in the form of an opt-in package published on NPM. A reference implementation for the opt-in process will be delivered as a NanoWallet Plugin to be included in a later release.

Process Definition

  1. The end-user should send a pre-formatted opt-in request for datasets they wish to own on the newly setup Catapult network. This is illustrated in the following image:

  1. At the time of a pre-defined snapshot block, all opt-in requests will be collected and will reflect the content of the nemesis block. An illustration of this is found in the following image:

  1. The nemesis block generation tool reads a pre-configured block-properties-file.properties and includes signed transaction payloads.

Key Advantages

Strength
All datasets copied from the NEM blockchain and re-created on the Catapult public blockchain network are verifiable and registered on Catapult with conventional signed Catapult transactions.
Opt-In requests are stored on a functioning public blockchain network and therefore cannot be altered / tampered with, by any of the entities involved in the migration process.
The resulting nemesis block can be audited / verified by any third party provided with the Catapult nemesis account public key information.
End-users give their consent for the migration: This avoids a migration enforcement and is less likely to result in legal consequences for executing parties.
The process can also be used post-launch with a different funds origin.

Key Disadvantages

Weakness
Strong deadline for pre-launch opt-in requests.
The procedure doesn’t fit all jurisdictions. Yet, probably none will.
Complicated solution that leverages two different blockchain protocols to migrate datasets.

Ongoing Discussions

The community has mentioned the idea to define a strategy leveraging processes from both: Opt-Out Migration and Opt-In Migration. This would permit to require fewer interactions from end-users.

It is, in fact, possible to define the following strategy:

  • All non-multisig accounts are migrated by default except for those that send an explicit opt-out request.
  • Multi-signature accounts that should be migrated must send an opt-in request.
  • Owners of namespaces that should be migrated must send an opt-in request.

The change in process definition is trivial. In fact, an opt-out request can be formatted the same as an opt-in request for non-multisig accounts. The migration committee is currently investigating the change to implement opt-out requests for non-multisig accounts.

As mentioned on Twitter and in our Forum, it is important that you share your feedback with us about this topic.

Next Steps

  • A block height for when the snapshot happens must be determined.
  • The migration committee is working on the execution of the proposed migration process.
  • The change of including an opt-out for non-multisig accounts is being evaluated.
  • Tests are currently done with regards to the collection of opt-in requests.

9 Likes

Thanks Greg! This is exactly one of the posts I would see more often. No matter what position you take in relation to migration.

That’s something I miss. I haven’t been able to read any feedback from projects here in the forum yet. For whatever reason.

1 Like

Ok, this is a fair feedback point.

I have to admit that NIP#8 got lesser attention than I/we imagined and I would have loved for more transparency in general migration decision/recommendation processes. But it is complicated already to have it work effectively in one organization ; so given we are now spreading knowledge / resources / cross-operating with different entities, it gets even more complicated.

Ultimately, the migration committee helped research pain points and implications / strategies of migrating distributed data ; The fact that we have a solution for it, that implies user’s consent, actually makes me proud. And the fact that this solution is a bit complicated to explain is a shame to put on distributed ledger technology, not on me, please! :smiley:

4 Likes

This scenario you are suggesting is still the best in my opinion:

  • All non-multisig accounts are migrated by default except for those that send an explicit opt-out request.
  • Multi-signature accounts that should be migrated must send an opt-in request.
  • Owners of namespaces that should be migrated must send an opt-in request.

Most people will be included in the automatic migration (non-multisig accounts). And all transactions will be verifiable on the catapult chain.

I would suggest get the launch date fixed ASAP and start informing people what they can or must do!

4 Likes

When multisig account migrates to Catapult, should multisig account signs multisig modify transaction with own private key?

One of the cosignatories needs to input it yes, thats the cosignatory that will initiate the process ;

  1. Dave account has 2of3 multisig with Alice, Bob and Carol cosignatories
  2. one of the cosignatories (lets say Alice) should initiate the process by sending a SignalOptinDTO and select whether the multisig on Catapult will be Dave or a new account.
    2.1) That same cosignatory (Alice) will send a ConvertSignalDTO which contains a catapult transaction that converts Dave or new account on catapult.
    2.2) Alice then also sends a CosigOptinDTO that contains Alice’s cosignature for the catapult transaction in 2.1
  3. Bob sends a CosigOptinDTO that contains Bob’s cosignature for the catapult transaction in 2.1
  4. Carol sends a CosigOptinDTO that contains Carol’s cosignature for the catapult transaction in 2.1

Noteable is that the step 2) will be ONE STEP for the enduser (not multiple steps …)

If the multisig private key is lost, it is possible to change the “destination account” so that the newly setup multisig uses a different private key.

1 Like

Thanks @gevs for this breakdown.
I, 2nd the importance of this point.
Point 1 - All transactions must be verifiable on both chain.
1.1 So the integrity of the transactions are maintained.
1.2 If ever a dispute arises regarding transactions, then logic, accounting principle and maths can be used to resolve the dispute, which should hold up in a court of law.
1.3 The fundamentals of the blockchain are upheld.

1 Like

This would be my preference, it would be a good balance between the various strategies. How would that work with exchanges? not sure about their backend but I am assuming most of them will have multi-sig in place and it will be an opt-in followed by redistribution to their users.

This would also need to be decided with care as the timing could affect price movements and how exchanges might respond to this ‘opportunity’ or ‘risk’ based on how they see it.

2 Likes

Thanks for your great questions!

We are producing a NPM library that can be integrated in many different ways. For instance, I have added a experimental implementation of the opt-in creation and collection on github and we are working on one for the NanoWallet. Knowing that most exchanges will not be using NanoWallet, it was important to make the opt-in process transparent and reproducible. As such, it is totally feasible for exchanges to opt-in without Nanowallet ;

I will be bringing up the exchanges topic on the Migration Committee today. More details need to be defined on that topic.

Expect a return in the next day :+1:

1 Like

The explicit opt-in for non multi-sig transactions seems to make sense. Could there any potential for load issues into the new chain? For existing partners would there be any issues with multiple counterparties having to agree given that the multi-sig transactions are automatically opt-in?

Sorry for coming back so late @jason.lee

Exchanges may be taking different steps and we are working on documentation on that topic. More details about the active discussions and actionable items for the Migration Committee can be found in the latest Migration Committee Community Update #4.

Could there any potential for load issues into the new chain?

You probably mean whether it would be possible to bloat the nemesis block? Since this is the first block of the network, it would be possible to make it accept a bigger amount of transactions not being affected by the following network configuration.