Symbol Launch - General Status Update (01-Feb-2021)

This is a general post to provide some information for the current state of the testing, delivery items expected this week and some discussions that will be started. I’ve seen a few questions in Telegram, Twitter and the Forum so have tried to pull things together in one place. Let me know if anything is missing.

The first one: When Snapshot? still has the same answer so is pointless asking - after the NGL Stress Test is complete, see below for steps to get to that point

A reminder, the process from here is:

  • NEMTus Testing event report assessed
  • An issues resolved and patch release provided
  • NGL Stress Test (will take 2-3 days once it starts)
  • A revised launch and snapshot date can be planned (not before mid Feb due to Testnet reset date)

Testnet Progress

The Testnet was reset in mid January and has been under various loads ever since, these include, to my knowledge:

  • XEMBook running a load test in the first few days
  • NEMTus tool testing for a few days
  • NEMTus tool was released and community appear to have been trial running it
  • NEMTus formal test event (both the NEMTus private/internal one and the Community one)
  • A NEM Hub test from DrCryptos

This has resulted in a Testnet that is now quite heavy in transactions and date, with a variety of community nodes, transactions, NGL nodes etc and is a more representative Mainnet like scenario.

As most people are aware, the NEMTus Community TestFest occurred last week. The testing was very well attended, I saw ~500 people on the youtube live stream alone.

The testing ran for 3-3.5 hours and maintained a strong load all the way through. The network held up well and we are awaiting NEMTus’ write up from the participants who submitted reports during/after the testing.

One issue was found a day before (the “Daoka-Canon” as it has been nicknamed) and reliably reproduced at the end of the stress test. This relates to Github issue 148 which has since been fixed and is now in testing, that testing shoujld conclude later today.

Once we receive the final test report from NEMTus (I believe it will be published publicly at the same time) then we can assess if anything else is an issue or not.

Next Release(s)

We have prepared a patch release, primarily for Core Server, REST and SDKs which includes the issue above and several others. This has passed internal manual testing and is going through automated regression testing which should finish later today/tomorrow morning.

We expect to issue this patch release early this week and it puts in place the platform to resolve Multi-Sig signing and Delegated Harvesting issues in the Wallets.

We may or may not issue a further patch release before the NGL Stress Test, this won’t be known until the NEMTus report is reviewed fully.

The Wallets will also have a release this week, but it will be later in the week - we want to get the Core + REST + SDK fixes out and validate behaviour on the wallet before releasing. This release is expected to include all knowns P1 and most P2 issues on the desktop wallet, a further wallet release may happen before launch depending on what is left outstanding.

Discussions/Information

We have various conversations/calls for input/information posts coming this week, these are just being reviewed for correctness and will be posted through the week, they include things like:

  • The process by which the mainnet Genesis/Nemesis block generation and deployment is being tested
  • A new way to validate/review your opted in balances on a copy of Mainnet
  • Discussion about early block rewards and harvesting
  • How to deploy Supernodes, Early Node Bonus and register VRF/Voting/Remote keys ahead of launch

And no doubt a few more as we go through the process above.

21 Likes

Thank You for clear info.

6 Likes

We posted technical issues from Daoka on nem2slack a few days ago.
And, we have just released the report in Japanese and made announcements. The English version is currently being prepared, so please wait for a while.

4 Likes

Thanks @h-gocchi it looks like these were four posts on Slack, from the report I don’t see anything extra either which is good news.
(EDIT: confirmed in private conversation, question removed)

Looking through the auto-translate of the report, some headline number for people which might be interesting on this thread, this is machine translate so may not quite 100% accurate but seems close:

  • Number of addresses that participated in the load tool: 667

  • Number of target nodes: 461

  • Total number of requests during the stress test:
    7,542,827 (18:00 - 22:00)
    7,249,639 (19:00 - 22:00)

  • Number of Tx captured during the stress test: 1,751,022

  • Average number of Tx per block: 4025.3

  • Block fulfillment rate: 67.1

  • Average performance per second: 134.2 TPS

Translated with www.DeepL.com/Translator (free version)

10 Likes

Thank you DaveH, as always cute, accurate and clear :slight_smile:

6 Likes

Thanks @h-gocchi and the NEMTus team for the English version

6 Likes

I’snt the unconfirmed transaction cache mentioned there the same issue that was found in december?

Your welcom.
We hope this will reach you all.

3 Likes

@DaveH and team. Great with keeping up with the updates.
No vague loosened sentences.
Clean update.
Thanks and keep it up !

4 Likes

No they are from the fix for that issue:

  • The issue was that there was no cap on the UT cache
  • The fix was to allow it to fill up, and then reject further transactions when it was full

So the messages received saying it was full are correct behaviour and allow the network to protect itself from that situation occurring, effectively by throttling until the cache has processes some transactions

3 Likes

Thanks got it. I guess that right now is the issue that should be fixed with the patch you mentioned in the statement above that should come today.

Can we excpect this to happen soon as well as the full NEMTus report is out now?

2 Likes

Does the fork mean that snapshot should be atleast after the fork will run 1 month?

It is just an upgrade so no not necessarily, that will be driven by how testing goes and any issues, it is not a reset and Testnet should continue on regardless of the fork.

Having to perform another fork is also a useful process to (re)test prior launch, it will definitely happen on Mainnet and last time the execution caused some issues on Testnet and was learned from. So it is a useful side benefit that it will occur again on Testnet. Not why it will happen, that is a technical change to fix the issue, but a useful benefit never the less.

6 Likes

Sorry for much questions, but will ther be another update this week or not?

Curios about some things from the update.
There should be NEMTus testing this week but i don’t see an announcment from them, is this one a open or a closed testing? (nvm, already saw daoka announcing to test it today)
Also it looks like no one still have assigned those 2 issues found recently.

hi why it takes so slow? i mean why??? nextweek will be 2ndweek of feb if there’s still no snapshot date maybe you dont have a plan to launch it?, always next week nextweek,

Seriously ? @DaveH has continuously and repeatedly offered comprehensive updates on the steps being taken to make the launch a success. I suggest you read what he posts.

6 Likes

I saw Twitter posts to confirm that the new release has passed the “Daoka-Cannon” retest earlier today.

The issues have been submitted for testing already (we don’t actively use the assigned field in Github at present) it will take some time to run through the full testing etc

1 Like

Additional Update (04-Feb-2021)

An additional update on the above, doesn’t really warrant a full post:

  • Symbol 0.10.0.6 has now been released

  • 0.10.0.6 been applied to a majority of nodes on the Testnet, a few community ones still remain on the old version but will no doubt catch up over next few days

  • Two remaining issues (#151 and #152) are in testing/bug fix at present, will take a few days for results

  • NGL will perform INTERIM stress tests on the Testnet in the next 24-48 hours, to ensure the recent release still passes the tests it passed previously. This will be a 400tps test and will run for 12-24 hours, it is likely to start in the next few hours and you may notice Testnet is under heavy load as a result for the rest of the week

  • The final test will not be until after the next release (anticipated release will be early next week subject to testing over the weekend)

  • A Desktop & Mobile Wallet + CLI release is being prepared and undergoing final testing with a view to releasing tomorrow or Saturday

  • The team are also going to work on cleaning up the repos in terms of closing out issues, reviewing PRs and triaging problems/labelling, we expect to complete this next week to help make visibility easier. I will also communicate a test and triage approach so the community can see how things are working and why issues do/don’t get picked up in Github, will take me a few days to write up though

18 Likes

Thanks for clear update!

4 Likes