Symbol Testnet Reset & Release (14-Jan-2021)

Announcements
#1

All,

With huge thanks to the all the development and test teams who have been working night and day for weeks now; we are able to make the announcement below.

All the issues from testnet have now been resolved and various patches, config and deployment improvements have been made, we are ready for release and Testnet reset.

Shortly we will be performing the below (target today):

  • Full Server release including new Bootstrap and network configs

  • Public Testnet reset

After that:

  • Commmunity nodes on Testnet

  • Community testing, and particularly NEMTus testing

  • Further stress tests with the nodes distributed

The exact timing will depend on various build jobs and validations but likely to be late in the day UTC time today (Fri morning Japan time).

An announcement with full instructions will be made when it is complete.

Server Release

A new version of Core-Server, Rest, SDKs, Bootstrap and various supporting components will be made, this contains all the patches and config fixes found from the issue resolution over the past 3-4 weeks, plus some that could not be applied on the old Testnet from before this issue occurred.

The new release also contains significant memory savings/optimisations both on the core server and the MongoDB. There are ongoing memory optimisation investigations as well which may or may not be incorporated in future patches depending on what is found.

Assuming no significant issues are found (P0 or P1 type issues) this will be the last major release prior to Mainnet and resolves all issues found in the last stress test. Some minor patches may be made, with full testing (see below).

A classification and resolution process will be written up so it is easy to communicate publicly. It will be posted soon to explain how the test team are have been approaching and will continue to approach issue classification and resolution between here and Mainnet.

Testnet Reset

It will be necessary to reset Testnet as communicated previously. There are several changes that cannot be cleanly applied to the old Testnet. Finality has also stalled due to a deployment issue during the soak test.

This reset gives us a clean environment with the Mainnet code that will be deployed at launch.

To perform this reset, we have been running a new 500 node network internally for ~1 week, which has had a significant number of transactions pushed into it (almost the same as the old Testnet) as part of the stress testing. This data is being left there on purpose (i.e. not reset to an empty chain) to ensure the remaining soak period is performed on a chain that has a substantial data in it.

That does mean it currently takes 3-5 hours to synchronise from zero. We are working on a quicker synchronisation approach via Bootstrap which will be available as soon as appropriately tested. It will download a copy of the chain and retrospectively validate/verify it, similar to the NIS1 approach that some will remember.

Short term it will be necessary to take a few hours to synchronise a fresh node and is an important test that we would like community nodes to perform.

@GodTanu we will try and fix the Link issue on bootstrap from the other thread in time for the release, but I cannot be certain until it is looked at

Testing Update

The Testnet has now had the following tests performed and passed:

  • Automation/Regression testing on each patch
  • 100tps soak test - passed - meets stated Mainnet target
  • 150tps soak test - passed - exceeded stated Mainnet target for 12 hours with no issues
  • 400tps soak test - passed - throttled at ~130-200tps and when load stopped, recovered
  • A final regression test was happening overnight on 2 minor patches, no issues are anticipated.

Next Steps

As communicated previously, the next steps are:

At that point we will then be able to plan a launch date, I am confirming in this message that launch will occur AT LEAST one month from the point of the release. It is depends heavily on the points above and no definitive date will be selected until they are complete and any issues that are raised have been assessed.

For the avoidance of doubt, I am therefore not confirming the launch is 1 month from today, I am saying it will not be less than that time and until we complete testing, the date will not be known.

34 Likes
NEM Symbol 0.10.0.5 Release & Testnet Reset (Updated 19/01/2021)
Symbol Testnet Update (11-Jan-2020)
#2

Thanks for the hard work of the team

Thanks most @Jaguar0625 @DaveH

Although I cannot help, I sincerely thank you for your contribution.

7 Likes
#3

It is very pleasant to read such a report, which does not contain empty words, but is filled with quality arguments! Thanks to the team for the work done, I hope such a report will add optimism to the supporters of the project, which has been so lacking lately! I am sure that the goals will be achieved and the project will be implemented! :slight_smile:

8 Likes
#4

Top prize goes to @leoinker for spotting that it is not the year 2020 anymore! (thankfully), fixed

2 Likes
#5

Nevertheless, it’s a bit unclear to me why the reference to the “Symbol Testnet Reset & Release” on the official NEM Twitter account is not directly linked to the post here in the forum.

#6

Well done team and keep the good work going. Good luck with testing!

Can we have more information about how long those tests will held? I mean NEMTus testing and NGL rerunnin stress tests.

1 Like
#7

Wait half-naked

2 Likes
#8

We are co-ordinating with NEMTus just now to see when their scheduling works now that they know when the release is available. Once we know we will update, it is likely to be something like NEMTus early-mid next week, then NGL stress tests afterwards - mid-end of the week, or early the week following. These are very approximate until we finish co-ordinating though. I will confirm more when we have the info.

2 Likes
#9

The default node of the desktop wallet is the node before Reset.
I want to select a node, but it shows up as an invalid node and cannot work.

Please update the NGL node or let me know if there is another workaround.
There is no CLI right now, so you have to use the SDK…

3 Likes
#10

A lot of nodes are running on older versions(v0.10.0).
Could you please review them as soon as possible.

3 Likes
#11

Now , It looks like most of the nodes are running on 0.10.0.5.
However, if I check in /node/server by REST API, I can see the following

{“serverInfo”:{“restVersion”:“2.2.2”,“sdkVersion”:“2.2.2”}}

The current latest version is 2.3.0, so symbol-sdk-0.23.0 does not work.
Are you sure they are v.0.10.0.5?

1 Like
#12

Hi @GodTanu thanks, just catching up, there was an issue with the release build that for some reason didn’t pick up those change, it will be fixed today:

2 Likes
#13

@TakaNobu

Checking the website: https://symbolnodes.org/nodes_testnet they look to be mostly updated now (exception of some community ones as to be expected). I suspect it may have taken a moment to update

Can you give me one of the links that says 2.2.2 please? The ones I checked at random look like below:

{“serverInfo”:{“restVersion”:“2.3.0”,“sdkVersion”:“2.3.0”}}

That is from:

It may just have taken some time for caches to clear etc somewhere or you may be looking at another node(s) I haven’t checked manually, just want to validate

EDIT:
I saw a conversation on Public slack after posting this, looks like the rolling upgrades took a little while to go through and some of the NGL nodes needed to be upgraded with the release from last night: https://nem2.slack.com/archives/C9E7N7H1N/p1610711520026900?thread_ts=1610692103.024500&cid=C9E7N7H1N

Should now be fixed on the core nodes, the others will be updated shortly

1 Like
#14

I’ve tried a lot of things, and I’ve learned something.

Node synchronization time is long (took 8 hours)
The quality of the desktop wallet is poor.
-The screen display is not real time.
-There are problems such as frequent timeouts.
-Unable to set fees (always times out if other than recommended).

-“Invalid node” is displayed when changing nodes.
In other words, the operation is unstable.

#15

I changed the connection of my desktop wallet to my node. It showed “invalid node” several times, but after many iterations I was able to change it.

Then I had to recreate the profile.

Many of the same problems occur when the default node is not changed. I would like to see this response given a higher priority.

1 Like
#16

Thanks @GodTanu:

Synchronisation, yes this is taking longer than it did yesterday. It is due to the number of nodes synchronising and amount of data. We are monitoring it and once the nodes all come online will rerun the synchronisations to make sure it was just a settling in problem, or not.

Desktop wallet, is struggling a little bit because of the network load while the synchronisations are happening some of the nodes are working more slowly due to the load.

I would stop testing the Wallet until the new build is released. We know the build has problems and will be frustrating. The new build is likely to be ready and tested in a few hours so should be there by the time you wake up.

3 Likes
#17

Better check it out before releasing it.
If you are a professional.

3 Likes
#18

Further to my reply above. An update is available on the main release thread:

On the wallet specifically:

The quality of the desktop wallet is poor.
-The screen display is not real time.
-There are problems such as frequent timeouts.
-Unable to set fees (always times out if other than recommended).

The first two issues in appear to be linked to this issue reported by Xembook which were not occurring during our testing and have only appeared since public release.

The last issue is a build issue and has been resolved.

We have taken the decision to not release the wallet build today. The timeouts/notifications etc lead to a poor user experience of the wallet and the problem is probably not the wallet directly given xembook is seeing similar without the wallet. Investigation into the underlying problem will take some more time and I will update as soon as possible.

I know this is disappointing but do not want to create further frustration or lost time by releasing it when we know it will show similar user experience as your workaround one.

I will update as soon as more information is known and want to assure people that this issue was not present in testing over the past 1 - 1.5 weeks it has shown as the network has had public nodes and upgrades applied.

#19

I assume no NEMTus testing will happen till the wallet is out? And we again in a point where there is no information about how long can this issue resolving take time?

#21

I will provide instructions on how to reproduce.

3 Likes