Hi All,
I wanted to make a comment on a few things we have found since launch, that have caused confusion/frustration and where we think the root cause has come from, and what has been done to address it. It is all configuration related, not code or bug related - which is good, because it means it can and is being resolved
There are two specific issues:
-
Network Fees - happened due to a misconfigured default that was applied to most nodes and all NGL ones, it has been fixed and should rectify over the coming hours. Community owners should check their
transactionSelectionStrategy -
Finality Display - happened due to a misconfigured default on non-voting nodes, this has been fixed and community node owners should check their
unfinalisedBlockDuration
Sorry for this slip up, it was a genuine mistake which has been fixed, see below - the community actually noticed and responded to protect the network before NGL could and showed the power of decentralisation in action.
In the scheme of a public chain launch, if a day or two of high fees that you can accept or reject and a strange graph on block explorer from non-voting nodes is all the main problems that happened - it is still quite smooth. But, this bit clearly could have been smoother.
Network Fees
The network was deployed with a couple of defaults set, these have cause an issue because they were present in Bootstrap as a default by mistake, and as a result initial on most community nodes and all NGL nodes, so as the network got going, the fees started rising. This was a mistake and is being fixed.
defaultDynamicFeeMultipler: 100
minFeeMultiplier: 100
transactionSelectionStrategy: maximize
The defaultDynamicFeeMultiplier is a network wide setting and is used to help work out the fee across a group of blocks where the group contains empty blocks - basically if its empty, the default is applied, it it is not then minFeeMultiplier from the block producing node is applied.
The minFeeMultiplier is a node setting and can be altered by node operators. It is set to the same same as the dynamic fee by default but owners can tune it to include different types of transactions when they a block is harvested on their node
The transactionSelectionStrategy can be set to minimize, maximize or oldest. The intention was to set it to oldest, unfortunately it was rolled out with maximize, which means all nodes, including NGL ones were looking to pick the highest fee transactions - coupled with the multiplier, this increased the fees.
So what we have set now on NGL nodes is as below, this was rolled out slow over the past 24 hours to avoid disrupting finalisation :
transactionSelectionStrategy: oldest
We are looking at appling a range of values for minFeeMultiplier from 10 to 90, so that we provide a backbone of nodes that will pick up lower value fees and process them, also provide a more heterogenous configuration across the network.
A Symbol Bootstrap release is being prepared which will be released tomorrow and rolled across our nodes afterwards - again slowly to avoid disrupting the network.
Something very encouraging happened yesterday on this section - a private group of node owners realised what was happening and reacted collectively to set their multipliers low and transaction selection strategy to oldest. They reacted quicker than NGL could (we have to be careful with finality and were recovering from launch) and use the market dynamics of the chain to counter the fees until we cold resolve our issue. Thankyou especially to @meyns who was very quick to spot this and advise others how to help. This shows the power of a distributed set of owners and the market dynamics of those settings.
It should not have been necessary and we apologies unreservedly for the slip up, it was unintentional. The fees collected by NGL in the first 48 hours were ALWAYS going to go into the supernode fund anyway, so NGL do not benefit from this mistake in anyway either.
Finality
Early on the first day of operation, some of you may have noticed that it appeared a lot of nodes were stuck at block 1 for finality for a long time. They weren’t - finality was trucking along perfectly fine.
What happened was that the only nodes that have all the information on finality rounds are those in the voting group at the time, the rest wait for proofs. Those proofs get picked up/noticed by the other nodes based on a setting called unfinalisedBlockDuration.
The setting in the defaults was set to 0 - because a lot of our nodes are voting nodes, for a voting node this MUST be set 0. But for a non-voting node, 0 means dont check the proofs until the end of the Finality Epoch (1440 blocks, 12 hours) so all that was happening was that the non voting nodes couldnt “see” finality until the Epochs ended.
This should have been set at 5m or 10m. Because it was the default it affected most community nodes and all of our non-voting ones, so Block Explorer looked like there was a problem.
We have changed the setting on all our nodes and advised a couple of groups or node holders to do likewise, it has also been changed in the default settings and will be included in the Bootstrap release tomorrow.