NIS Synchronization Problems

Hey all. I've been running my node just fine since beta launch until I think maybe sometime yesterday. It seems my client got stuck at block 6980 and was not syncing from there on. I rebooted my NIS and local node and that solved the problem, but I'm wondering why it stopped in the first place.

Also, interestingly I tried to send a couple of transactions while I was not synchronized and after I synchronized I saw them as confirmed with like 60 confirmations so it seems that they were sent even though I was not synchronized on the blockchain. Is that intended?

I briefly looked through my NIS logs (speaking of which, I have about 5 or 6 of them and they are each nearly 100 MB each… something might need to be done about that) and the only thing I could see that might be the problem was the following:

10 24, 2014 5:36:43 &#21320;&#21069; org.nem.core.async.NemAsyncTimerVisitor b<br />&#35686;&#21578;: Timer CHECKING CHAIN SYNCHRONIZATION raised exception: org.nem.core.connect.FatalPeerException: java.net.ConnectException: Connection refused: no further information<br />java.util.concurrent.CompletionException: org.nem.core.connect.FatalPeerException: java.net.ConnectException: Connection refused: no further information<br /> at java.util.concurrent.CompletableFuture.internalComplete(Unknown Source)<br /> at java.util.concurrent.CompletableFuture$ThenApply.run(Unknown Source)<br /> at java.util.concurrent.CompletableFuture.postComplete(Unknown Source)<br /> at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source)<br /> at org.nem.core.connect.HttpMethodClient$a.failed(HttpMethodClient.java:202)<br /> at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:130)<br /> at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.connectionRequestFailed(DefaultClientExchangeHandlerImpl.java:308)<br /> at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.access$100(DefaultClientExchangeHandlerImpl.java:63)<br /> at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl$1.failed(DefaultClientExchangeHandlerImpl.java:351)<br /> at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:130)<br /> at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager$InternalPoolEntryCallback.failed(PoolingNHttpClientConnectionManager.java:471)<br /> at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:130)<br /> at org.apache.http.nio.pool.RouteSpecificPool.failed(RouteSpecificPool.java:161)<br /> at org.apache.http.nio.pool.AbstractNIOConnPool.requestFailed(AbstractNIOConnPool.java:527)<br /> at org.apache.http.nio.pool.AbstractNIOConnPool$InternalSessionRequestCallback.failed(AbstractNIOConnPool.java:746)<br /> at org.apache.http.impl.nio.reactor.SessionRequestImpl.failed(SessionRequestImpl.java:154)<br /> at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvent(DefaultConnectingIOReactor.java:175)<br /> at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:147)<br /> at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:348)<br /> at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:189)<br /> at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase.doExecute(CloseableHttpAsyncClientBase.java:67)<br /> at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase.access$000(CloseableHttpAsyncClientBase.java:38)<br /> at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:57)<br /> at java.lang.Thread.run(Unknown Source)<br />Caused by: org.nem.core.connect.FatalPeerException: java.net.ConnectException: Connection refused: no further information<br /> at org.nem.core.connect.HttpMethodClient$a.failed(HttpMethodClient.java:199)<br /> ... 19 more<br />Caused by: java.net.ConnectException: Connection refused: no further information<br /> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)<br /> at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)<br /> at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvent(DefaultConnectingIOReactor.java:173)<br /> ... 7 more

Any ideas?


Hey all. I've been running my node just fine since beta launch until I think maybe sometime yesterday. It seems my client got stuck at block 6980 and was not syncing from there on. I rebooted my NIS and local node and that solved the problem, but I'm wondering why it stopped in the first place.

Also, interestingly I tried to send a couple of transactions while I was not synchronized and after I synchronized I saw them as confirmed with like 60 confirmations so it seems that they were sent even though I was not synchronized on the blockchain. Is that intended?

I briefly looked through my NIS logs (speaking of which, I have about 5 or 6 of them and they are each nearly 100 MB each... something might need to be done about that) and the only thing I could see that might be the problem was the following:

Any ideas?


Can you upload your log somewhere?


Hey all. I've been running my node just fine since beta launch until I think maybe sometime yesterday. It seems my client got stuck at block 6980 and was not syncing from there on. I rebooted my NIS and local node and that solved the problem, but I'm wondering why it stopped in the first place.

Also, interestingly I tried to send a couple of transactions while I was not synchronized and after I synchronized I saw them as confirmed with like 60 confirmations so it seems that they were sent even though I was not synchronized on the blockchain. Is that intended?

I briefly looked through my NIS logs (speaking of which, I have about 5 or 6 of them and they are each nearly 100 MB each... something might need to be done about that) and the only thing I could see that might be the problem was the following:

Any ideas?


Can you upload your log somewhere?


Yep I will zip them all up and upload them just give me a second. :)

On BTT makoto said he also had the same issue so it is not just me. Hopefully the logs will shed some light on things.

Logs here: https://mega.co.nz/#!qcEDEKjJ!PIDq3bZSp_w08Ohyic0QNbVUD46aSmQShn47SJAzZBo

Let me know if that is everything you need! :slight_smile:

Now I had that same exact problem.  I thought it was just my client, but you also had a problem at exactly 6980.  That can't be a coincidence then.  Here is where I reported it.  https://forum.ournem.com/index.php?topic=2546.msg7491#msg7491

@amytheplanarshift: The "connection refused" exception is normal in the sense that every node will get those exceptions. I usually means that a remote node has its port 7890 closed. Transactions can be send even though you are not synchronized. Those transaction just have to be valid in the full chain.
I will take a look at your logs a little later.


Hey all. I've been running my node just fine since beta launch until I think maybe sometime yesterday. It seems my client got stuck at block 6980 and was not syncing from there on. I rebooted my NIS and local node and that solved the problem, but I'm wondering why it stopped in the first place.


Same happened to me.

I thought it was me messing around with remote harvesting.

Tell me if you need my logs too.

The issue has something to with remote harvesting. Your node, Amy, did not accept an importance transfer transaction at height 6981. That is why it got stuck at 6980.
But I haven't found the cause yet.

My Synchonization is extremely slow at 9100 to 9177 and still syncing now after 2 hours. Any reason?

This is what I have done. Boot NEM from webstart. Shut down NCC and left NEM monitor in tray together with NIS running, fired up standalone NCC. It was running ok for a few days. NEM monitor shows NCC running, all is fine.

Then I decided to shut down NIS from the Webstart version and fired up NIS standalone. Now, it is so slow. I have yet to kill everything and reboot.

Just reporting something I observed.


Edit: It appears that it is fully synced! Except that it says it is "one day behind" which in fact is synchronized!! WHy should this display when it should be synchronized?

I am on block 9208 now and have not had any issues since that last hiccup. Running smooth for me as far as I can tell. I'll try to make a couple of transactions and see how it goes, one second.

EDIT: I just sent 5 quick transactions within 30 seconds and all were confirmed just fine in block 9210. Seems to be no issue here at the moment.


My Synchonization is extremely slow at 9100 to 9177 and still syncing now after 2 hours. Any reason?

This is what I have done. Boot NEM from webstart. Shut down NCC and left NEM monitor in tray together with NIS running, fired up standalone NCC. It was running ok for a few days. NEM monitor shows NCC running, all is fine.

Then I decided to shut down NIS from the Webstart version and fired up NIS standalone. Now, it is so slow. I have yet to kill everything and reboot.

Just reporting something I observed.


Edit: It appears that it is fully synced! Except that it says it is "one day behind" which in fact is synchronized!! WHy should this display when it should be synchronized?


Can you supply the NIS log?


My Synchonization is extremely slow at 9100 to 9177 and still syncing now after 2 hours. Any reason?

This is what I have done. Boot NEM from webstart. Shut down NCC and left NEM monitor in tray together with NIS running, fired up standalone NCC. It was running ok for a few days. NEM monitor shows NCC running, all is fine.

Then I decided to shut down NIS from the Webstart version and fired up NIS standalone. Now, it is so slow. I have yet to kill everything and reboot.

Just reporting something I observed.


Edit: It appears that it is fully synced! Except that it says it is "one day behind" which in fact is synchronized!! WHy should this display when it should be synchronized?


Can you supply the NIS log?


pm you.

Seems the network is on an unresolvable fork.


Seems the network is on an unresolvable fork.


I saw this problem now at morning.

I let the bot sending money every 10 seconds and now I noticed I got somme blocks and there was no fee. That means I have not  got even my own fee send by bot and importance got higher from 1,09 to 1.29.

main screen
[img width=800 height=435]http://s22.postimg.org/npk664ti9/Nem_tela_inicial.png[/img]

3 seconds after main screen screen shot

[img width=800 height=434]http://s1.postimg.org/kteqqpp0f/blocos_minerados.png[/img]

Notice the harvested blocks are ahead the sync on main screen.

Up

Edit: I complete restarted my node. Second chain.bob.ninja Im in the main branch now.
I just founded weird information about my acc:

[img width=800 height=370]http://s10.postimg.org/gpurp1ix5/conta.png[/img]

@Parana: This could also be a slow reaction of the web interface. I experienced this once but after some seconds it was all good.


@Parana: This could also be a slow reaction of the web interface. I experienced this once but after some seconds it was all good.


No. thas not it.

My harvested blocks should have at least my bot's transactions fee( every 10 seconds I had one). That means My harvested blocks should have at least 15 nem as fee. The chain.bob.ninja showd last blocks as 0 transactions while my bot was running.

I got a branch chain.

I let the bot whith 20 secs transaction time. The notebook was toasting  ;D
;)

looks like restarting node make everything ok.

ethereum has a strange schemme for mining blocks.

Every time someone finds a block the network restart the mining operations.

Im not so sure how we do, but this seems interesting.

Everyone sync the utc time, start mining operations and as a bloc is found, it goes loop it all again.

Just an idea.

:wink:


tons of this information in NIS log took my VPS 10G disk space, completely halted my VPS.
Why is NIS taking so many open files?

2014-10-30 18:55:06.255:WARN:oejs.ServerConnector:qtp2006112337-18-acceptor-0-ServerConnector@13d9a8d3{HTTP/1.1}{0.0.0.0:7890}:
java.io.IOException: Too many open files
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:241)
        at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:336)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:467)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
        at java.lang.Thread.run(Thread.java:745)


Too many open files may mean too many open connections.
Is it a Linux box? If so what is the output of
[code]
ulimit -n
[/code]
If you run it with same user NIS runs?

I also have trouble syncing on ubuntu.

lsof -p PROCESS_ID gives me hundreds of TCP connections with CLOSE_WAIT status.

I'm the only dev to get this error so far, so we are trying to find out what is causing this.



I also have trouble syncing on ubuntu.

lsof -p PROCESS_ID gives me hundreds of TCP connections with CLOSE_WAIT status.

I'm the only dev to get this error so far, so we are trying to find out what is causing this.


Thanks for the update.
And I want to know how my non-graphical Ubuntu VPS can use http://chain.nem.ninja/checknis/ for a check, please help me with that.


Are you using digital ocean?
Type your IP like this: http://chain.nem.ninja/checknis/?ip=123.456.789.012