NIS node stops syncing intermittently

Tech Support
#1

I’ve tried setting up a node using the below docker images criptalia/nem-node and rb2nem/nem-docker but it stuck intermittently. After a restart, it works sometimes then again stops.

Below is the docker file,

version: ‘2’
services:
node-nem:
image: criptalia/nem-node
container_name: nemnode
restart: always
ports:

  • “5332:7890”
  • “7778:7778”
    volumes:
  • /ABC/node-nem:/root/nem
    mem_limit: 6g

Recent logs,

2021-03-17 07:52:51.773 SEVERE DIRECT fatal error encountered while communicating with <Node [[c=#86c0e9]jusan[/c] ] @ [jusan.nem.ninja]>: java.util.concurrent.CompletionException: org.nem.core.connect.FatalPeerException: org.apache.http.ConnectionClosedException: Connection closed (org.nem.peer.services.NodeRefresher a)
2021-03-17 07:52:51.773 INFO Updating “Node [[c=#86c0e9]jusan[/c] ] @ [jusan.nem.ninja]” → FAILURE (org.nem.peer.services.NodeRefresher update)
Exception in thread “I/O dispatcher 8” Exception in thread “I/O dispatcher 3” Exception in thread “I/O dispatcher 1” java.lang.OutOfMemoryError: Java heap space
**> **
> Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "I/O dispatcher 4"
at java.nio.HeapByteBuffer.(HeapByteBuffer.java:57)
at java.nio.ByteBuffer.allocate(ByteBuffer.java:335)
at org.apache.http.nio.util.HeapByteBufferAllocator.allocate(HeapByteBufferAllocator.java:52)
at org.apache.http.nio.util.ExpandableBuffer.(ExpandableBuffer.java:65)
at org.apache.http.impl.nio.reactor.SessionOutputBufferImpl.(SessionOutputBufferImpl.java:86)
at org.apache.http.impl.nio.NHttpConnectionBase.(NHttpConnectionBase.java:222)
at org.apache.http.impl.nio.DefaultNHttpClientConnection.(DefaultNHttpClientConnection.java:138)
at org.apache.http.impl.nio.conn.ManagedNHttpClientConnectionImpl.(ManagedNHttpClientConnectionImpl.java:76)
at org.apache.http.impl.nio.conn.ManagedNHttpClientConnectionFactory.create(ManagedNHttpClientConnectionFactory.java:105)
at org.apache.http.impl.nio.conn.ManagedNHttpClientConnectionFactory.create(ManagedNHttpClientConnectionFactory.java:56)
at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager$InternalConnectionFactory.create(PoolingNHttpClientConnectionManager.java:601)
at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager$InternalConnectionFactory.create(PoolingNHttpClientConnectionManager.java:571)
at org.apache.http.nio.pool.AbstractNIOConnPool.requestCompleted(AbstractNIOConnPool.java:513)
at org.apache.http.nio.pool.AbstractNIOConnPool$InternalSessionRequestCallback.completed(AbstractNIOConnPool.java:836)
at org.apache.http.impl.nio.reactor.SessionRequestImpl.completed(SessionRequestImpl.java:135)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processNewChannels(AbstractIOReactor.java:424)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:287)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
at java.lang.Thread.run(Thread.java:748)
2021-03-17 07:55:11.962:WARN:oejut.QueuedThreadPool:qtp1512759723-43: Unexpected thread death: org.eclipse.jetty.util.thread.QueuedThreadPool$3@70e2f570 in qtp1512759723{STARTED,8<=8<=500,i=1,q=0}
Exception in thread “I/O dispatcher 5” java.lang.OutOfMemoryError: Java heap space
Exception in thread “I/O dispatcher 7” java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
at java.util.Collections$UnmodifiableCollection.iterator(Collections.java:1038)
at org.apache.http.impl.nio.reactor.BaseIOReactor.validate(BaseIOReactor.java:212)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
at java.lang.Thread.run(Thread.java:748)
java.lang.OutOfMemoryError: Java heap space
at java.util.Collections$UnmodifiableCollection.iterator(Collections.java:1038)
at org.apache.http.impl.nio.reactor.BaseIOReactor.validate(BaseIOReactor.java:212)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
at java.lang.Thread.run(Thread.java:748)
Exception in thread “qtp1512759723-43” java.lang.OutOfMemoryError: Java heap space
2021-03-17 07:55:12.498 WARNING Timer UPDATE NODE EXPERIENCES raised exception: org.nem.core.connect.FatalPeerException: org.apache.http.ConnectionClosedException: Connection closed unexpectedly
java.util.concurrent.CompletionException: org.nem.core.connect.FatalPeerException: org.apache.http.ConnectionClosedException: Connection closed unexpectedly
at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:593)
at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
at org.nem.core.connect.HttpMethodClient$HttpMethodClientFutureCallback.failed(HttpMethodClient.java:201)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:134)
at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.executionFailed(DefaultClientExchangeHandlerImpl.java:101)
at org.apache.http.impl.nio.client.AbstractClientExchangeHandler.failed(AbstractClientExchangeHandler.java:426)
at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.closed(HttpAsyncRequestExecutor.java:140)
at org.apache.http.impl.nio.client.InternalIODispatch.onClosed(InternalIODispatch.java:71)
at org.apache.http.impl.nio.client.InternalIODispatch.onClosed(InternalIODispatch.java:39)
at org.apache.http.impl.nio.reactor.AbstractIODispatch.disconnected(AbstractIODispatch.java:100)
at org.apache.http.impl.nio.reactor.BaseIOReactor.sessionClosed(BaseIOReactor.java:279)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processClosedSessions(AbstractIOReactor.java:440)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.hardShutdown(AbstractIOReactor.java:576)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:305)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.nem.core.connect.FatalPeerException: org.apache.http.ConnectionClosedException: Connection closed unexpectedly
at org.nem.core.connect.HttpMethodClient$HttpMethodClientFutureCallback.wrapException(HttpMethodClient.java:210)
… 15 more
Caused by: org.apache.http.ConnectionClosedException: Connection closed unexpectedly
… 11 more
(org.nem.core.async.NemAsyncTimerVisitor notifyOperationCompleteExceptionally)
2021-03-17 08:06:13.030 WARNING forcibly aborting request to http://78.47.148.106:7890/node/sign-of-life (org.nem.core.connect.HttpMethodClient lambda$sendRequest$2)
2021-03-17 08:06:13.030 WARNING Timer BROADCAST raised exception: java.util.concurrent.CancellationException
> java.util.concurrent.CompletionException: java.util.concurrent.CancellationException
at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:593)
at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
at java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2265)
at org.nem.core.connect.HttpMethodClient$HttpMethodClientFutureCallback.cancelled(HttpMethodClient.java:216)
at org.apache.http.concurrent.BasicFuture.cancel(BasicFuture.java:150)
at org.apache.http.concurrent.BasicFuture.cancel(BasicFuture.java:157)
at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.executionCancelled(DefaultClientExchangeHandlerImpl.java:116)
at org.apache.http.impl.nio.client.AbstractClientExchangeHandler.connectionRequestCancelled(AbstractClientExchangeHandler.java:357)
at org.apache.http.impl.nio.client.AbstractClientExchangeHandler.access$200(AbstractClientExchangeHandler.java:62)
at org.apache.http.impl.nio.client.AbstractClientExchangeHandler$1.cancelled(AbstractClientExchangeHandler.java:397)
at org.apache.http.concurrent.BasicFuture.cancel(BasicFuture.java:150)
at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager$1.cancelled(PoolingNHttpClientConnectionManager.java:295)
at org.apache.http.concurrent.BasicFuture.cancel(BasicFuture.java:150)
at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager$2.cancel(PoolingNHttpClientConnectionManager.java:304)
at org.apache.http.impl.nio.client.AbstractClientExchangeHandler.cancel(AbstractClientExchangeHandler.java:443)
at org.apache.http.client.methods.AbstractExecutionAwareRequest.abort(AbstractExecutionAwareRequest.java:90)
at org.nem.core.connect.HttpMethodClient.lambda$sendRequest$2(HttpMethodClient.java:129)
at java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:656)
at java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:632)
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
at org.nem.core.async.SleepFuture$1.run(SleepFuture.java:24)
at java.util.TimerThread.mainLoop(Timer.java:555)
at java.util.TimerThread.run(Timer.java:505)
Caused by: java.util.concurrent.CancellationException
at java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2263)
… 21 more
(org.nem.core.async.NemAsyncTimerVisitor notifyOperationCompleteExceptionally)
2021-03-17 08:06:19.684 WARNING forcibly aborting request to http://108.61.182.27:7890/node/info (org.nem.core.connect.HttpMethodClient lambda$sendRequest$2)
2021-03-17 08:06:19.685 SEVERE DIRECT fatal error encountered while communicating with <Node [Hi, I am Alice5 ] @ [108.61.182.27]>: java.util.concurrent.CompletionException: java.util.concurrent.CancellationException (org.nem.peer.services.NodeRefresher a)
2021-03-17 08:06:19.685 INFO Updating “Node [Hi, I am Alice5 ] @ [108.61.182.27]” → FAILURE (org.nem.peer.services.NodeRefresher update)

sudo du -h,

304M ./nis/logs
1.8G ./nis/data
2.1G ./nis
2.1G .

sudo docker stats,

CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
602f3cdd5463 nemnode 0.12% 1.478GiB / 6GiB 24.63% 2.25MB / 124kB 1.96GB / 112MB 49

Anyone can help with this?

#3

Did you tried without docker?
I mean all you need is java 8 installed and change start script (increase Xmx). And start.

#4

No. I haven’t tried without docker.
Could you please ping me the link of installer for ubuntu?

Can we build our own docker image from the NIS repository?

Any idea in future are they going to create official docker image and provide support to it?

#5

Just extract and run: https://bob.nem.ninja/nis-0.6.97.tgz (java 8 required + add more memory to start script as Xmx)
Additionaly you have presynced database: https://bob.nem.ninja/nis5_mainnet.h2-2967k.db.zip (it’s located in /nem/nis

#6

Thank you!!

#7

Hi,

Can you please name the start script file?

#8

Linux - nix.runNis.sh
Windows - runNis.bat

#9

Thank You!

#10

Hi,

I can wee this in the start script,
java -Xms512M -Xmx1G -cp “.:./:…/libs/” org.nem.deploy.CommonStarter

So I need to increase memory for both Xms and Xmx?

what is the difference?

#11

For example

java -Xms4G -Xmx8G -XX:+UseG1GC -cp ".:./*:../libs/*" org.nem.deploy.CommonStarter &
#12

Thanks!

These sizes 4GB and 8GB are enough to sync with the latest block?

#13

And if HISTORICAL_ACCOUNT_DATA setting is on?

#14

If HISTORICAL_ACCOUNT_DATA you need 200G

#15

Thanks!

#16

Hi,

I’ve increased the memory limit but now the CPU hits 700-800% and then it stops syncing.
Any idea on CPU? How much is required? Can you please help so that I can assign it?

#17

I don’t have any problem syncing with my laptop (Intel® Core™ i5 8gen 8250U 1,6 - 3,4 GHz). Without historical data of couse.

#18

Oky. Thanks!