NIS node stopped syncing at 1690515 block with IllegalStateException warning

Technical Discussion
#1

I’ve tried setting up a node using the below docker images criptalia/nem-node and rb2nem/nem-docker but its stuck at this block height 1690515 and in logs I can see the below warning,

2021-03-15 09:27:02.390 INFO synchronizing with Node [Hi, I am Huge Alice 2 ] @ [176.9.68.110] (org.nem.peer.services.NodeSynchronizer b)
2021-03-15 09:27:02.536 WARNING Timer SYNC raised exception: java.lang.IllegalStateException: Request cannot be executed; I/O reactor status: STOPPED
java.util.concurrent.CompletionException: java.lang.IllegalStateException: Request cannot be executed; I/O reactor status: STOPPED
at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273)
at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280)
at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1629)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: Request cannot be executed; I/O reactor status: STOPPED
at org.apache.http.util.Asserts.check(Asserts.java:46)
at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase.ensureRunning(CloseableHttpAsyncClientBase.java:90)
at org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(InternalHttpAsyncClient.java:123)
at org.apache.http.impl.nio.client.CloseableHttpAsyncClient.execute(CloseableHttpAsyncClient.java:75)
at org.apache.http.impl.nio.client.CloseableHttpAsyncClient.execute(CloseableHttpAsyncClient.java:108)
at org.apache.http.impl.nio.client.CloseableHttpAsyncClient.execute(CloseableHttpAsyncClient.java:92)
at org.nem.core.connect.HttpMethodClient.sendRequest(HttpMethodClient.java:118)
at org.nem.core.connect.HttpMethodClient.post(HttpMethodClient.java:89)
at org.nem.peer.connect.HttpCommunicator.a(HttpCommunicator.java:37)
at org.nem.nis.connect.AuditedCommunicator.a(AuditedCommunicator.java:31)
at org.nem.nis.sz.uht.a(HttpConnector.java:145)
at org.nem.nis.sz.uht.a(HttpConnector.java:157)
at org.nem.nis.sz.uht.i(HttpConnector.java:110)
at org.nem.nis.pz.mtf.fG(RemoteBlockLookupAdapter.java:29)
at org.nem.nis.sync.BlockChainComparer$Impl.fr(BlockChainComparer.java:94)
at org.nem.nis.sync.BlockChainComparer$Impl.fp(BlockChainComparer.java:60)
at org.nem.nis.sync.BlockChainComparer.a(BlockChainComparer.java:32)
at org.nem.nis.sync.BlockChainUpdater.a(BlockChainUpdater.java:148)
at org.nem.nis.sync.BlockChainUpdater.c(BlockChainUpdater.java:83)
at org.nem.nis.BlockChain.b(BlockChain.java:112)
at org.nem.nis.BlockChain.a(BlockChain.java:104)
at org.nem.nis.tae.a(CountingBlockSynchronizer.java:39)
at org.nem.peer.services.NodeSynchronizer.b(NodeSynchronizer.java:50)
at org.nem.peer.PeerNetwork.gt(PeerNetwork.java:141)
at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626)
… 3 more
(org.nem.core.async.NemAsyncTimerVisitor notifyOperationCompleteExceptionally)

Heartbeat of node is, {“code”: 1, “type”: 2, “message”: “ok”}
Status of node is, {“code”:5,“type”:4,“message”:“status”}

Can anyone help me with this?

#2

I guess it was out of memory. Try increasing -Xmx above 4GB

#3

Thanks! By increasing the memory limit it crossed 1690515 and running.

#4

Again node is stuck at 1704911 block with the same warning. Now mem-limit is also increased.
Anyone have any idea?

Heartbeat response:

{
    "code": 1,
    "type": 2,
    "message": "ok"
}

Status response:

{
“code”: 5,
“type”: 4,
“message”: “status”
}

Logs:

2021-03-16 07:32:59.943 INFO synchronizing with Node [Hi, I am Alice7 ] @ [104.238.161.61] (org.nem.peer.services.NodeSynchronizer b)
2021-03-16 07:33:00.148 WARNING Timer SYNC raised exception: java.lang.IllegalStateException: Request cannot be executed; I/O reactor status: STOPPED
java.util.concurrent.CompletionException: java.lang.IllegalStateException: Request cannot be executed; I/O reactor status: STOPPED
at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273)
at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280)
at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1629)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: Request cannot be executed; I/O reactor status: STOPPED
at org.apache.http.util.Asserts.check(Asserts.java:46)
at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase.ensureRunning(CloseableHttpAsyncClientBase.java:90)
at org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(InternalHttpAsyncClient.java:123)
at org.apache.http.impl.nio.client.CloseableHttpAsyncClient.execute(CloseableHttpAsyncClient.java:75)
at org.apache.http.impl.nio.client.CloseableHttpAsyncClient.execute(CloseableHttpAsyncClient.java:108)
at org.apache.http.impl.nio.client.CloseableHttpAsyncClient.execute(CloseableHttpAsyncClient.java:92)
at org.nem.core.connect.HttpMethodClient.sendRequest(HttpMethodClient.java:118)
at org.nem.core.connect.HttpMethodClient.post(HttpMethodClient.java:89)
at org.nem.peer.connect.HttpCommunicator.a(HttpCommunicator.java:37)
at org.nem.nis.connect.AuditedCommunicator.a(AuditedCommunicator.java:31)
at org.nem.nis.sz.uht.a(HttpConnector.java:145)
at org.nem.nis.sz.uht.a(HttpConnector.java:157)
at org.nem.nis.sz.uht.i(HttpConnector.java:110)
at org.nem.nis.pz.mtf.fG(RemoteBlockLookupAdapter.java:29)
at org.nem.nis.sync.BlockChainComparer$Impl.fr(BlockChainComparer.java:94)
at org.nem.nis.sync.BlockChainComparer$Impl.fp(BlockChainComparer.java:60)
at org.nem.nis.sync.BlockChainComparer.a(BlockChainComparer.java:32)
at org.nem.nis.sync.BlockChainUpdater.a(BlockChainUpdater.java:148)
at org.nem.nis.sync.BlockChainUpdater.c(BlockChainUpdater.java:83)
at org.nem.nis.BlockChain.b(BlockChain.java:112)
at org.nem.nis.BlockChain.a(BlockChain.java:104)
at org.nem.nis.tae.a(CountingBlockSynchronizer.java:39)
at org.nem.peer.services.NodeSynchronizer.b(NodeSynchronizer.java:50)
at org.nem.peer.PeerNetwork.gt(PeerNetwork.java:141)
at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626)
… 3 more
(org.nem.core.async.NemAsyncTimerVisitor notifyOperationCompleteExceptionally)
2021-03-16 07:33:02.900 WARNING Timer CHECKING CHAIN SYNCHRONIZATION raised exception during start: java.lang.IllegalStateException: Request cannot be executed; I/O reactor status: STOPPED (org.nem.core.async.AsyncTimer getNextChainLink)
2021-03-16 07:33:02.939 INFO pulling node experiences from Node [AQUALIFE2 ] @ [163.44.168.183] (org.nem.peer.services.NodeExperiencesUpdater a)
2021-03-16 07:33:02.939 WARNING Timer UPDATE NODE EXPERIENCES raised exception during start: java.lang.IllegalStateException: Request cannot be executed; I/O reactor status: STOPPED (org.nem.core.async.AsyncTimer getNextChainLink)

#5

How much memory allocated in start script? Maybe not enough

#6

mem_limit is configured as 6GB in the docker file. Need to increase it?

Here is the content of docker-compose.yaml,

version: ‘2’
services:
node-nem:
image: criptalia/nem-node
container_name: nemnode
restart: always
ports:
- “5332:7890”
- “7778:7778”
volumes:
- /ABC/node-nem:/root/nem
mem_limit: 6g

Result of ‘sudo du -h’ command is,

250M ./nis/logs
1.8G ./nis/data
2.0G ./nis
2.0G .

Result of ‘sudo docker stats’,

CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
-------------------- nemnode 0.10% 1.108GiB / 6GiB 18.47% 9.58MB / 654kB 2.03GB / 521MB 52

#7

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.