Introduction to local harvesting and monitoring
In the NEM community, mining is called harvesting. It comes in 2 forms, local harvesting and delegated harvesting. See how local and delegated harvesting works for details. In summary: with delegated harvesting another computer is harvesting on your behalf, and with local harvesting you are in own control (or so I assume) using your own computer.
I like local harvesting. The mining fee is lower than delegated harvesting, and it’s technically more challenging. And I also like it because I can say that I’m a “miner” too (o well, “harvester”). I personally don’t expect to make profit from it, being aware that local harvesting uses some energy too (in case of NEM is relatively little). But I still find it cool.
So it’s technically challenging, not only to set it up (which is still reasonably straightforward), but also to keep the local harvesting process (I mean the local NEM server called NIS) running. The network connection is in my case quite thin. Also, the NIS is using a local database which sometimes gets stuck. It can typically be my bad luck, but I mean really stuck, so that even a Windows restart doesn’t help. Then, only throwing away (or replacing) the database comes to the rescue.
When the NIS stops, the harvesting stops. Not cool. So, I came up with my own simple NIS monitor, written in Java, a process which keeps an eye on the NIS, reboots the Windows, and restarts the NIS, if necessary.
Use my notes to your own benefit, as well as at your own risk!
Setup of the NIS
To setup local harvesting, I paid a 0.15 fee via my NEM wallet (via “Manager delegated account”), and after hours, the harvesting panel showed a Remote status ACTIVE. I used my password to let my wallet reveal the delegated private key, copied it, and closed the wallet. Make sure it is the delegated private key which you copy!!! (Not your private wallet key, because your private key must be kept even more private than the delegated private key. The delegated private key should never leave your computer, but your wallet private key is an even larger secret!)
For installing the NIS, see How to activate and start delegated harvesting in the nanowallet for details (Local NIS and further). I used version 0.6.93. Once you have it running it’s worth to adjust a couple of settings, so stop the NIS.
Config.properties
In the nispackage, there is a nis folder containing the file config.properties. The part with the bootKey and bootName must approximately look as follows:
# NIS auto boot configuration
# CAUTION: Keep in mind that if someone will read key placed here, he might STEAL your XEM
# nis.bootKey: private key of an account that should be used to boot NIS
# nis.bootName: name of the NIS node (you can use anything you like)
nis.bootKey = bd45ab23de12db12fe6792ad56ebc12abd45ab23de12db12fe6792ad56ebc12a
nis.bootName = you can come up with any name
It is the bootKey where your copied delegated private key must go. (And no, I didn’t publish my real delegated private key.) See also NIS auto start and auto harvest for details. Furthermore, I changed the unlockedLimit setting to a lower number than the default 4, for example also 2 seems to work fine. Most default values I kept the same.
With this bootKey setting ready, the wallet can be kept closed while running the NIS. Harvesting should work fine.
Note that in the same nis folder, there are db.properties. I didn’t adjust any of these few values. But who knows, maybe some changes can prevent the rare database problem I was describing above.
runNis.bat
It’s worth to take a look at the content of the runNis.bat, which is used to start up the NIS, and change the content to:
pushd nis
java -Xms4G -Xmx8G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -cp ".;./*;../libs/*" org.nem.deploy.CommonStarter
popd
The changes are in the parameters directly after the “java”. If I remember correctly, changing the startup script was advised somewhere on the nem forum by a very helpful moderator called “BloodyRookie”, see for an example of his help (in another context) Adjusted java runtime settings
So I use more memory than the default runNis.bat, and an adjusted garbage collecting. I’m under the impression that these adjustments further stabilized the NIS; possibly it reduced memory/hard drive swaps.
NIS monitor (for Java programmers)
And yet, I was still not satisfied with the NIS stability. There had to be a way to keep the NIS running as much as possible, even after for example a Windows update, without the need of manually starting everything up again. If your Windows starts up without password protection, everything can be started up automatically. (And do keep your computer at a safe place, preferably surrounded with a security system of some kind!)
In short, I needed a monitoring process that:
-
monitors the NIS while the NIS is running
-
makes backups and restores of the NIS database
-
restarts the computer if things get stuck
-
is restarted by the computer after the computer restarts (if you follow me…)
-
starts up the NIS
In terms of states (or “stages”, if you like) it looks as follows:
Diagram 1
Where the long arrow has been drawn in Diagram 1, Windows is starting up. Windows 10 has the following personal startup folder:
C:\Users\[YOUR_NAME]\AppData\Roaming\Microsoft\Windows\Start Menu\Programs\Startup
There is also a system startup folder, I believe, which may also be used instead. Anyway, inside a startup folder I put a .cmd script with the content:
java -jar “[location of the jar]\NisStarter.jar”
The cmd script refers to a runnable Java archive (jar file), which is the NIS monitor. The exact Java code with which the Java archive has been generated will follow below.
Once the Windows is started, Windows starts up the cmd file, which in turn starts the NIS monitor in its initial state (state 1 in Diagram 1). The monitor restores a database backup if present, and starts up the NIS.
Simple solution for process start up
Usually in Java, process handling is done using java.lang.ProcessBuilder, or an alike API. For ease and simplicity, I used the awt.Robot class instead, which double clicks on my desktop on position 270 (horizontal) by 700 (vertical), where I have a shortcut to the runNis.bat file. And next to this shortcut, I have yet another shortcut to a restart.bat file with the content “shutdown -r -f”. Because my Windows command windows pop up without covering up these shortcuts, it works like a charm. Here a screenshot of my desktop (the runNis shortcut is the one selected, and at the right is the (shortcut to) the Windows restart script):
NIS started
Once the NIS has been started up, it will start loading blocks (from the block-chain) and the state of the monitor changes to “started”. For simplicity we left out from Diagram 1 the fact that in this “started” state 2 the NIS is also being monitored. The NIS can become stuck, and if so, a complete reboot follows. But normally, the blocks are loaded.
The monitor reads the block number of the local NIS, as well as of a remote NIS, and compares the two. If the two numbers differ less than three, the monitor assumes the two are in sync, and it’s time for the next state of Diagram 1. In this sync state 3, a backup of the block-chain database is made every hour. Every two minutes, the monitor verifies that the NIS is still in sync with the rest of the network (or to be precise, with the remote NIS).
If out of sync (by more than 100 blocks), stage 4 is entered, which means a complete reboot. The reboot is necessary to make sure that any redundant stuck processes are killed, that the database is being restored, and that the NIS can make a fresh start.
Running the monitor for the mining process
I have deliberately written the whole monitor inside one single Java class, in a “main” package, in a Java project named “NisStarter”. If you have installed Java 7 or 8 (including the JDK) and the standard Java IDE Eclipse, and you know how to handle Eclipse, it is easy to:
- make a new Java project “NisStarter”,
- inside the project a package “main”,
- inside the package a class named “StartNis”,
- simply paste the code which follows below and replace [YOUR_NAME] by the meaningful variant,
- choose, if you can, a URL_REMOTE of your own please, before all of us start bombarding poor Alice5 with requests,
- right-click the project or the class, export the class to the runnable archive, which is mentioned inside the .cmd file of the startup folder (see earlier),
- make sure the short cut links (to the runNis and to the restart batch file) are in position
- restart Windows and keep your fingers crossed
- once in sync, the output of the NIS should contain lines about harvesting attempts
I hope my rather technical writing notes will help anyone to start running and monitoring his own local miner. From own experience, it works for my harvesting NIS, but the code is written in a way that can possibly be applied to other mining engines as well. Happy harvesting!
======= HERE FOLLOWS CODE OF StartNis.java (PART 1/3 FOLLOWED BY SCROLLABLE PART 2/3) =======
======= EASY TO MISS IS PART 3, CONSISTING OF A SINGLE } ======
package main;
import java.awt.AWTException;
import java.awt.Robot;
import java.awt.event.InputEvent;
import java.io.BufferedReader;
import java.io.File;
import java.io.IOException;
import java.io.InputStreamReader;
import java.net.URL;
import java.net.URLConnection;
import java.nio.charset.Charset;
import java.nio.file.Files;
import java.nio.file.StandardCopyOption;
import java.util.Date;
import java.util.logging.Level;
import java.util.logging.Logger;
public class StartNis {
// constants
private static final String DB = “C:\Users\[YOUR_NAME]\nem\nis\data\nis5_mainnet.h2.db”;
private static final String DB_BACKUP = “C:\Users\[YOUR_NAME]\nem\nis\data\nis5_mainnet.h2.db.backup”;
private static final String IN_SYNC = “InSync”;
private static final String STARTED = “Started”;
private static final String INITIAL = “Initial”;
private static final String URL_LOCAL = “http://localhost:7890/chain/height”;
private static final String URL_REMOTE = “http://alice5.nem.ninja:7890/chain/height”;
// instance fields
private String state = INITIAL;
private Logger logger = Logger.getLogger(StartNis.class.getName());
private Long blockNr;
private Long previousLocalBlockNr;
private Long previousPreviousLocalBlockNr;
private Long remoteBlockNr;
private int counterForDbBackup;
private Date dateTimeSyncReached;
private Date dateTimeDbBackup;
/**
* Main function.
*
* @param args
* - no arguments needed or used
*/
public static void main(String[] args) {
StartNis startNis = new StartNis();
while (true) {
startNis.remoteBlockNr = Long.valueOf(startNis.filterDigitsAndSignFromString(startNis.callURL(URL_REMOTE)));
if (INITIAL.equals(startNis.state)) {
startNis.restoreDatabase();
startNis.startupNis();
// adjust state
startNis.state = STARTED;
} else if (STARTED.equals(startNis.state)) {
startNis.verifyThatReadingBlocksIsAlive();
startNis.adjustStateWhenNisIsInSyncWithNemNetwork();
} else if (IN_SYNC.equals(startNis.state)) {
startNis.restartWindowsWhenNisOutOfSync();
// if this part is reached the NIS is in sync, and it's worth to
// make a database backup
startNis.backupDatabaseWhenInSync();
}
startNis.logger.info("Current state " + startNis.state);
startNis.waitFor2Minutes();
}
}
private void backupDatabaseWhenInSync() {
// backup database
counterForDbBackup++;
if (counterForDbBackup == 30) {
counterForDbBackup = 0;
makeBackupOfDatabase();
}
logger.info("Last database backup: " + dateTimeDbBackup);
}
private void restartWindowsWhenNisOutOfSync() {
// restart if out of sync
blockNr = Long.valueOf(filterDigitsAndSignFromString(callURL(URL_LOCAL)));
if (-1 == blockNr || null == blockNr || (Math.abs(remoteBlockNr - blockNr) > 100)) {
restartWindows();
} else {
logger.info("remote at " + remoteBlockNr + " and local at " + blockNr + ". Sync: " + dateTimeSyncReached);
}
}
private void adjustStateWhenNisIsInSyncWithNemNetwork() {
if (Math.abs(remoteBlockNr - blockNr) < 3) {
state = IN_SYNC;
dateTimeSyncReached = new Date();
logger.info("sync reached at " + dateTimeSyncReached);
logger.info("remote at " + remoteBlockNr + " and local at " + blockNr);
}
}
private void verifyThatReadingBlocksIsAlive() {
previousPreviousLocalBlockNr = previousLocalBlockNr;
previousLocalBlockNr = blockNr;
blockNr = Long.valueOf(filterDigitsAndSignFromString(callURL(URL_LOCAL)));
if (-1 == blockNr || null == blockNr || blockNr == previousPreviousLocalBlockNr) {
restartWindows();
} else {
logger.info("remote at " + remoteBlockNr + " and local at " + blockNr);
}
}
private void restoreDatabase() {
// restore db
File dbBackup = new File(DB_BACKUP);
if (dbBackup.exists()) {
try {
File possiblyCorruptDb = new File(DB);
Files.copy(dbBackup.toPath(), possiblyCorruptDb.toPath(), StandardCopyOption.REPLACE_EXISTING,
StandardCopyOption.COPY_ATTRIBUTES);
} catch (IOException e) {
logger.log(Level.SEVERE, e.getMessage(), e);
}
}
}
private void waitFor2Minutes() {
// wait for 2 minutes (= 120 seconds = 120000 milliseconds)
try {
Thread.sleep(120000);
} catch (InterruptedException e) {
logger.log(Level.SEVERE, e.getMessage(), e);
}
}
private void makeBackupOfDatabase() {
try {
File goodDb = new File(DB);
File dbBackup = new File(DB_BACKUP);
Files.copy(goodDb.toPath(), dbBackup.toPath(), StandardCopyOption.REPLACE_EXISTING,
StandardCopyOption.COPY_ATTRIBUTES);
dateTimeDbBackup = new Date();
} catch (IOException e) {
logger.log(Level.SEVERE, e.getMessage(), e);
}
}
private void startupNis() {
Robot robot;
try {
robot = new Robot();
robot.mouseMove(270, 700);
doubleClick(robot);
logger.info("NIS process started at " + new Date());
} catch (AWTException e) {
logger.log(Level.SEVERE, e.getMessage(), e);
}
}
private void doubleClick(Robot robot) {
robot.mousePress(InputEvent.BUTTON1_DOWN_MASK);
robot.mouseRelease(InputEvent.BUTTON1_DOWN_MASK);
robot.mousePress(InputEvent.BUTTON1_DOWN_MASK);
robot.mouseRelease(InputEvent.BUTTON1_DOWN_MASK);
}
private void restartWindows() {
// start
Robot robot;
try {
robot = new Robot();
robot.mouseMove(360, 700);
doubleClick(robot);
logger.info("Windows restart at " + new Date());
// full stop
System.exit(-1);
} catch (AWTException e) {
logger.log(Level.SEVERE, e.getMessage(), e);
}
}
private String filterDigitsAndSignFromString(String stringWithDigits) {
String result = "";
char[] digits = stringWithDigits.toCharArray();
for (char c : digits) {
if (Character.isDigit(c) || '-' == c) {
result = result + c;
}
}
return result;
}
private String callURL(String urlString) {
logger.info("Requested URL:" + urlString);
StringBuilder sb = new StringBuilder();
URLConnection urlConn = null;
InputStreamReader in = null;
try {
URL url = new URL(urlString);
urlConn = url.openConnection();
if (urlConn != null) {
urlConn.setReadTimeout(60 * 1000);
if (urlConn.getInputStream() != null) {
in = new InputStreamReader(urlConn.getInputStream(), Charset.defaultCharset());
BufferedReader bufferedReader = new BufferedReader(in);
if (bufferedReader != null) {
int cp;
while ((cp = bufferedReader.read()) != -1) {
sb.append((char) cp);
}
bufferedReader.close();
}
}
}
in.close();
} catch (IOException e) {
logger.log(Level.SEVERE, "Requested URL " + urlString + " not found.");
return "-1";
}
return sb.toString();
}
}
