NEM Catapult Developer Resources

crackTheCode · November 3, 2019, 1:18pm

Hello NEM Devs!

I will be putting together a list of Catapult resources that you may find useful for your development needs. As I, or others, develop more tools for Catapult, they will be added accordingly to this forum thread. Common errors that one may face while developing or setting up a Catapult node will also be placed here.

If I missed something, feel free to contribute!

Catapult Server Resources

These resources will allow you to study and analyze more about the core of the NEM2 blockchain: catapult-server

catapult-server-docs - This is a doxygen-generated set of documents for catapult-server. It includes graphs that illustrate how various parts of the code interact with one another. Layout suggestions appreciated!

You can find the docs here!

Node Setup Resources

These resouces will help you get up and running with a Catapult node, whether you want to run locally or on a remote server.

cat-install-scripts - A set of scripts to install all Catapult dependencies for Linux and macOS. You should use these before using cat-config. They contain the following functionality:
- Install base system dependencies
- Update cmake and gcc
- install Catapult-specific dependencies: Boost, ZeroMQ, MongoC/CXX, RocksDB, Google Tests & Benchmark.
- Fully configure cmake with installed dependencies for Catapult compilation
- Build Catapult.

You can find the cat-install scripts and how to use them here!

cat-config-scripts - Normally, spinning up a Catapult node can be a bit of a tedious task. Using docker is not always an option either. These scripts (for Mac and Linux) were originally created by core developer Jaguar0625. I took them and added a few extra features to enhance the user experience. They contain the following functionality:
- Reset your Catapult node.
- Configure your node for peer , api or dual mode.
- Configure your node with the necessary extensions for each respective mode.
- Configure resources with a user-given keypair and necessary catapult-server paths.
- Generate a nemesis block properties file with boot key and paths.
- Generate mosaic ids for the given public key.
- Generate a list of harvester keys and addresses, and configure the nemesis block file with them.
- Generate the nemesis block, 64 byte hashes.dat, and a fresh data directory.

You can find the cat-config scripts and how to use them here!

Catapult Tools Guide

Along with the core catapult.server executable that Catapult provides, there are several useful tools that come along with a compiled catapult-server. These tools allow developers to check the health of the network, generate nemesis blocks, generate keypairs and addresses, recover the server in the event of an abrupt exit, and more. They are found inside of /path/to/catapult-server/build/bin.

`catapult.tools.address`

Description

Generates keypairs and addresses.

Help Output

Address Tool
Copyright (c) Jaguar0625, gimre, BloodyRookie, Tech Bureau, Corp.
catapult version: 0.4.0.1 95e0c6a [master]

Address Tool options:
  -h [ --help ]     print help message
  -l [ --loggingConfigurationPath ] arg
                    the path to the logging configuration file
  -g [ --generate ] arg (=10)
                    number of random keys to generate
  -p [ --public ] arg
                    show address associated with public key
  -s [ --secret ] arg
                    show address and public key associated with private key
  -n [ --network ] arg (=public)
                    network, possible values: mijin, mijin-test, public (default), public-test
  -e [ --useHighEntropySource ] 
                    true if a high entropy source should be used for randomness

Usage

./catapult.tools.address -g <number_of_keys> -n <network_type (i.e mijin, mijin-test, public (default), public-test)>

If you want the save the output, add an output stream to the end: > myKeys.txt.

`catapult.tools.nemgen`

Description

This tool creates a nemesis (genesis) block out of a block-properties.properties file. This nemesis block is used to start your Catapult chain.

Help Output

Nemesis Block Generator Tool
Copyright (c) Jaguar0625, gimre, BloodyRookie, Tech Bureau, Corp.
catapult version: 0.4.0.1 95e0c6a [master]

Nemesis Block Generator Tool options:
  -h [ --help ]     print help message
  -l [ --loggingConfigurationPath ] arg
                    the path to the logging configuration file
  -r [ --resources ] arg (=..)
                    the path to the resources directory
  -p [ --nemesisProperties ] arg
                    the path to the nemesis properties file
  -s [ --summary ] arg
                    the path to summary output file (default: <bindir>/summary.txt)
  -n [ --no-summary ] 
                    don't generate summary file
  -t [ --useTemporaryCacheDatabase ] 
                    true if a temporary cache database should be created and destroyed

Usage

./catapult.tools.nemgen -p <nemesis_block_properties_path> -r <path_where_resources_are_kept>

`catapult.tools.benchmark`

Description

This tool utilizes cpu-based performance tests of catapult functions, such as hashing or encryption.

Help Output

Benchmark Tool
Copyright (c) Jaguar0625, gimre, BloodyRookie, Tech Bureau, Corp.
catapult version: 0.4.0.1 95e0c6a [master]

Benchmark Tool options:
  -h [ --help ]     print help message
  -l [ --loggingConfigurationPath ] arg
                    the path to the logging configuration file
  -t [ --num threads ] arg (=0)
                    the number of threads
  -p [ --num partitions ] arg (=0)
                    the number of partitions
  -o [ --ops / partition ] arg (=1000)
                    the number of operations per partition
  -s [ --data size ] arg (=148)
                    the size of the data to generate

Usage

Unless you wish to test using your own parameters, you may just run:
./catapult.tools.benchmark

`catapult.tools.network`

Description

This tool outputs node info, partner node connectivity info, and a partner node table for known peers within peers-p2p.json and peers-api.json.

Help Output

Catapult Block Chain Network Tool
Copyright (c) Jaguar0625, gimre, BloodyRookie, Tech Bureau, Corp.
catapult version: 0.4.0.1 95e0c6a [master]

Catapult Block Chain Network Tool options:
  -h [ --help ]     print help message
  -l [ --loggingConfigurationPath ] arg
                    the path to the logging configuration file
  -r [ --resources ] arg (=..)
                    the path to the resources directory

Usage

./catapult.tools.network -r <path_where_resources_are_kept>

`catapult.tools.statusgen`

Description

This tool is used for generating the various statues that a Catapult server instance may log in certain scenarios. Example: it’s used for the catapult-rest repo.

Help Output

Status Generator
Copyright (c) Jaguar0625, gimre, BloodyRookie, Tech Bureau, Corp.
catapult version: 0.4.0.1 95e0c6a [master]

Status Generator options:
  -h [ --help ]     print help message
  -l [ --loggingConfigurationPath ] arg
                    the path to the logging configuration file

Usage

./catapult.tools.statusgen

`catapult.tools.health`

Description

Checks the health and displays statistics (like the unconfirmed transaction cache) for the nodes found inside of peers-p2p.json and peers-api.json.

Help Output

Catapult Block Chain Health Tool
Copyright (c) Jaguar0625, gimre, BloodyRookie, Tech Bureau, Corp.
catapult version: 0.4.0.1 95e0c6a [master]

Catapult Block Chain Health Tool options:
  -h [ --help ]     print help message
  -l [ --loggingConfigurationPath ] arg
                    the path to the logging configuration file
  -r [ --resources ] arg (=..)
                    the path to the resources directory

Usage

./catapult.tools.health -r <path_where_resources_are_kept>

Common Node Errors Or Issues

This section will highlight common errors or difficulties one may encounter whilst setting a Catapult node. These shouldn’t happen with the scripts, but more information’s sake they’re always good to know about.

Mosaic Outflows Not Matching

The error usually looks something like this:

14:19:34.719477 0x000000010dfbf5c0: <error> (local::HostUtils.h@43) Throwing exception: harvesting outflows (0) do not add up to power ten multiple of expected importance (15000000)

or this

14:19:34.719477 0x000000010dfbf5c0: <error> (local::HostUtils.h@43) Throwing exception: harvesting outflows (15000000) do not add up to power ten multiple of expected importance (17000000)

What does it mean?

This error can happen in the event of one of two things:

The supply of the cat.harvest mosaic the nemesis block does not match the supply inside of the config-network.properties file.
The mosaic ID inside of the config-network.properties does not match the one generated inside of the nemesis block.

How to fix it?

For scenario one, simply change the totalChainImportance in your resources config-network.properties to match the supply set in the nemesis block. For scenario two, you will have to find the correct mosaic ID that the nemgen tool creates. One way to do this is to run nemgen (see above in Catapult Tools Guide), and output to a txt or log file, and search for cat.<harvest | currency>.

Catapult Client Development

This section will cover the resources needed to develop native or cross-platform apps on Catapult.

Ionic with Capacitor

Built on standard web technology, Ionic helps teams build and ship beautiful cross-platform hybrid and Progressive Web Apps with ease. However, due to the nem2-sdk depending on some node-specific dependencies, it can be hard to get Ionic to recognize that some of them need to be pollyfilled, i.e, you may have run into some of these:

[ng] ERROR in ./node_modules/request/lib/har.js
[ng] Module not found: Error: Can't resolve 'fs' in '/Users/bader/Documents/Coding/ionic-nem2-patch/nem2-ionic/node_modules/request/lib'
[ng] ERROR in ./node_modules/forever-agent/index.js
[ng] Module not found: Error: Can't resolve 'net' in '/Users/bader/Documents/Coding/ionic-nem2-patch/nem2-ionic/node_modules/forever-agent'
[ng] ERROR in ./node_modules/nem2-sdk/node_modules/ws/lib/websocket.js
[ng] Module not found: Error: Can't resolve 'net' in '/Users/bader/Documents/Coding/ionic-nem2-patch/nem2-ionic/node_modules/nem2-sdk/node_modules/ws/lib'
[ng] ERROR in ./node_modules/tough-cookie/lib/cookie.js
[ng] Module not found: Error: Can't resolve 'net' in '/Users/bader/Documents/Coding/ionic-nem2-patch/nem2-ionic/node_modules/tough-cookie/lib'
[ng] ERROR in ./node_modules/tunnel-agent/index.js
[ng] Module not found: Error: Can't resolve 'net' in '/Users/bader/Documents/Coding/ionic-nem2-patch/nem2-ionic/node_modules/tunnel-agent'
[ng] ERROR in ./node_modules/forever-agent/index.js
[ng] Module not found: Error: Can't resolve 'tls' in '/Users/bader/Documents/Coding/ionic-nem2-patch/nem2-ionic/node_modules/forever-agent'

To solve this, I have put together a simple patch file + a template project for developing Ionic apps on Catapult!

You can find the template + patch.js file here!

NEM2 Dart SDK (can use with Flutter)

Kudos to @fvm , the only author of this wonderfully useful package!

This SDK is currently a work in progress and should not be used for production.

The nem2-sdk-dart is an unofficial SDK written in Dart. Developers can use this SDK to build applications for multiple platforms using a single codebase: Mobile for both iOS and Android (with Flutter), Web, or Command-line/Server-side, and to allow communication with the NEM2 (Catapult) REST API enabled servers.

As it can be used with Flutter, it is able to be used to build beautiful, natively-compiled applications for mobile, web, and desktop from a single codebase.

You can find the nem2-sdk-dart here!

crackTheCode · July 5, 2019, 4:11pm

Added Catapult tools and common errors sections.

crackTheCode · July 14, 2019, 1:06pm

Added client-side development resources. If I missed any, please drop them here in this thread!

fvm · July 14, 2019, 1:15pm

Wow. Thanks for compiling this great dev guide!

crackTheCode · November 3, 2019, 1:19pm

Added cat-install-scripts to install Catapult + dependencies.