NEM 1.0 / NEM 2.0 cryptographic algorithm (for hardware-signing)

r3n3 · December 22, 2019, 7:55am

Dear @gevs, @BloodyRookie (and other NEM tech wizards),
cc. @jtey1 (LuxTag) and @musdom (LuxTag)

We at LuxTag are working on un-copyable tags using NFC embedded Hardware Security Modules which do the cryptographic signing. They keep the randomly generated private key inside the chip and do the signing, similar to common crypto hardware wallets.

Recently, we spoke to a manufacturer to check on compatibility with NEM cryptography, so that the signing could happen on-chip.

They said their chips are compatible with the common cryptography and as such with the main blockchain projects. They call NEM an “exotic blockchain”.
They said that NEM 1.0 would be able to work with minor tweaks, as it uses ED25519 SHA2-512.
NEM 2.0 (Codenames Catapult, Symbol) would not work as it deviates from industry standard. To my understanding, this relates to the use of SHA3-512.
The chip manufacturer seems to advise us to change from the “exotic and incompatible blockchain NEM1 / NEM2” towards a more mainstream platform. (Allegedly, even Hyperledger Fabric and others are out-of-the box compatible)

Can I ask for opinions and recommendations regarding the hardware module signing (hashing/signing) for NEM 2.0?

Thank you and happy 4th advent sunday .
( https://www.german-way.com/history-and-culture/holidays-and-celebrations/christmas/advent/ )
Rene, LuxTag.io

gevs · December 23, 2019, 12:03pm

Hello there @r3n3 and thanks for reaching out.

NIS uses Keccak-SHA3-512, it is indeed deviating from “standard” sha3 blockchain projects.
Good - they probably are saying that standard SHA3 would be better.
Catapult uses standard SHA3-512 in private mode, but Keccak-SHA3-512 for public mode. This means that it should also make Catapult compatible as of point 2).

Hope to answer your question

BloodyRookie · December 23, 2019, 2:01pm

I guess the guys didn’t closely look at NIS1? It is not using sha2.

r3n3 · December 24, 2019, 1:40am

Thanks guys.
It was sad how NEM is being called “exotic, non mainstream”. I guess that’s what we’re all working on to change!

I found https://nemtech.github.io/concepts/cryptography.html useful, too. It talks about SHA3-256 and the KECCAK SHA3-256 for (the blockchain platform formerly known as) Catapult.

@gevs You say

whereby the document at nemtech.github.io above explains the -256 version.

What is right? Sorry for asking these questions. Am not a cryptography expert.
I want to avoid confusions when I go back to the hardware security module chip manufacturer.

GodTanu · December 24, 2019, 2:14am

Is this helpful?
This is a nem2-sdk original implementation.

https://github.com/nemtech/nem2-sdk-typescript-javascript/blob/master/src/core/crypto/nacl_catapult.ts

References
https://yu-kimura.jp/2019/12/20/nem-curve25519/

r3n3 · December 24, 2019, 8:20am

Very good write-up in Japanese!
I read the translation. Still, am a bit confused on the “why” and “256 / 512” question because Greg and NEMtech give different information.

https://translate.google.com/translate?sl=ja&tl=en&u=https%3A%2F%2Fyu-kimura.jp%2F2019%2F12%2F20%2Fnem-curve25519%2F

gevs · December 24, 2019, 2:21pm

@dgarcia360 there is an inconsistency in docs i believe. We use 64 bytes keys and hashes, and therefor should be keccak-sha512, correct me if wrong please

dgarcia360 · December 26, 2019, 8:01pm

The docs seem correct, but we are probably talking about different things. The info was taken from the catapult-server code PublicKeyToAddress function:

github.com

nemtech/catapult-server/blob/master/src/catapult/model/Address.cpp#L51-L69


	Address PublicKeyToAddress(const Key& publicKey, NetworkIdentifier networkIdentifier) {
		// step 1: sha3 hash of the public key
		Hash256 publicKeyHash;
		CatapultHash(publicKey, publicKeyHash);


		// step 2: ripemd160 hash of (1)
		Address decoded;
		crypto::Ripemd160(publicKeyHash, reinterpret_cast<Hash160&>(decoded[1]));


		// step 3: add network identifier byte in front of (2)
		decoded[0] = utils::to_underlying_type(networkIdentifier);


		// step 4: concatenate (3) and the checksum of (3)
		Hash256 step3Hash;
		CatapultHash(RawBuffer{ decoded.data(), Hash160::Size + 1 }, step3Hash);
		std::copy(step3Hash.cbegin(), step3Hash.cbegin() + Checksum_Size, decoded.begin() + Hash160::Size + 1);


		return decoded;
	}

From what I have understood, the values CatapultHash can take are Keccak_256 or Sha3_256, depending on the config value SIGNATURE_SCHEME_KECCAK:

github.com

nemtech/catapult-server/blob/master/src/catapult/model/Address.cpp#L33-L37


#ifdef SIGNATURE_SCHEME_KECCAK
		constexpr auto CatapultHash = crypto::Keccak_256;
#else
		constexpr auto CatapultHash = crypto::Sha3_256;
#endif

To sign entities, Catapult uses SHA3-512 or Keccack-512.

github.com

nemtech/catapult-server/blob/621ccbd1e7ade62fd22fe4f2dcfd88f311ae2e0a/src/catapult/crypto/Signer.cpp#L125-L179


	void Sign(const KeyPair& keyPair, std::initializer_list<const RawBuffer> buffersList, Signature& computedSignature) {
		uint8_t *RESTRICT encodedR = computedSignature.data();
		uint8_t *RESTRICT encodedS = computedSignature.data() + Encoded_Size;


		// hash the private key to improve randomness
		Hash512 privHash;
		HashPrivateKey(keyPair.privateKey(), privHash);


		// r = H(privHash[256:512] || data)
		// "EdDSA avoids these issues by generating r = H(h_b, ..., h_2b?1, M), so that
		//  different messages will lead to different, hard-to-predict values of r."
		Hash512 hash_r;
		HashBuilder hasher_r;
		hasher_r.update({ privHash.data() + Hash512::Size / 2, Hash512::Size / 2 });
		hasher_r.update(buffersList);
		hasher_r.final(hash_r);


		bignum256modm r;
		expand256_modm(r, hash_r.data(), 64);

This file has been truncated. show original

github.com

nemtech/catapult-server/blob/621ccbd1e7ade62fd22fe4f2dcfd88f311ae2e0a/src/catapult/crypto/CryptoUtils.cpp#L27-L32


	void HashPrivateKey(const PrivateKey& privateKey, Hash512& hash) {
#ifdef SIGNATURE_SCHEME_KECCAK
		Keccak_512({ privateKey.data(), privateKey.size() }, hash);
#else
		Sha3_512({ privateKey.data(), privateKey.size() }, hash);
#endif

Indeed, this part is not fully defined in the docs yet. We can add an example as we are doing with the public key -> address derivation.

AndriusB · December 27, 2019, 5:50pm

Guys, can you please explain the differences between SHA-3 and Keccak-SHA-3 that Private and Public catapult networks use? Because as I am reading SHA-3 standard by NIST it seems that it was submitted by Keccak team and it is the same algorithm.

r3n3 · December 28, 2019, 5:07am

+1
I’d like to be enlightened, too.
Layman websites like Coinguides.org say “Keccak (pronounced as “ketchak”) also known as SHA-3 (Secure Hash Algorithm 3). It is the latest generation secure hashing algorithm released by NIST (National Institutes of Standards and Technology) in 2012.”

gevs · December 28, 2019, 1:16pm

They are different. Keccak was originally drafted and proposed in NIST standard but then some padding changes were introduced which did not make it into the standard.

Therefor they are slightly different and there is also a lot of confusion in some packages on the net… cryptoJs for example had only a small one line comment about “this is using KECCAK, not sha3”, yet naming the function sha3…

AndriusB · December 28, 2019, 7:05pm

Ok, but why to stick with Keccak-SHA3 instead of standartised version of SHA3 on Public network then. It will bring lots of compatibility and support problems with HW vendors like HSM, SIM and etc that will be using standartised cryptography. By sticking with nin standard cryptography you limiting the possibility to use HW solutions for signing transactions on a public network. This seems not rational at all, because security wise you are not gaining anyting there.

gimre · January 10, 2020, 11:26am

to maintain backward compatibility with nem1 addresses. (to be able to actually make symbol nemesis transfers)