Merge pull request #960 from HorizenOfficial/dev

0.9.0 to master

.gitignore

@@ -28,3 +28,6 @@ qa/.metals/metals.lock.db
 qa/.metals/metals.mv.db
 qa/.metals/metals.log
 .scalafmt.conf
+
+# Jacoco reports
+coverage-reports/**

.travis.yml

@@ -17,6 +17,9 @@ env:
   global:
     # Separate branches by space
     - PROD_RELEASE_BRANCHES='master'
+    - NODE_VERSION="v16"
+    - TEST_CMD="./run_sc_tests.sh"
+    - API_ZEN_REPO_URL="https://api.github.com/repos/HorizenOfficial/zen"
 
 before_script:
   - source ci/setup_env.sh

CHANGELOG.md

@@ -1,3 +1,13 @@
+**0.9.0**
+1. libevm dependency updated to 1.0.0.
+2. Added support for EVM and native smart contracts interoperability.
+3. Sparkz dependency updated to 2.2.0.
+4. Improved storage versioning (fullsynch time reduced by 5x)
+5. Minor fixes:
+    * [eth RPC endpoint] debug_traceCall now returns a more accurate error response for reverted transactions 
+    * [eth RPC endpoint] debug_traceCall and debug_traceTransaction now return a correct value for the gasUsed field when topmost call is a call to a Solidity Smart Contract function.
+    * Certificates older than 4 epochs are now deleted from the storage only if more recent certificates appeared.
+
 **0.8.1**
 1. Improved precision of eth_gasPrice RPC call
 
@@ -141,4 +151,4 @@
 6. Possibility to declare custom Transactions/Boxes/Secrets/etc.
 7. Possibility to extend/manage basic API.
 8. Web interface and command line tool for interaction with the Node.
-9. Sidechain Bootstrapping Tool to configure sidechain network according to the mainchain network.
+9. Sidechain Bootstrapping Tool to configure sidechain network according to the mainchain network.

README.md

@@ -35,7 +35,7 @@ For more details see [zendoo-sc-cryptolib](https://github.com/HorizenOfficial/ze
 
 * Java 11 or newer
 * Scala 2.12.10+
-* Python 3
+* Python 3.10
 * Maven
 
 On some Linux OSs during backward transfers certificates proofs generation a extremely big RAM consumption may happen, that will lead to the process force killing by the OS.
@@ -48,7 +48,7 @@ While we keep monitoring the memory footprint of the proofs generation process,
  - After the installation, just run `export LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so.1` before starting the sidechain node, or run the sidechain node adding `LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so.1` at the beginning of the java command line as follows:
 
 ```
-LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so.1 java -cp ./target/sidechains-sdk-simpleapp-0.8.1.jar:./target/lib/* io.horizen.examples.SimpleApp <path_to_config_file>
+LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so.1 java -cp ./target/sidechains-sdk-simpleapp-0.9.0.jar:./target/lib/* io.horizen.examples.SimpleApp <path_to_config_file>
 ```
  - In the folder `ci` you will find the script `run_sc.sh` to automatically check and use jemalloc library while starting the sidechain node. 
 

ci/run_python_tests.sh

@@ -0,0 +1,182 @@
+#!/bin/bash
+
+set -euo pipefail
+
+# Functions
+function fn_die() {
+  echo -e "$1" >&2
+  exit "${2:-1}"
+}
+
+function import_gpg_keys() {
+  # shellcheck disable=SC2207
+  declare -r my_arr=( $(echo "${@}" | tr " " "\n") )
+
+  if [ "${#my_arr[@]}" -eq 0 ]; then
+    fn_die "Error: There are ZERO gpg keys to import. ZEN_REPO_MAINTAINER_KEYS variable is not set. Exiting ..."
+  else
+    # shellcheck disable=SC2145
+    printf "%s\n" "Tagged build, fetching keys:" "${@}" ""
+    for key in "${my_arr[@]}"; do
+      gpg -v --batch --keyserver hkps://keys.openpgp.org --recv-keys "${key}" ||
+      gpg -v --batch --keyserver hkp://keyserver.ubuntu.com --recv-keys "${key}" ||
+      gpg -v --batch --keyserver hkp://pgp.mit.edu:80 --recv-keys "${key}" ||
+      gpg -v --batch --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys "${key}" ||
+      { fn_die "Error: ${key} can not be found on GPG key servers. Please upload it to at least one of the following GPG key servers:\nhttps://keys.openpgp.org/\nhttps://keyserver.ubuntu.com/\nhttps://pgp.mit.edu/ Exiting ...";}
+    done
+  fi
+}
+
+function check_signed_tag() {
+  local tag="${1}"
+
+  if git verify-tag -v "${tag}"; then
+    echo "${tag} is a valid signed tag"
+  else
+    fn_die "Error: ${tag} signature is NOT valid. Exiting ..."
+  fi
+}
+
+
+if [ -z "${NODE_VERSION:-}" ]; then
+  fn_die "Error: NODE_VERSION variable is not set. Exiting ..."
+fi
+
+if [ -z "${TEST_CMD:-}" ]; then
+  fn_die "Error: TEST_CMD variable is not set. Exiting ..."
+fi
+
+if [ -z "${TEST_ARGS:-}" ]; then
+  fn_die "Error: TEST_ARGS variable is not set. Exiting ..."
+fi
+
+if [ -z "${API_ZEN_REPO_URL:-}" ]; then
+  fn_die "Error: API_ZEN_REPO_URL variable is not set. Exiting ..."
+fi
+
+if [ -z "${ZEN_REPO_MAINTAINER_KEYS:-}" ]; then
+  fn_die "Error: ZEN_REPO_MAINTAINER_KEYS variable is not set. Exiting ..."
+fi
+
+CURRENT_DIR="${PWD}"
+
+# Step 1
+echo "" && echo "=== Get latest ZEN repo PROD build id and commit hash ===" && echo ""
+
+zen_tag="$(curl -H "Accept: application/vnd.github.v3+json" https://api.github.com/repos/HorizenOfficial/zen/git/refs/tags | jq -r '[.[] | select(.ref | test("refs/tags/v[0-9]\\.[0-9]\\.[0-9]$"))][-1].ref' | sed -e 's|refs/tags/||')"
+check_runs="$(curl -sL -H "Accept: application/vnd.github+json" -H "X-GitHub-Api-Version: 2022-11-28" "${API_ZEN_REPO_URL}/commits/${zen_tag}/check-runs")"
+travis_build_id="$(basename "$(jq -rc '.check_runs[0].details_url' <<< "${check_runs}")")"
+commit_sha="$(jq -rc '.check_runs[0].head_sha' <<< "${check_runs}")"
+
+travis_urls="
+#amd64
+https://f001.backblazeb2.com/file/ci-horizen/amd64-linux-ubuntu_focal-${travis_build_id}-${commit_sha}.tar.gz.sha256
+https://f001.backblazeb2.com/file/ci-horizen/amd64-linux-ubuntu_focal-${travis_build_id}-${commit_sha}.tar.gz
+"
+
+
+# Step 2
+echo "" && echo "=== Create folder structure ===" && echo ""
+
+base_dir="${CURRENT_DIR}/zen_release"
+is_cached=false
+
+if [ -d "${base_dir}/travis_files" ]; then
+  echo "${base_dir} folder already exists, using cache!"
+  cd "${base_dir}"/travis_files
+  is_cached=true
+else
+  mkdir -p "${base_dir}"/{travis_files,src}
+
+  # Step 3
+  echo "" && echo "=== Downloading ZEN artifacts from remote bucket ===" && echo ""
+
+  cd "${base_dir}"/travis_files
+  echo "$travis_urls" > ./travis_urls.txt
+  sudo apt-get update
+  sudo apt-get -y --no-install-recommends install aria2
+  aria2c -x16 -s16 -i ./travis_urls.txt --allow-overwrite=true --always-resume=true --auto-file-renaming=false
+fi
+
+
+# Step 4
+echo "" && echo "=== Checksum verification ===" && echo ""
+
+if shasum -a256 -c ./*.sha256; then
+  echo "Checksum verification passed."
+else
+  fn_die "Error: checksum verification failed. Exiting ..."
+fi
+
+# Step 5
+release_folder="zen-${zen_tag}-amd64"
+if [ "$is_cached" = false ]; then
+  echo "" && echo "=== Extract artifacts from tar ===" && echo ""
+  tar_file="$(find "$(realpath ${base_dir}/travis_files/)" -type f -name "*.tar.gz")"
+
+  mkdir -p "${base_dir}/src/${release_folder}"
+  tar -xzf "${tar_file}" -C "${base_dir}/src/${release_folder}"
+fi
+
+# Step 6
+echo "" && echo "=== Verify git tag signed by allowlisted maintainer ===" && echo ""
+
+cd "${base_dir}/src/${release_folder}"
+GNUPGHOME="$(mktemp -d 2>/dev/null || mktemp -d -t "GNUPGHOME")"
+export GNUPGHOME
+import_gpg_keys "${ZEN_REPO_MAINTAINER_KEYS}"
+check_signed_tag "${zen_tag}"
+( gpgconf --kill dirmngr || true )
+( gpgconf --kill gpg-agent || true )
+rm -rf "${GNUPGHOME:?}"
+unset GNUPGHOME
+
+# Step 7
+echo "" && echo "=== Export BITCOINCLI, BITCOIND and SIDECHAIN_SDK path as env vars, needed for python tests ===" && echo ""
+
+BITCOINCLI="${base_dir}/src/${release_folder}/src/zen-cli"
+BITCOIND="${base_dir}/src/${release_folder}/src/zend"
+SIDECHAIN_SDK="${CURRENT_DIR}"
+
+if [[ ! -f "$BITCOINCLI" ]]; then
+  fn_die "Error: zen-cli does not exist in the given path. Exiting ..."
+fi
+if [[ ! -f "$BITCOIND" ]]; then
+  fn_die "Error: zend does not exist in the given path. Exiting ..."
+fi
+if [[ ! -d "$SIDECHAIN_SDK" ]]; then
+  fn_die "Error: Sidechain-SDK does not exist in the given path. Exiting ..."
+fi
+
+export BITCOINCLI
+export BITCOIND
+export SIDECHAIN_SDK
+
+# Step 8
+echo "" && echo "=== Fetch zen params ===" && echo ""
+${base_dir}/src/${release_folder}/zcutil/fetch-params.sh || { retval="$?"; echo "Error: was not able to fetch zen params."; exit $retval; }
+
+# Step 9
+echo "" && echo "=== Building SideChain SDK ===" && echo ""
+cd $CURRENT_DIR
+mvn clean install -Dmaven.test.skip=true || { retval="$?"; echo "Error: was not able to complete mvn clean install of Sidechain SDK."; exit $retval; }
+
+# Step 10
+echo "" && echo "=== Installing nodejs ===" && echo ""
+source ~/.nvm/nvm.sh
+nvm install "${NODE_VERSION}" || { retval="$?"; echo "Error: was not able to nvm install node ${NODE_VERSION}"; exit $retval; }
+
+# Step 11
+echo "" && echo "=== Installing yarn ===" && echo ""
+npm install --global yarn || { retval="$?"; echo "Error: was not able to install yarn with npm install."; exit $retval; }
+
+# Step 12
+echo "" && echo "=== Installing Python dependencies ===" && echo ""
+pip install --no-cache-dir --upgrade pip
+pip install --no-cache-dir -r ./requirements.txt
+cd qa/
+pip install --no-cache-dir -r ./SidechainTestFramework/account/requirements.txt
+
+# Step 13
+echo "" && echo "=== Run tests ===" && echo ""
+"${TEST_CMD}" "${TEST_ARGS}"

ci/run_sc.sh

@@ -2,7 +2,7 @@
 
 set -eo pipefail
 
-SIMPLE_APP_VERSION="${SIMPLE_APP_VERSION:-0.8.0}"
+SIMPLE_APP_VERSION="${SIMPLE_APP_VERSION:-0.9.0}"
 
 if [ -d "$1" ] && [ -f "$2" ]; then
   path_to_jemalloc="$(ldconfig -p | grep "$(arch)" | grep 'libjemalloc\.so\.1$' | tr -d ' ' | cut -d '>' -f 2)"

coverage-reports/generate_report.sh

@@ -0,0 +1,57 @@
+#!/bin/bash
+
+# this script generates jacoco code coverage report
+# before running it, it is necessary to have the following env vars set - $SIDECHAIN_SDK, $BITCOIND and $BITCOINCLI
+# additionally, it is necessary to have org.jacoco.cli-0.8.9-nodeps.jar which can be found here - https://repo1.maven.org/maven2/org/jacoco/org.jacoco.cli/0.8.9/
+# this script should be run in the root of coverage-reports folder
+
+# Specify snapshot version
+SNAPSHOT_VERSION_TAG="0.9.0"
+
+# Check if SIDECHAIN_SDK is set and not empty
+if [ -z "$SIDECHAIN_SDK" ]; then
+  echo "Error: SIDECHAIN_SDK is not set or is empty. Please set the environment variable."
+  exit 1
+fi
+
+# Step 1: Change to the project directory
+echo "Changing to the project directory..."
+cd ../sdk || exit 1
+
+# Step 2: Run 'mvn clean install' in the project directory
+# it is necessary to let test phase run as well, because this phase creates .exec file for this code coverage report
+echo "Running 'mvn clean install' in the project directory..."
+mvn clean install
+
+# Check if BITCOIND and BITCOINCLI are set and not empty before running python tests
+if [ -n "$BITCOIND" ] && [ -n "$BITCOINCLI" ]; then
+  # Step 3: Execute run_sc_tests.sh
+  # this phase runs python tests which append to the previously created .exec file to get the full code coverage report
+  echo "Executing script for integration tests..."
+  cd ../qa || exit 1
+  ./run_sc_tests.sh -jacoco
+else
+    echo "Warning: Either BITCOIND or BITCOINCLI is not set or is empty. Please set both environment variables."
+fi
+
+# Step 3: Execute run_sc_tests.sh
+# this phase runs python tests which append to the previously created .exec file to get the full code coverage report
+echo "Executing script for integration tests..."
+cd ../qa || exit 1
+./run_sc_tests.sh -jacoco
+
+# Step 4: Generate the JaCoCo code coverage report
+# this phase creates the detailed report with html files for easier browsing
+echo "Generating JaCoCo code coverage report..."
+
+JACOCO_JAR_PATH="$HOME/.m2/repository/org/jacoco/org.jacoco.cli/0.8.9/org.jacoco.cli-0.8.9-nodeps.jar"
+EXEC_PATH="$SIDECHAIN_SDK/coverage-reports/sidechains-sdk-${SNAPSHOT_VERSION_TAG}/sidechains-sdk-${SNAPSHOT_VERSION_TAG}-jacoco-report.exec"
+CLASSFILES_PATH="$SIDECHAIN_SDK/sdk/target/classes"
+HTML_PATH="$SIDECHAIN_SDK/coverage-reports/sidechains-sdk-${SNAPSHOT_VERSION_TAG}/sidechains-sdk-${SNAPSHOT_VERSION_TAG}-jacoco-report"
+
+java -jar "$JACOCO_JAR_PATH" \
+  report "$EXEC_PATH" \
+  --classfiles "$CLASSFILES_PATH" \
+  --html "$HTML_PATH"
+
+echo "Code coverage report generation complete."

doc/index.md

@@ -1,3 +1,6 @@
 # Horizen Sidechain SDK Release Notes
 
-## Version [0.8.0](/doc/release/0.8.0.md) 
+## Version [0.9.0](/doc/release/0.9.0.md) 
+## Version [0.8.1](/doc/release/0.8.1.md) 
+## Version [0.8.0](/doc/release/0.8.0.md) 
+

doc/release/0.8.0.md

@@ -1,4 +1,5 @@
 # Release notes - version 0.8.0
+
 ---
 
 ## Notes about new/updated Features
@@ -28,7 +29,7 @@ sparkz {
 ```
 
 ### Forgers Network
-This feature adds a dedicated connection pool reserved for connecting to forger nodes. The size of the dedicated connection pool is configured by `maxForgerConnections` option in `network` section of configuration file. The default value is 20. This limit is independent of 
+This feature adds a dedicated connection pool reserved for connecting to forger nodes. The size of the dedicated connection pool is configured by `maxForgerConnections` option in `network` section of configuration file. The default value is 20. This limit  works in addition to the one defined in the property network.maxOutgoingConnections.
 
 Configuration option `isForgerNode` in the `network` section has to be set to `true` to indicate that the node is a forger and other nodes should prioritize connecting with it. Default value is `false`.
 
@@ -66,4 +67,4 @@ The intent of this feature is to minimize block propagation time between forgers
     - other performance improvements
 
 ---
-Full [Changelog](/CHANGELOG.md) file here
+Full [Changelog](/CHANGELOG.md) file here

doc/release/0.8.1.md

@@ -0,0 +1,13 @@
+# Release notes - version 0.8.1
+
+---
+
+## Notes about new/updated Features
+
+eth RPC call eth_gasPrice  now returns an estimation of the gasPrice more accurate.\
+The previous one was overestimating the minimum gasPrice needed in some corner cases.\
+Following the new endpoint estimation, will be possible to pay lower average fees than before.\
+The update will be immediately visible, not requirying an hardfork activation.
+
+---
+Full [Changelog](/CHANGELOG.md) file here