[GLUTEN-6893][VL] Change to using native libs generated by vcpkg build in Gluten scala tests

dev/ci-velox-buildshared-centos-8.sh

@@ -10,9 +10,8 @@ yum install sudo patch java-1.8.0-openjdk-devel wget -y
 wget https://downloads.apache.org/maven/maven-3/3.8.8/binaries/apache-maven-3.8.8-bin.tar.gz
 tar -xvf apache-maven-3.8.8-bin.tar.gz && mv apache-maven-3.8.8 /usr/lib/maven
 export PATH="${PATH}:/usr/lib/maven/bin"
+echo "PATH=${PATH}:/usr/lib/maven/bin" >> $GITHUB_ENV
 
 source /opt/rh/gcc-toolset-9/enable
 ./dev/builddeps-veloxbe.sh --run_setup_script=OFF --enable_ep_cache=OFF --build_tests=ON \
     --build_examples=ON --build_benchmarks=ON --build_protobuf=ON
-
-cd ./cpp/build && ctest -V

dev/ci-velox-buildstatic-centos-7.sh

@@ -3,8 +3,6 @@
 set -e
 
 source /opt/rh/devtoolset-9/enable
-cd $GITHUB_WORKSPACE/
-source ./dev/vcpkg/env.sh
 export NUM_THREADS=4
-./dev/builddeps-veloxbe.sh --build_tests=OFF --build_benchmarks=OFF --build_arrow=OFF --enable_s3=ON \
-                           --enable_gcs=ON --enable_hdfs=ON --enable_abfs=ON
+./dev/builddeps-veloxbe.sh --enable_vcpkg=ON --build_arrow=OFF --build_tests=OFF --build_benchmarks=OFF \
+                           --build_examples=OFF --enable_s3=ON --enable_gcs=ON --enable_hdfs=ON --enable_abfs=ON

dev/vcpkg/ports/simdjson/vcpkg.json

@@ -17,8 +17,7 @@
   "default-features": [
     "deprecated",
     "exceptions",
-    "threads",
-    "utf8-validation"
+    "threads"
   ],
   "features": {
     "deprecated": {

docs/developers/UsingGperftoolsInCH.md

@@ -1,3 +1,10 @@
+---
+layout: page
+title: Debug for CH Backend with gpertools Tool
+nav_order: 11
+has_children: true
+parent: /developer-overview/
+---
 We need using gpertools to find the memory or CPU issue. That's what this document is about.
 
 ## Install gperftools

docs/developers/UsingJemallocWithCH.md

@@ -1,3 +1,10 @@
+---
+layout: page
+title: Use Jemalloc for CH Backend
+nav_order: 12
+has_children: true
+parent: /developer-overview/
+---
 We need using jemalloc to find the memory issue. That's what this document is about.
 
 ## Change code of jemalloc

docs/developers/VeloxUDF.md

@@ -1,3 +1,10 @@
+---
+layout: page
+title: Velox UDF and UDAF
+nav_order: 13
+has_children: true
+parent: /developer-overview/
+---
 # Velox User-Defined Functions (UDF) and User-Defined Aggregate Functions (UDAF)
 
 ## Introduction

docs/developers/velox-backend-CI.md

@@ -0,0 +1,24 @@
+---
+layout: page
+title: Velox Backend CI
+nav_order: 14
+parent: Developer Overview
+---
+# Velox Backend CI
+
+GHA workflows are defined under `.github/workflows/`.
+
+## Docker Build
+We have a weekly job to build a docker based on `Dockerfile.gha` for CI verification, defined in `docker_image.yml`.
+
+## Vcpkg Caching
+Gluten main branch is pulled down during docker build. And vcpkg will cache binary data of all dependencies defined under dev/vcpkg.
+These binary data is cached into `/var/cache/vcpkg` and CI job can re-use them in new build. By setting `VCPKG_BINARY_SOURCES=clear` in env.,
+reusing vcpkg cache can be disabled.
+
+## Arrow Libs Pre-installation
+Arrow libs are pre-installed in docker, assuming they are not actively changed, then not necessarily to be re-built every time.
+
+## Updating Docker Image
+Two GitHub secrets `DOCKERHUB_USER` & `DOCKERHUB_TOKEN` can be used to push docker image to docker hub: https://hub.docker.com/r/apache/gluten/tags.
+Note GitHub secrets are not retrievable in PR from forked repo.