Add LGPU with single GPU backend (#499)

* add LGPU cpp layer

* update measurement

* add openmp to adjgpu

* Auto update version

* Add support for building multiple backend simulators (#497)

* Add PL_BACKEND_LIST

* Update the support

* Exclude Python bindings

* Update HermitianObs name scope conflicts

* Auto update version

* Cleanup

* Update CI to build and check C++ tests of multiple backends (Linux)

* Update changelog

* Update .github/workflows/tests_linux.yml

* Apply code review suggestions

* Update .github/workflows/tests_linux.yml

* add python layer & isingxy gate in the cpp layer

* add batched adjoint method

* Update DefaultQubit to DefaultQubitLegacy (#500)

* Update DefaultQubit to DefaultQubitLegacy

* Update changelog

* update pylint disable on fallback

* Auto update version

* add batch support for adjoint method

* add gitignore

* tidy up code

* Auto update version

* make format

* revert complexT delete in LKokkosBingds

* make format

* update based on tidy

* fix tidy format

* add_gpu_runners_tests

* add cuquantum_sdk path to ci workflow

* debug

* add path to cuquantum sdk

* add python layer tests in ci workflow

* ci tests

* quick fix

* skip pr ci for some workflows

* quick fix

* quick fix

* update python ci tests

* remove dependency on lightning_qubit in ci

* fix directory

* fix directory

* quick fix

* quick fix

* test for cuda-12

* update measurement

* updata cu12 workflows

* add getDataVector support to LQubitRaw

* install lightning.qubit before lightning.gpu in ci

* update test_obs

* activate all CI checks

* quick fix

* tidy up code

* make format

* update ci for more tests

* tidy up code

* tidy up code

* tidy up code

* make format

* fix for codecov

* codecov fix

* quick fix

* quick fix

* quick fix

* quick test

* fix test

* fix tests

* another quick fix

* coverage fix

* update ci tests

* update ci for no binary

* codecov fix

* update adj tests for no binary case

* update python layer tests

* fix codecov

* make format

* revert to cu11

* enable more py tests

* update CI

* upload codecov ci

* add more tests for statevectorcudamanaged

* add more unit tests

* add more tests

* make format

* add more cpp tests

* skip cpp tests pauli param gates

* make format

* add more files to gitignore

* Auto update version

* Trigger CI

* update gpu runner

* quick fix

* update fix

* fix for fp32 support in expval calculation

* quick fix

* fix for cray_mpich_serialize_py

* copy to move for hamiltonian operation

* resolve comments py layer

* remove omp support in LGPU

* update version

* Auto update version

* fix based on comments

* Add L-GPU and L-Kokkos as package extras (#515)

* Add L-GPU and L-Kokkos as package extras

* Auto update version

* Update changelog

* Temp enable the x86 wheel cache

* Return wheel storage functionality to normal

* Update readme

* Auto update version

* Trigger CI

* Update README.rst
---------

* Auto update version

* make format

* remove sparseH

* remove applyHostMatrixGate

* Add wheel_linux_x86_64_cu11.yml (#517)

* Add wheel_linux_x86_64_cu11.yml

* echo COMPILER=g++ >>

* python3.9 => python

* reinstall g++11

* Try that

* Use env vars for versions.

* Fix var syntax.

* Hardcode versions

* Fix custatevec-cu11

* Revert triggers.

* Update changelog [skip ci]

* resolve more comments

* add more tests to non_param gates

* resolve cpp tests comments

* remove unused methods in measurement class

* remove unused methods

* resolve more comments

* add changelog and matrixhasher

* quick update

* tidy up code and add more tests

* add more unit tests for algorithm

* more unit tests in py layer

* add adjoint tests for sparseH

* make format

* quick fix

* quick fix

* another fix

* fix

* fix codecov complaints

* resolve comments

* add appropriate license

* tidy up

* quick fix

.github/CHANGELOG.md

@@ -2,6 +2,12 @@
 
 ### New features since last release
 
+* Integrate Lightning-GPU into the Lightning monorepo. The new backend is named `lightning_gpu` and includes all single-GPU features.
+  [(#499)] (https://github.com/PennyLaneAI/pennylane-lightning/pull/499)
+
+* Build Linux wheels for Lightning-GPU (CUDA-11).
+  [(#517)](https://github.com/PennyLaneAI/pennylane-lightning/pull/517)
+
 * Add `Dockerfile` in `docker` and `make docker` workflow in `Makefile`. The Docker images and documentation are available on [DockerHub](https://hub.docker.com/repository/docker/pennylaneai/pennylane).
   [(#496)](https://github.com/PennyLaneAI/pennylane-lightning/pull/496)
 
@@ -78,7 +84,7 @@
 
 This release contains contributions from (in alphabetical order):
 
-Ali Asadi, Amintor Dusko, Vincent Michaud-Rioux, Lee J. O'Riordan
+Ali Asadi, Amintor Dusko, Vincent Michaud-Rioux, Lee J. O'Riordan, Shuli Shu
 
 ---
 

.github/workflows/tests_gpu_cu11.yml

@@ -0,0 +1,325 @@
+name: Testing::Linux::x86_64 (Lightning-GPU)
+on:
+  pull_request:
+  push:
+    branches:
+      - master
+
+env:
+  CI_CUDA_ARCH: 86
+  COVERAGE_FLAGS: "--cov=pennylane_lightning --cov-report=term-missing --cov-report=xml:./coverage.xml --no-flaky-report -p no:warnings --tb=native"
+  GCC_VERSION: 11
+  TF_VERSION: 2.10.0
+  TORCH_VERSION: 1.11.0+cpu
+
+concurrency:
+  group: tests_gpu-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  builddeps:
+    runs-on:
+      - self-hosted
+      - ubuntu-22.04
+      - gpu
+
+    strategy:
+      max-parallel: 1
+      matrix:
+        os: [ubuntu-22.04]
+        pl_backend: ["lightning_gpu"]
+
+    steps:
+      - name: Validate GPU version and installed compiler
+        run: |
+          source /etc/profile.d/modules.sh
+          module use /opt/modules
+          module load cuda/11.8
+          echo "${PATH}" >> $GITHUB_PATH
+          echo "LD_LIBRARY_PATH=${LD_LIBRARY_PATH}" >> $GITHUB_ENV
+          nvcc --version
+          nvidia-smi
+
+  cpptestswithLGPU_cu11:
+    needs: [builddeps]
+    strategy:
+      matrix:
+        os: [ubuntu-22.04]
+        pl_backend: ["lightning_gpu"]
+
+    name: C++ tests (Lightning-GPU)
+    runs-on: ${{ matrix.os }}
+
+    steps:
+      - name: Validate GPU version and installed compiler
+        run: |
+          source /etc/profile.d/modules.sh
+          module use /opt/modules
+          module load cuda/11.8
+          echo "${PATH}" >> $GITHUB_PATH
+          echo "LD_LIBRARY_PATH=${LD_LIBRARY_PATH}" >> $GITHUB_ENV
+          nvcc --version
+          nvidia-smi
+
+      - name: Checkout PennyLane-Lightning-GPU
+        uses: actions/checkout@v3
+        with:
+          path: main
+          fetch-depth: 2
+
+      - uses: actions/setup-python@v4
+        name: Install Python
+        with:
+          python-version: '3.9'
+
+      - name: Setup Python virtual environment
+        id: setup_venv
+        env:
+          VENV_NAME: ${{ github.workspace }}/venv_${{ steps.setup_python.outputs.python-version }}_${{ github.sha }}
+        run: |
+          # Clear any pre-existing venvs
+          rm -rf venv_*
+
+          # Create new venv for this workflow_run
+          python --version
+          python -m venv ${{ env.VENV_NAME }}
+
+          # Add the venv to PATH for subsequent steps
+          echo ${{ env.VENV_NAME }}/bin >> $GITHUB_PATH
+
+          # Adding venv name as an output for subsequent steps to reference if needed
+          echo "venv_name=${{ env.VENV_NAME }}" >> $GITHUB_OUTPUT
+
+      - name: Display Python-Path
+        id: python_path
+        run: |
+          py_path=$(which python)
+          echo "Python Interpreter Path => $py_path"
+          echo "python=$py_path" >> $GITHUB_OUTPUT
+          
+          pip_path=$(which python)
+          echo "PIP Path => $pip_path"
+          echo "pip=$pip_path" >> $GITHUB_OUTPUT
+
+      - name: Install required packages
+        run: |
+          python -m pip install ninja cmake custatevec-cu11
+      
+      - name: Install Latest PennyLane
+        if: inputs.pennylane-version == 'latest'
+        run: python -m pip install git+https://github.com/PennyLaneAI/pennylane.git@master
+
+      - name: Build and run unit tests
+        run: |
+          cd main
+          cmake . -BBuild \
+            -DCUQUANTUM_SDK=$(python -c "import site; print( f'{site.getsitepackages()[0]}/cuquantum')")\
+            -DBUILD_TESTS=ON \
+            -DENABLE_PYTHON=OFF \
+            -DPL_BACKEND=${{ matrix.pl_backend }} \
+            -DCMAKE_CXX_COMPILER=$(which g++-$GCC_VERSION) \
+            -DENABLE_COVERAGE=ON \
+            -G Ninja
+          cmake --build ./Build
+          cd ./Build
+          mkdir -p ./tests/results_${{ github.job }}_${{ matrix.pl_backend }}
+          for file in *runner ; do ./$file --order lex --reporter junit --out ./tests/results_${{ github.job }}_${{ matrix.pl_backend }}/report_$file.xml; done;
+          lcov --directory . -b ../pennylane_lightning/core/src --capture --output-file coverage.info
+          lcov --remove coverage.info '/usr/*' --output-file coverage.info
+          mv coverage.info coverage-${{ github.job }}-${{ matrix.pl_backend }}.info
+      - name: Upload test results
+        uses: actions/upload-artifact@v3
+        if: always()
+        with:
+          name: ubuntu-tests-reports-${{ github.job }}_${{ matrix.pl_backend }}
+          path: ./main/Build/tests/results_${{ github.job }}_${{ matrix.pl_backend }}
+          if-no-files-found: error
+
+      - name: Upload code coverage results
+        uses: actions/upload-artifact@v3
+        with:
+          name: ubuntu-codecov-results-cpp
+          path: ./main/Build/coverage-${{ github.job }}-${{ matrix.pl_backend }}.info
+          if-no-files-found: error
+
+  pythontestswithLGPU:
+    needs: [builddeps]
+    strategy:
+      matrix:
+        os: [ubuntu-22.04]
+        pl_backend: ["lightning_gpu"]
+        default_backend: ["lightning_qubit"]
+
+    name: Python tests with LGPU
+    runs-on: ${{ matrix.os }}
+
+    steps:
+      - name: Validate GPU version and installed compiler
+        run: |
+          source /etc/profile.d/modules.sh
+          module use /opt/modules
+          module load cuda/11.8
+          echo "${PATH}" >> $GITHUB_PATH
+          echo "LD_LIBRARY_PATH=${LD_LIBRARY_PATH}" >> $GITHUB_ENV
+          nvcc --version
+          nvidia-smi
+
+      - name: Checkout PennyLane-Lightning
+        uses: actions/checkout@v3
+        with:
+          path: main
+
+      - uses: actions/setup-python@v4
+        name: Install Python
+        with:
+          python-version: '3.9'
+
+      - name: Setup Python virtual environment
+        id: setup_venv
+        env:
+          VENV_NAME: ${{ github.workspace }}/venv_${{ steps.setup_python.outputs.python-version }}_${{ github.sha }}
+        run: |
+          # Clear any pre-existing venvs
+          rm -rf venv_*
+
+          # Create new venv for this workflow_run
+          python --version
+          python -m venv ${{ env.VENV_NAME }}
+
+          # Add the venv to PATH for subsequent steps
+          echo ${{ env.VENV_NAME }}/bin >> $GITHUB_PATH
+
+          # Adding venv name as an output for subsequent steps to reference if needed
+          source ${{ env.VENV_NAME }}/bin/activate
+          echo "venv_name=${{ env.VENV_NAME }}" >> $GITHUB_OUTPUT
+
+      - name: Display Python-Path
+        id: python_path
+        run: |
+          py_path=$(which python)
+          echo "Python Interpreter Path => $py_path"
+          echo "python=$py_path" >> $GITHUB_OUTPUT
+          
+          pip_path=$(which python)
+          echo "PIP Path => $pip_path"
+          echo "pip=$pip_path" >> $GITHUB_OUTPUT
+
+      - name: Install Latest PennyLane
+        # We want to install the latest PL on non workflow_call events
+        if: inputs.pennylane-version == 'latest'  || inputs.pennylane-version == ''
+        run: python -m pip install git+https://github.com/PennyLaneAI/pennylane.git@master
+
+      - name: Install required packages
+        run: |
+          python -m pip install pip~=22.0
+          python -m pip install ninja cmake custatevec-cu11 pytest pytest-mock flaky pytest-cov
+
+      - name: Build and install package
+        env: 
+          CUQUANTUM_SDK: $(python -c "import site; print( f'{site.getsitepackages()[0]}/cuquantum')")
+        run: |
+          cd main
+          rm -rf build
+          CMAKE_ARGS="-DPL_BACKEND=${{ matrix.default_backend }} -DENABLE_PYTHON=ON -DCMAKE_CXX_COMPILER=$(which g++-$GCC_VERSION)" \
+          python -m pip install -e . -vv
+          rm -rf build
+
+          rm -rf build
+          CMAKE_ARGS="-DPL_BACKEND=${{ matrix.pl_backend }} -DENABLE_PYTHON=ON -DCMAKE_CXX_COMPILER=$(which g++-$GCC_VERSION) -DPython_EXECUTABLE=${{ steps.python_path.outputs.python }}" \
+          python -m pip install -e . -vv
+
+      - name: Run PennyLane-Lightning-GPU unit tests
+        if: ${{ matrix.pl_backend != 'all'}}
+        env: 
+          OMP_NUM_THREADS: 1
+          OMP_PROC_BIND: false
+        run: |
+          cd main/
+          DEVICENAME=`echo ${{ matrix.pl_backend }} | sed "s/_/./g"`
+          pl-device-test --device ${DEVICENAME} --skip-ops --shots=20000 $COVERAGE_FLAGS --cov-append 
+          pl-device-test --device ${DEVICENAME} --shots=None --skip-ops $COVERAGE_FLAGS --cov-append
+          PL_DEVICE=${DEVICENAME} python -m pytest tests/ $COVERAGE_FLAGS 
+          mv coverage.xml coverage-${{ github.job }}-${{ matrix.pl_backend }}.xml
+
+      - name: Install all backend devices
+        if: ${{ matrix.pl_backend == 'all' }}
+        env: 
+          CUQUANTUM_SDK: $(python -c "import site; print( f'{site.getsitepackages()[0]}/cuquantum')")
+        run: |
+          cd main
+          rm -rf build
+          CMAKE_ARGS="-DPL_BACKEND=${{matrix.default_backend}} -DENABLE_PYTHON=ON -DCMAKE_CXX_COMPILER=$(which g++-$GCC_VERSION)" \
+          python -m pip install -e . -vv
+          rm -rf build
+          
+          CMAKE_ARGS="-DPL_BACKEND=${{ matrix.pl_backend }} -DENABLE_PYTHON=ON -DCMAKE_CXX_COMPILER=$(which g++-$GCC_VERSION)" \
+          python -m pip install -e . -vv
+
+      - name: Run PennyLane-Lightning unit tests for lightning.qubit with all devices installed
+        if: ${{ matrix.pl_backend == 'all' }}
+        env: 
+          OMP_NUM_THREADS: 1
+          OMP_PROC_BIND: false
+        run: |
+          cd main/
+          PL_DEVICE=lightning.qubit python -m pytest tests/ $COVERAGE_FLAGS
+          pl-device-test --device lightning.qubit --skip-ops --shots=20000 $COVERAGE_FLAGS --cov-append
+          pl-device-test --device lightning.qubit --shots=None --skip-ops $COVERAGE_FLAGS --cov-append
+          PL_DEVICE=lightning.gpu python -m pytest tests/ $COVERAGE_FLAGS
+          pl-device-test --device lightning.gpu --skip-ops --shots=20000 $COVERAGE_FLAGS --cov-append
+          pl-device-test --device lightning.gpu --shots=None --skip-ops $COVERAGE_FLAGS --cov-append
+          mv coverage.xml coverage-${{ github.job }}-${{ matrix.pl_backend }}.xml
+
+      - name: Upload code coverage results
+        uses: actions/upload-artifact@v3
+        with:
+          name: ubuntu-codecov-results-python
+          path: ./main/coverage-${{ github.job }}-${{ matrix.pl_backend }}.xml
+
+  upload-to-codecov-linux-python:
+    needs: [pythontestswithLGPU]
+    name: Upload coverage data to codecov
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v3
+
+      - name: Download coverage reports
+        uses: actions/download-artifact@v3
+        with:
+          name: ubuntu-codecov-results-python
+
+      - name: Upload to Codecov
+        uses: codecov/codecov-action@v3
+        with:
+          fail_ci_if_error: true
+          token: ${{ secrets.CODECOV_TOKEN }}
+
+  upload-to-codecov-linux-cpp:
+    needs: [cpptestswithLGPU_cu11]
+    name: Upload coverage data to codecov
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v3
+
+      - name: Download coverage reports
+        uses: actions/download-artifact@v3
+        with:
+          name: ubuntu-codecov-results-cpp
+
+      - name: Upload to Codecov
+        uses: codecov/codecov-action@v3
+        with:
+          fail_ci_if_error: true
+          token: ${{ secrets.CODECOV_TOKEN }}
+
+      - name: Cleanup
+        if: always()
+        run: |
+          rm -rf ${{ steps.setup_venv.outputs.venv_name }}
+          rm -rf *
+          rm -rf .git
+          rm -rf .gitignore
+          rm -rf .github
+          pip cache purge

.github/workflows/tests_without_binary.yml

@@ -31,7 +31,7 @@ jobs:
     strategy:
       matrix:
         os: [ubuntu-22.04]
-        pl_backend: ["lightning_qubit", "lightning_kokkos"]
+        pl_backend: ["lightning_qubit", "lightning_kokkos", "lightning_gpu"]
 
     steps:
       - name: Checkout PennyLane-Lightning
@@ -66,7 +66,7 @@ jobs:
           python -m pip uninstall -y pennylane && python -m pip install -U pennylane
 
       - name: Install the pennylane_lightning package
-        if: ${{ matrix.pl_backend == 'lightning_kokkos'}}
+        if: ${{ matrix.pl_backend == 'lightning_kokkos' || matrix.pl_backend == 'lightning_gpu'}}
         run: |
           cd main
           SKIP_COMPILATION=True PL_BACKEND="lightning_qubit" pip install -e . -vv