Add build-time flag to enable OpenMP in Lightning gate kernels

[mlxd]

Before submitting

Please complete the following checklist when submitting a PR:

• All new features must include a unit test.
  If you've fixed a bug or added code that should be tested, add a test to the
  tests directory!

• All new functions and code must be clearly commented and documented.
  If you do make documentation changes, make sure that the docs build and
  render correctly by running make docs.

• Ensure that the test suite passes, by running make test.

• Add a new entry to the .github/CHANGELOG.md file, summarizing the
  change, and including a link back to the PR.

• Ensure that code is properly formatted by running make format.

When all the above are checked, delete everything above the dashed
line and fill in the pull request template.

---

Context: The default threading behaviour for lightning.qubit is to assume single threaded execution at the gate level, but allow multi-threaded execution over observables with the adjoint differentiation pipeline. This allows us to always ensure the best (SIMD) gate kernel performance, but limits the performance for situations with deep circuits on large CPUs (high core count, and big caches). This PR adds a compile-time flag, defaulting to off, to allow lightning.qubit to add OpenMP supports to all kernels (LM, AVX2, and AVX512). this will allow for better performance tuning on HPC systems, while preserving the default behavior during release.

Description of the Change: As above. Option enabled through a LQ_ENABLE_KERNEL_OMP CMake option in the LQ backend, which is default disabled.
Benefits: Allows better performance tuning on HPC systems.

Possible Drawbacks: If on by default, OMP overhead may slow down small circuits. Testing on CI will show no benefit (1 physical core only), so no additional tests added.

Related GitHub Issues:

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (df43949) 98.18% compared to head (ed91025) 98.67%.
Report is 1 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #510      +/-   ##
==========================================
+ Coverage   98.18%   98.67%   +0.48%     
==========================================
  Files          10      169     +159     
  Lines        1210    24308   +23098     
==========================================
+ Hits         1188    23985   +22797     
- Misses         22      323     +301     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[maliasadi]

Thanks @mlxd! LGTM 🎉 I have a few questions regarding the support, but there is nothing major to address from my side.

[AmintorDusko]

LGTM!

[maliasadi]

🪂

[multiphaseCFD]

Thanks @mlxd for the nice work!

[mlxd]

Thanks for all of the comments folks!