Merge staging (#438)

* trigger first staging build

* Inital Hack of NEO4j DB creation

* Added: Neo4j docker run

* Added NEO_DB Class

* Create Inital Path API response

* Build basic UI for testing

* Mock score and WIP UI

* implement scoring and basic tests

* scoring passing tests

* Update the UI

* Updated: Dropdowns are now dynamic

* Localise neo4j

* Added Navigation method

* Add share and nav links

* readme improvement

* Hide table on new search

* Optermise query to remove relates to

* Get duel running method working and show empty values

* Refactor grouping & scoring code locations

* Add colour to strength raiting

* Add gap analysis tests

* Short drop down list

* Styling improvements and legends

* rm scikit version pin

* loosen requirement in sqlalchemy

* more deps changes

* fix deps

* Disable auto db pop (#372)

* Add gap analysis tests

* Short drop down list

* Disable CRE_Graph auto population on startup

* revert conflict mess with tests

* Migrate Neo4j population to seperate function

* Add CLI population command for DB

* linting fixes

* fix pillow

* Use python 3.11.4 for CI

* Parse cre_dep object from neo4j Node

* Standardise Gap Analysis API Object

* Fix bad parse field

* Use Standard display format and create GA type

* GA: Show all strong links by default (or min of 3)

* Better route display

* GA add support for tools

* Fix existing tests

* Add parsing tests

* make parse_node static

* Use Enum types

* Better route display

* linting

* Minimal GA styling

* Fix: GA share link spaces

* linting

* Added: Direct link grade and removed external link icon

* Fix: linked via standard was possible

* Added: Per link score penalty

* Fix: GA caching not taking queries (#412)

* Make Direct Dark Green (#411)

* Make Direct Dark Green

* Update GapAnalysis.tsx

Signed-off-by: John Harvey <[email protected]>

---------

Signed-off-by: John Harvey <[email protected]>

* GA Reduce related punishment (#416)

* Rename gap to map (#419)

* GA make strong status <=2

* Rename URL and UI from Gap to Map analysis

* GA fix bad links (#417)

Fix bad links in GA (use standard function)

* Speed up CICD (#418)

* Rename share link (#415)

* Neo4j ORM (#408)

* Make Direct Dark Green

* Update GapAnalysis.tsx

Signed-off-by: John Harvey <[email protected]>

* Trial implementation of Neo4J ORM

* Populate Neo4j DB using model

* More progress in conversion

* get ORM working

* Fix tests

* Remove old NeoDB driver connection

* Fix: id issue

* Fix: Tags strings splitting

* linting fixes

* Remove  NeomodelPath reference

* Revert rebase bug

---------

Signed-off-by: John Harvey <[email protected]>

* Remove BOLT from neo4j Setup requirement (#428)

* pin neomodel to a decent version

* relax six requirements

* Sort Base standard alphabetically (#427)

* Use allShortestPaths over shortestPath (#431)

* Added Cross-cutting concerns to GA blacklist (#430)

* Added Cross-cutting concerns to GA blacklist

* GA page linting

* Remove double brackets

* Fix GA loading disappearing

* rename variable

* Revert "Use allShortestPaths over shortestPath (#431)"

This reverts commit e33d13d.

* Precompute map analysis (#433)

* add time metrics

* progress

* Revert "Revert "Use allShortestPaths over shortestPath (#431)""

This reverts commit 167828c.

* map analysis through redis

* new reqs

* add worker actually

* abort if background job fails

* lint

* minor cleanup

* move hash to utils and make db method write to redis optionally

* make ga job results shared among clients

* lint

* Split ga response (#434)

* add time metrics

* map analysis through redis

* move hash to utils and make db method write to redis optionally

* lint

* Added: Front end support for weaker links calls

* Added: DB seperation for smaller inital response and cache extras

* Added: DB side of part split response logic

* Refactor and tests

---------

Co-authored-by: Spyros <[email protected]>

* skip returning ids for failed jobs

* add optional tls to redis, heroku requirement

* add optional tls to redis, heroku requirement

* add optional tls to redis, heroku requirement

* add optional tls to redis, heroku requirement

* add optional tls to redis, heroku requirement

* add optional tls to redis, heroku requirement

* use db to cache gap analysis results instead of redis (#435)

* use db to cache gap analysis results instead of redis

* lint

* typo

* fix 1 test, fix bugs, make cache key into its own function

* lint

* fix-mock

* migration

* Fix: bad ref

* fix tests

* lint

* minor changes

* lint

---------

Co-authored-by: john681611 <[email protected]>

* Get e2e tests reporting correctly (#395)

* try pip cacheing

* try yarn cache

* try background service method

* e2e test fixes

* tmp increase timeouts for e2e tests due to new neo4j experiments

* fix e2e tests

* make e2e workflow match makefile

* Revert actions simplification

* increase timing

* balance timings

---------

Signed-off-by: Spyros <[email protected]>
Co-authored-by: Spyros <[email protected]>
Co-authored-by: Spyros <[email protected]>

* minor fixes

* fix-lint

* fix caching

* fix e2e?

---------

Signed-off-by: John Harvey <[email protected]>
Signed-off-by: Spyros <[email protected]>
Co-authored-by: john681611 <[email protected]>
Co-authored-by: John Harvey <[email protected]>

.github/workflows/e2e.yml

@@ -4,10 +4,25 @@ jobs:
   build:
     name: Test-e2e
     runs-on: ubuntu-latest
+    timeout-minutes: 10
     steps:
       - name: Check out code
         uses: actions/checkout@v2
+      - uses: actions/setup-python@v4
+        with:
+          python-version: '3.11.4' 
+          cache: 'pip'
+      - uses: actions/setup-node@v3
+        with:
+          cache: 'yarn'
       - name: Install python dependencies
         run: sudo apt-get update && sudo apt-get install -y python3-setuptools python3-pip chromium-browser libgbm1 && make install-deps 
-      - name: Test-e2e
-        run: make e2e
+      - name: Run App in background
+        run: |
+          yarn build
+          [ -d "./venv" ] && . ./venv/bin/activate
+          export FLASK_APP=./cre.py
+          export FLASK_CONFIG=development
+          FLASK_CONFIG=development flask run& 
+          yarn test:e2e
+          exit $?

.github/workflows/test.yml

@@ -7,9 +7,14 @@ jobs:
     steps:
       - name: Check out code
         uses: actions/checkout@v2
+      - uses: actions/setup-python@v4
+        with:
+          python-version: '3.11.4' 
+          cache: 'pip'
+      - uses: actions/setup-node@v3
+        with:
+          cache: 'yarn'
       - name: Install python dependencies
         run: sudo apt-get update && sudo apt-get install -y python3-setuptools python3-pip && make install-deps
       - name: Test
         run: make test
-      - name: Test-e2e
-        run: make e2e

.gitignore

@@ -16,6 +16,7 @@ Vagrantfile
 
 ## act secrets
 .secrets/
+.env
 
 ### conventions ###
 venv/
@@ -30,4 +31,8 @@ yarn-error.log
 coverage/
 
 ### Dev db
-standards_cache.sqlite
+standards_cache.sqlite
+
+### Neo4j
+neo4j/
+.neo4j/

Makefile

@@ -1,21 +1,36 @@
 
 .ONESHELL:
 
-.PHONY: dev-run run test covers install-deps dev docker lint frontend clean all
+.PHONY: run test covers install-deps dev docker lint frontend clean all
 
 prod-run:
 	cp cres/db.sqlite standards_cache.sqlite; gunicorn cre:app --log-file=-
 
-dev-run:
-	. ./venv/bin/activate && FLASK_APP=cre.py FLASK_CONFIG=development flask run
+docker-neo4j:
+	docker start cre-neo4j 2>/dev/null   || docker run -d --name cre-neo4j --env NEO4J_PLUGINS='["apoc"]'  --env NEO4J_AUTH=neo4j/password --volume=`pwd`/.neo4j/data:/data --volume=`pwd`/.neo4j/logs:/logs --workdir=/var/lib/neo4j -p 7474:7474 -p 7687:7687 neo4j
+
+docker-redis:
+	docker start redis-stack 2>/dev/null || docker run -d --name redis-stack -p 6379:6379 -p 8001:8001 redis/redis-stack:latest
+
+start-containers: docker-neo4j docker-redis
+
+start-worker:
+	. ./venv/bin/activate
+	FLASK_APP=`pwd`/cre.py python cre.py --start_worker
+
+dev-flask:
+	. ./venv/bin/activate
+	FLASK_APP=`pwd`/cre.py  FLASK_CONFIG=development flask run
+
 e2e:
 	yarn build
 	[ -d "./venv" ] && . ./venv/bin/activate
 	export FLASK_APP=$(CURDIR)/cre.py
 	export FLASK_CONFIG=development
-	FLASK_CONFIG=development flask run&
-
+	flask run &
+	sleep 5
 	yarn test:e2e
+	sleep 20
 	killall yarn
 	killall flask
 
@@ -79,4 +94,8 @@ import-all:
 	[ -d "./venv" ] && . ./venv/bin/activate
 	rm -rf standards_cache.sqlite && make migrate-upgrade && export FLASK_APP=$(CURDIR)/cre.py && python cre.py --add --from_spreadsheet https://docs.google.com/spreadsheets/d/1eZOEYgts7d_-Dr-1oAbogPfzBLh6511b58pX3b59kvg && python cre.py --generate_embeddings && python cre.py --zap_in --cheatsheets_in --github_tools_in  --capec_in --owasp_secure_headers_in --pci_dss_4_in --juiceshop_in &&	python cre.py --generate_embeddings
 
+import-neo4j:
+	[ -d "./venv" ] && . ./venv/bin/activate
+	export FLASK_APP=$(CURDIR)/cre.py && python cre.py --populate_neo4j_db
+
 all: clean lint test dev dev-run

Procfile

@@ -1 +1,2 @@
-web: gunicorn cre:app --log-file=-
+web: gunicorn cre:app --log-file=-g
+worker: FLASK_APP=`pwd`/cre.py python cre.py --start_worker

README.md

@@ -60,11 +60,22 @@ To add a remote spreadsheet to your local database you can run
 <pre>python cre.py --add --from_spreadsheet < google sheets url></pre>
 
 To run the web application for development you can run
-<pre>make dev-run</pre>
+<pre>
+$ make start-containers
+$ make start-worker 
+
+# in a seperate shell
+$ make dev-flask
+</pre>
 
 Alternatively, you can use the dockerfile with
 <pre>make docker && make docker-run</pre>
 
+Some features like Gap Analysis require a neo4j DB running, you can start this with
+<pre>make docker-neo4j</pre>
+enviroment varaibles for app to connect to neo4jDB (default):
+- NEO4J_URL (neo4j//neo4j:password@localhost:7687)
+
 To run the web application for production you need gunicorn and you can run from within the cre_sync dir
 <pre>make prod-run</pre>
 
@@ -84,4 +95,4 @@ Please see [Contributing](CONTRIBUTING.md) for contributing instructions
 
 Roadmap
 ---
-For a roadmap of what we would like to be done please see the [issues](https://github.com/OWASP/common-requirement-enumeration/issues).
+For a roadmap of what we would like to be done please see the [issues](https://github.com/OWASP/common-requirement-enumeration/issues).

application/cmd/cre_main.py

@@ -17,7 +17,6 @@
 from application.utils.external_project_parsers import (
     capec_parser,
     cwe,
-    ccmv3,
     ccmv4,
     cheatsheets_parser,
     misc_tools_parser,
@@ -375,14 +374,6 @@ def run(args: argparse.Namespace) -> None:  # pragma: no cover
     if args.export:
         cache = db_connect(args.cache_file)
         cache.export(args.export)
-    if args.csa_ccm_v3_in:
-        ccmv3.parse_ccm(
-            ccmFile=sheet_utils.readSpreadsheet(
-                alias="",
-                url="https://docs.google.com/spreadsheets/d/1b5i8OV919aiqW2KcYWOQvkLorL1bRPqjthJxLH0QpD8",
-            ),
-            cache=db_connect(args.cache_file),
-        )
     if args.csa_ccm_v4_in:
         ccmv4.parse_ccm(
             ccmFile=sheet_utils.readSpreadsheet(
@@ -426,6 +417,12 @@ def run(args: argparse.Namespace) -> None:  # pragma: no cover
         generate_embeddings(args.cache_file)
     if args.owasp_proj_meta:
         owasp_metadata_to_cre(args.owasp_proj_meta)
+    if args.populate_neo4j_db:
+        populate_neo4j_db(args.cache_file)
+    if args.start_worker:
+        from application.worker import start_worker
+
+        start_worker(args.cache_file)
 
 
 def db_connect(path: str):
@@ -530,3 +527,11 @@ def owasp_metadata_to_cre(meta_file: str):
     },
     """
     raise NotImplementedError("someone needs to work on this")
+
+
+def populate_neo4j_db(cache: str):
+    logger.info(f"Populating neo4j DB: Connecting to SQL DB")
+    database = db_connect(path=cache)
+    logger.info(f"Populating neo4j DB: Populating")
+    database.neo_db.populate_DB(database.session)
+    logger.info(f"Populating neo4j DB: Complete")