Merge staging (#439)

* trigger first staging build

* Inital Hack of NEO4j DB creation

* Added: Neo4j docker run

* Added NEO_DB Class

* Create Inital Path API response

* Build basic UI for testing

* Mock score and WIP UI

* implement scoring and basic tests

* scoring passing tests

* Update the UI

* Updated: Dropdowns are now dynamic

* Localise neo4j

* Added Navigation method

* Add share and nav links

* readme improvement

* Hide table on new search

* Optermise query to remove relates to

* Get duel running method working and show empty values

* Refactor grouping & scoring code locations

* Add colour to strength raiting

* Add gap analysis tests

* Short drop down list

* Styling improvements and legends

* rm scikit version pin

* loosen requirement in sqlalchemy

* more deps changes

* fix deps

* Disable auto db pop (#372)

* Add gap analysis tests

* Short drop down list

* Disable CRE_Graph auto population on startup

* revert conflict mess with tests

* Migrate Neo4j population to seperate function

* Add CLI population command for DB

* linting fixes

* fix pillow

* Use python 3.11.4 for CI

* Parse cre_dep object from neo4j Node

* Standardise Gap Analysis API Object

* Fix bad parse field

* Use Standard display format and create GA type

* GA: Show all strong links by default (or min of 3)

* Better route display

* GA add support for tools

* Fix existing tests

* Add parsing tests

* make parse_node static

* Use Enum types

* Better route display

* linting

* Minimal GA styling

* Fix: GA share link spaces

* linting

* Added: Direct link grade and removed external link icon

* Fix: linked via standard was possible

* Added: Per link score penalty

* Fix: GA caching not taking queries (#412)

* Make Direct Dark Green (#411)

* Make Direct Dark Green

* Update GapAnalysis.tsx

Signed-off-by: John Harvey <[email protected]>

---------

Signed-off-by: John Harvey <[email protected]>

* GA Reduce related punishment (#416)

* Rename gap to map (#419)

* GA make strong status <=2

* Rename URL and UI from Gap to Map analysis

* GA fix bad links (#417)

Fix bad links in GA (use standard function)

* Speed up CICD (#418)

* Rename share link (#415)

* Neo4j ORM (#408)

* Make Direct Dark Green

* Update GapAnalysis.tsx

Signed-off-by: John Harvey <[email protected]>

* Trial implementation of Neo4J ORM

* Populate Neo4j DB using model

* More progress in conversion

* get ORM working

* Fix tests

* Remove old NeoDB driver connection

* Fix: id issue

* Fix: Tags strings splitting

* linting fixes

* Remove  NeomodelPath reference

* Revert rebase bug

---------

Signed-off-by: John Harvey <[email protected]>

* Remove BOLT from neo4j Setup requirement (#428)

* pin neomodel to a decent version

* relax six requirements

* Sort Base standard alphabetically (#427)

* Use allShortestPaths over shortestPath (#431)

* Added Cross-cutting concerns to GA blacklist (#430)

* Added Cross-cutting concerns to GA blacklist

* GA page linting

* Remove double brackets

* Fix GA loading disappearing

* rename variable

* Revert "Use allShortestPaths over shortestPath (#431)"

This reverts commit e33d13d.

* Precompute map analysis (#433)

* add time metrics

* progress

* Revert "Revert "Use allShortestPaths over shortestPath (#431)""

This reverts commit 167828c.

* map analysis through redis

* new reqs

* add worker actually

* abort if background job fails

* lint

* minor cleanup

* move hash to utils and make db method write to redis optionally

* make ga job results shared among clients

* lint

* Split ga response (#434)

* add time metrics

* map analysis through redis

* move hash to utils and make db method write to redis optionally

* lint

* Added: Front end support for weaker links calls

* Added: DB seperation for smaller inital response and cache extras

* Added: DB side of part split response logic

* Refactor and tests

---------

Co-authored-by: Spyros <[email protected]>

* skip returning ids for failed jobs

* add optional tls to redis, heroku requirement

* add optional tls to redis, heroku requirement

* add optional tls to redis, heroku requirement

* add optional tls to redis, heroku requirement

* add optional tls to redis, heroku requirement

* add optional tls to redis, heroku requirement

* use db to cache gap analysis results instead of redis (#435)

* use db to cache gap analysis results instead of redis

* lint

* typo

* fix 1 test, fix bugs, make cache key into its own function

* lint

* fix-mock

* migration

* Fix: bad ref

* fix tests

* lint

* minor changes

* lint

---------

Co-authored-by: john681611 <[email protected]>

* Get e2e tests reporting correctly (#395)

* try pip cacheing

* try yarn cache

* try background service method

* e2e test fixes

* tmp increase timeouts for e2e tests due to new neo4j experiments

* fix e2e tests

* make e2e workflow match makefile

* Revert actions simplification

* increase timing

* balance timings

---------

Signed-off-by: Spyros <[email protected]>
Co-authored-by: Spyros <[email protected]>
Co-authored-by: Spyros <[email protected]>

* minor fixes

* fix-lint

* fix caching

* fix e2e?

* add tabs

* lint

---------

Signed-off-by: John Harvey <[email protected]>
Signed-off-by: Spyros <[email protected]>
Co-authored-by: john681611 <[email protected]>
Co-authored-by: John Harvey <[email protected]>

application/frontend/src/pages/Search/components/BodyText.tsx

@@ -30,6 +30,9 @@ export const SearchBody = () => {
         into an AI model to create the world's first security-specialized chatbot. This ensures you get a more
         reliable answer, and also a reference to a reputable source.
       </b>
+      <b>
+        Use <a href="/map_analysis">Map Analysis</a> to find how any two Resources connect with eachother
+      </b>
       <h2>HOW?</h2>
       <p>
         OpenCRE links each section of a resource (like a standard or guideline) to a shared topic, known as a

application/frontend/src/scaffolding/Header/Header.tsx

@@ -13,6 +13,10 @@ const getLinks = (): { to: string; name: string }[] => [
     to: `/`,
     name: 'Open CRE',
   },
+  {
+    to: `/chatbot`,
+    name: 'OpenCRE Chat',
+  },
   {
     to: `/map_analysis`,
     name: 'Map analysis',

application/frontend/src/test/basic-e2etest.ts

@@ -162,7 +162,7 @@ describe('App.js', () => {
   it('can smartlink', async () => {
     const response = await page.goto('http://127.0.0.1:5000/smartlink/standard/CWE/1002');
     expect(response.url()).toBe('http://127.0.0.1:5000/node/standard/CWE/sectionid/1002');
-    
+
     const redirectResponse = await page.goto('http://127.0.0.1:5000/smartlink/standard/CWE/404');
     page.waitForNavigation('networkidle2');
     expect(redirectResponse.url()).toBe('https://cwe.mitre.org/data/definitions/404.html');