Releases: minvws/nl-kat-coordination
v1.5.0rc2
OpenKAT 1.5: multi organization support
The main feature of the 1.5 release of OpenKAT is the ability to scan multiple organizations from one OpenKAT instance and manage the data and users for each of them. It comes with an API to automatically deploy organizations and users from an external source.
This gives you the ability to manage OpenKAT for a large group of organizations while maintaining the separation of data and users for each of them. It also allows you to supply credentials to users within those organizations and to give them access to their own data, make reports etc.
In addition to this several other features have been added that support this use case, such as the ability to filter and manage and add objects in bulk and add scanning and inheritance of scan levels from a higher level. They were requested by some of our large scale partners and we will continue working on such features in the forthcoming releases.
Features:
- Most importantly, this is the first version of KAT that offers fully implemented multi-organization support. New organizations can be added in the Rocky admin interface (/admin), after which databases in xtdb and KAT-alogus will be automatically added. Clients can be added for each organization that cannot view data of other organizations
- Certificates can now be created without having a relation to a website
- IPAddresses now inherit a scan level from a netblock if that netblock is known
- NMAP can now scan ip ranges
- Rocky now offers bulk actions for the deletion of OOIs and giving clearance levels to OOIs
- Rocky now offers filtering options based on clearance levels and clearance level types
IMPORTANT
This feature includes a new version of XTDB. Therefore, the env setting XTDB_TYPE has to be set to xtdb-multinode. Also, all data in xtdb will be lost and cannot be moved to this version. If you use a locally installed version, you can change to the xtdb version in this repo: https://github.com/dekkers/xtdb-http-multinode.
Full Changelog
Coordination
What's Changed
- Dynamically checkout the latest release tag with
clone-stable
by @Darwinkel in #96 - Give CREATEDB rights to rocky db user by @dekkers in #98
- Update Sphinx docs by @Darwinkel in #97
- Set KAT_VERSION in .env-dist to v1.4.0 by @dekkers in #104
- Enable auto-reload of code for all services by @dekkers in #99
- Passing over xtdb work from Jeroen by @Rieven in #92
- documentation update: introduction into OpenKAT by @ring-ring-ring in #103
New Contributors
Full Changelog: v1.4.0rc1...v1.5.0rc2
Octopoes
What's Changed
- Release 1.4 by @dekkers in minvws/nl-kat-octopoes#31
- Feature/scan level filtering by @Lisser in minvws/nl-kat-octopoes#21
- Update
Certificate
model by @ammar92 in minvws/nl-kat-octopoes#25 - Feature/scan profile filtering by @Lisser in minvws/nl-kat-octopoes#22
- Add Python 3.11 to the unit test matrix by @Darwinkel in minvws/nl-kat-octopoes#29
- [OpenKAT v1.5] Support multinode XTDB by @Rieven in minvws/nl-kat-octopoes#20
- Add image meta data model v1 by @Donnype in minvws/nl-kat-octopoes#38
- implement by @Lisser in minvws/nl-kat-octopoes#59
- Updated dependabot config by @ammar92 in minvws/nl-kat-octopoes#60
New Contributors
- @Lisser made their first contribution in minvws/nl-kat-octopoes#21
- @ammar92 made their first contribution in minvws/nl-kat-octopoes#25
- @Darwinkel made their first contribution in minvws/nl-kat-octopoes#29
- @Rieven made their first contribution in minvws/nl-kat-octopoes#20
- @Donnype made their first contribution in minvws/nl-kat-octopoes#38
Full Changelog: minvws/nl-kat-octopoes@v1.4.0rc1...v1.5.0rc2
Bytes
What's Changed
- Release 1.4 by @dekkers in minvws/nl-kat-bytes#28
- Test multiple Python versions through Docker and Makefile build arg by @Darwinkel in minvws/nl-kat-bytes#25
- Bump setuptools from 62.3.2 to 65.6.3 by @dependabot in minvws/nl-kat-bytes#27
- Optional input_ooi by @Donnype in minvws/nl-kat-bytes#20
New Contributors
- @Darwinkel made their first contribution in minvws/nl-kat-bytes#25
- @Donnype made their first contribution in minvws/nl-kat-bytes#20
Full Changelog: minvws/nl-kat-bytes@v1.4.0rc1...v1.5.0rc2
Boefjes
What's Changed
- Censys normalizer updates by @ammar92 in minvws/nl-kat-boefjes#26
- [OpenKAT v1.5] make organisation field longer by @Rieven in minvws/nl-kat-boefjes#37
- Release 1.4 by @dekkers in minvws/nl-kat-boefjes#52
- Fix plugins related to
Certificate
by @ammar92 in minvws/nl-kat-boefjes#41 - [Nmap] Introduce Nmap ports boefje by @zcrt in minvws/nl-kat-boefjes#40
- [Nmap] Add settings for top-ports and protocol by @zcrt in minvws/nl-kat-boefjes#43
- [bug] RE: LeakIX handle HTTP 204 response by @zcrt in minvws/nl-kat-boefjes#56
- Change workflow file to test multiple Python versions by @Darwinkel in minvws/nl-kat-boefjes#44
- [Nmap] [NetBlock] ✨ introduce Nmap IP range scanning by @zcrt in minvws/nl-kat-boefjes#58
- Fix nmap schema and add test showing integers are not yet supported by @Donnype in minvws/nl-kat-boefjes#60
- Also fix nmap_ip_range integer by @Donnype in minvws/nl-kat-boefjes#68
- implement connection between ipaddresses and netblocks by @Lisser in minvws/nl-kat-boefjes#69
- [OpenKAT v1.5] Do not create _dev organisation in seed by @dekkers in minvws/nl-kat-boefjes#46
- Updated dependabot config by @ammar92 in minvws/nl-kat-boefjes#70
New Contributors
- @Rieven made their first contribution in minvws/nl-kat-boefjes#37
- @Darwinkel made their first contribution in minvws/nl-kat-boefjes#44
Full Changelog: minvws/nl-kat-boefjes@v1.4.0...v1.5.0rc2
Rocky
What's Changed
- Release 1.4.0 by @noamblitz in minvws/nl-kat-rocky#75
- add missing dmarc, dkim and spf findings by @noamblitz in minvws/nl-kat-rocky#35
- Fix ignored flake8 errors and change tests to pytest by @dekkers in minvws/nl-kat-rocky#87
- fix boefje settings forms for integer fields by @noamblitz in minvws/nl-kat-rocky#91
- untested but adding cached network reference support to csv importer. by @underdarknl in minvws/nl-kat-rocky#95
- fix flake8 error on katalogus forms by @underdarknl in minvws/nl-kat-rocky#99
- Release 1.4 by @dekkers in minvws/nl-kat-rocky#102
- Run pytest in CI by @dekkers in minvws/nl-kat-rocky#113
- Bump json5 from 2.2.1 to 2.2.3 by @dependabot in minvws/nl-kat-rocky#103
- feature(scan level filter): implement by @Lisser in minvws/nl-kat-rocky#82
- CSV import introspection by @ammar92 in minvws/nl-kat-rocky#98
- Change Organization model and add API by @dekkers in minvws/nl-kat-rocky#109
- Feature/scan profile filter by @Lisser in minvws/nl-kat-rocky#83
- [Feature] [#C86] Add certificate expiry check by @zcrt in minvws/nl-kat-rocky#88
- Updated file structure for css files within rocky. by @HeleenSG in minvws/nl-kat-rocky#122
- fix(ooi_report): take description from finding_type if None in Finding by @Lisser in minvws/nl-kat-rocky#118
- Update landing_page.html by @ring-ring-ring in minvws/nl-kat-rocky#124
- Added tag styles and colors. Updated to manon v1.1.5 by @HeleenSG in minvws/nl-kat-rocky#133
- Add support for XTDB multinode by @dekkers in minvws/nl-kat-rocky#123
- Add task list params by @jpbruinsslot in minvws/nl-kat-rocky#89
- Fix scan history template artifact by @jpbruinsslot in minvws/nl-kat-rocky#135
- [OpenKAT v1.5] Fix boefje detail checkboxes function at scan oois by @Rieven in minvws/nl-kat-rocky#111
- Feature/multi select by @Donnype in minvws/nl-kat-rocky#86
- Fix/scan levels by @Donnype in minvws/nl-kat-rocky#154
- [OpenKAT v1.5] Multi XTDB and organizations management through organization code as slug by @Rieven in minvws/nl-kat-rocky#115
- Ignore minor updates by @ammar92 in minvws/nl-kat-rocky#155
- refactor raising of scanprofile by @...
v1.4.0
Summary
OpenKAT will be even more open:
OpenKAT has been open since summer 2022, with public releases after an internal release process. OpenKAT 1.4.0 is the last release that went through that process as we've moved to a fully open development process. We're one of the first government backed OSS projects to take this step and everyone is invited to join in.
The OpenKAT community is growing with contributions from within the healthcare sector, general non-profits and government suppliers. The uptake has been good and with each release the software becomes more mature and easier to use. We welcome everyone and look forward to working together on OpenKAT.
Features:
This release has several features, mostly aimed at stability and usability of the system. Several items are noteworthy:
-
Mime-types have been introduced to decouple boefjes and whiskers, our scanners and normalizers. It opens up a world of possibilities for new modules for OpenKAT.
-
Queuing in Mula, the scheduler, has been improved.
-
Pre-commit workflow: the new workflow will help anyone who wishes to contribute to OpenKAT, bringing the developers on the team and everyone else on the same level
-
Debian packages are now auto-generated for each module based on the latest release. We're working on a repository to make 'apt install openkat' a reality.
IMPORTANT
Add DATABASE_MIGRATION=true your .env to run all migrations on building.
Full Changelog
Coordination
What's Changed
- Fix table formatting by @dekkers in #72
- Add make down command by @helmo in #68
- Fix docker volume name in cleanup task by @helmo in #69
- Trigger compile on
main
branch by @Darwinkel in #95 - Add missing keiko in Makefile and docker-compose by @dekkers
- Release 1.3.0 by @dekkers
- Add workflow and dockerfile for a debian build image by @errieman
- add git to build images to allow installation of octopoes via git by @errieman
- add ssh client to debian build image by @errieman
- Add more build dependencies to Debian Dockerfile by @dekkers
- Add missing KATALOGUS_API for Octopoes by @dekkers
- Also update Ubuntu dependencies by @dekkers
- Initial pre-commit, Sphinx, and GitHub Pages pipeline by @Darwinkel
New Contributors
Full Changelog: v1.3.0...v1.4.0
Rocky
What's Changed
- Download PDF Robot Flow by @reincode050
- Unify and update linter configs by @reincode050
- fix element not found by @Rieven
- Fix/styling checkboxes treeview by @Rieven
- add groups field to admin by @noamblitz
- support octopoes SCM versioning in docker dev setup by @errieman
- Improve User views in admin by @dekkers
- make rocky compatible with new scan profile model by @Lisser
- change name to id in normalizer task list by @noamblitz
- Create dependabot.yml by @sigio
- Update requirements.txt by @underdarknl
- Bump pylint from 2.13 to 2.15.5 by @dependabot
- build octopoes in main image by @errieman
- bump octopoes to 2.6.2 by @noamblitz
- accept jpg from katalogus by @noamblitz
- Bump django-formtools from 2.3 to 2.4 by @dependabot
- Bump sqlparse from 0.4.2 to 0.4.3 by @dependabot
- Bump uwsgi from 2.0.20 to 2.0.21 by @dependabot
- Required/mandatory settings for Boefjes by @Rieven
- perf(octopoes): improve pagination by @Lisser
- Fix for getting consumable oois at plugin detail by @Rieven
- Change date to time for checking OOI in future by @Rieven
- Translations before release by @Rieven
- Bump soupsieve from 2.2.1 to 2.3.2.post1 by @dependabot
- Bump phonenumbers from 8.12.30 to 8.12.57 by @dependabot
- Bump xhtml2pdf from 0.2.5 to 0.2.8 by @dependabot
- Fix toggle not working at ooi detail by @Rieven
- Bump toolz from 0.11.1 to 0.12.0 by @dependabot
- Bump psycopg2-binary from 2.8.6 to 2.9.5 by @dependabot
- Bump reportlab from 3.6.6 to 3.6.12 by @dependabot
- Bump colorama from 0.4.5 to 0.4.6 by @dependabot
- Bump requests from 2.26.0 to 2.28.1 by @dependabot
- Health link in footer by @Rieven
- Add total objects at all objects list by @Rieven
- Adds setting on organization member to assign and acknowledge OOI clearance levels on redteamers by @TwistMeister
- Bump urllib3 from 1.26.5 to 1.26.12 by @dependabot
- Bump python-dotenv from 0.19.0 to 0.21.0 by @dependabot
- Bump mypy from 0.971 to 0.991 by @dependabot
- Bump idna from 3.3 to 3.4 by @dependabot
- Bump qrcode from 6.1 to 7.3.1 by @dependabot
- Merge release back to develop by @dekkers
- Bump certifi from 2022.6.15 to 2022.9.24 by @dependabot
- Add scan levels to dev account in setup by @TwistMeister
- Improve onboarding "setup accounts" flow by @TwistMeister
- chore:upgraded to manon v1.1.0 by @HeleenSG
- Fixes task list download URL so it downloads zip with meta and RAW file by @TwistMeister
- Update stand alone checkboxes in forms in onboarding flow, to comply with new manon version by @TwistMeister
- Improve text "Got it, generate my report" to "start scanning" by @TwistMeister
- Bump vulture from 2.5 to 2.6 by @dependabot
- Bump djlint from 1.12.1 to 1.19.7 by @dependabot
- Revert "Bump djlint from 1.12.1 to 1.19.7" by @ammar92
- Bump pytz from 2021.1 to 2022.6 by @dependabot
- Set button states and heading styles by @TwistMeister
- Fix broken dashboard by @Rieven
- Move templates and locale directory inside rocky dir by @errieman
- Rocky deb package with venv by @errieman
- Changes to support new scheduler setup by @jpbruinsslot
- Feature/support download multiple raw files by @Donnype
- fix: Updated to manon v1.1.1 which includes the footer fix by @HeleenSG
- Fix normalizers task list by @noamblitz
- Bump certifi from 2022.9.24 to 2022.12.7 by @dependabot
- Fix: re-use uuid by @noamblitz
- bump octopoes to 2.8.0 by @noamblitz
- Upgraded manon version by @HeleenSG
- Fixed: qr-code alignment and form message alignment by @HeleenSG
- Upgraded to manon v1.1.3 by @HeleenSG
- Fix setuptools version scheme by @dekkers in minvws/nl-kat-rocky#90
- Fix use of git tag in Debian package workflow by @dekkers in minvws/nl-kat-rocky#92
New Contributors
- @jpbruinsslot made their first contribution
Full Changelog: minvws/nl-kat-rocky@v1.3.0...v1.4.0
Boefjes
What's Changed
- fix env file by @underdarknl in minvws/nl-kat-boefjes#16
- LeakIX handle HTTP 204 response by @zcrt in minvws/nl-kat-boefjes#19
- Censys by @ammar92 in minvws/nl-kat-boefjes#20
- [log4shell] [#14] make REPLY_FQDN variable by @zcrt in minvws/nl-kat-boefjes#25
- [bug] [shodan] log no info instead of failing boefje by @zcrt in minvws/nl-kat-boefjes#22
- RE: [log4shell] [#14] [#25] make REPLY_FQDN variable by @zcrt in minvws/nl-kat-boefjes#28
- fixes, including formatting by @Lisser in minvws/nl-kat-boefjes#31
- Implement pre-commit hooks and some improvements by @ammar92
- Bump docker from 5.0.2 to 6.0.1 by @dependabot
- Bump jsonschema from 4.16.0 to 4.17.0 by @dependabot
- Bump sqlalchemy from 1.4.42 to 1.4.43 by @dependabot
- Bump click from 8.0.3 to 8.1.3 by @dependabot
- Bump fastapi from 0.85.1 to 0.86.0 by @dependabot
- Fix fierce error handling by @Lisser
- Bump dnspython from 2.1.0 to 2.2.1 by @dependabot
- Bump sqlalchemy from 1.4.43 to 1.4.44 by @dependabot
- Bump fastapi from 0.86.0 to 0.87.0 by @dependabot
- Add covers for: webpage analysis, greenhosting, leakIX and many new ports open by @TwistMeister
- Fix requirements by @dekkers
- Merge release back to develop by @dekkers
- Updates pre-commit pipeline by @ammar92
- Debian package with venv by @errieman
- Changes to support new scheduler setup by @jpbruinsslot
- Fix alembic invocation by @dekkers
- Feature/multiple raw returns by @Donnype
- Feature/multiple raw returns by @noamblitz
- Refactor: prepare-for-containerization support by @Donnype
- Fix setuptools version scheme by @dekkers in minvws/nl-kat-boefjes#42
- Debian package fixes by @dekkers in minvws/nl-kat-boefjes#45
New Contributors
- @TwistMeister made their first contribution
- @jpbruinsslot made their first contribution
- @underdarknl made their first contribution in minvws/nl-kat-boefjes#16
- @zcrt made their first contribution in minvws/nl-kat-boefjes#19
- @ammar92 made their first contribution in minvws/nl-kat-boefjes#20
- @Lisser made their first contribution in minvws/nl-kat-boefjes#31
Full Changelog: minvws/nl-kat-boefjes@v1.3.0...v1.4.0
Octopoes
What's Changed
- Add pytest, show test coverage and add more tests by @dekkers
- Replace the CI pipeline with
pre-commit
by @Darwinkel - Feature/robot improvements by @Lisser
- chore(robot): explicit host mapping in docker compose by @Lisser
- Bump setuptools from 65.5.0 to 65.5.1 by @dependabot
- Bump fastapi from 0.85.1 to 0.86.0 by @dependabot
- chore(dependencies): fix dependencies by @Lisser
- Feature/pre commit compliance by @Lisser
- refactor(*): comply to vulture by @Lisser
- Fix pushing the scan profile increments events to the scheduler by @Lisser
- Updates pre-commit pipeline by @ammar92
- Merge release back to develop by @dekkers
- Add mypy pre-commit stubs by @Darwinkel
- Run unit ...
v1.4.0rc2
v1.4.0rc1
Summary
OpenKAT will be even more open:
OpenKAT has been open since this years' summer, with public releases after an internal release process. OpenKAT 1.4.0 is the last release that went through that process as we've moved to a fully open development process. We're one of the first government backed OSS projects to take this step and everyone is invited to join in.
The OpenKAT community is growing with contributions from within the healthcare sector, general non-profits and government suppliers. The uptake has been good and with each release the software becomes more mature and easier to use. We welcome everyone and look forward to working together on OpenKAT.
Features:
This release has several features, mostly aimed at stability and usability of the system. Several items are noteworthy:
-
Mime-types have been introduced to decouple boefjes and whiskers, our scanners and normalizers. It opens up a world of possibilities for new modules for OpenKAT.
-
Queuing in Mula, the scheduler, has been improved.
-
Pre-commit workflow: the new workflow will help anyone who wishes to contribute to OpenKAT, bringing the developers on the team and everyone else on the same level
-
Debian packages are now auto-generated for each module based on the latest release. We're working on a repository to make 'apt install openkat' a reality.
IMPORTANT
Add DATABASE_MIGRATION=true your .env to run all migrations on building.
Full Changelog
Coordination
What's Changed
- Add missing keiko in Makefile and docker-compose by @dekkers
- Release 1.3.0 by @dekkers
- Add workflow and dockerfile for a debian build image by @errieman
- add git to build images to allow installation of octopoes via git by @errieman
- add ssh client to debian build image by @errieman
- Add more build dependencies to Debian Dockerfile by @dekkers
- Add missing KATALOGUS_API for Octopoes by @dekkers
- Also update Ubuntu dependencies by @dekkers
- Initial pre-commit, Sphinx, and GitHub Pages pipeline by @Darwinkel
Full Changelog: v1.3.0...v1.4.0rc1
Rocky
What's Changed
- Download PDF Robot Flow by @reincode050
- Unify and update linter configs by @reincode050
- fix element not found by @Rieven
- Fix/styling checkboxes treeview by @Rieven
- add groups field to admin by @noamblitz
- support octopoes SCM versioning in docker dev setup by @errieman
- Improve User views in admin by @dekkers
- make rocky compatible with new scan profile model by @Lisser
- change name to id in normalizer task list by @noamblitz
- Create dependabot.yml by @sigio
- Update requirements.txt by @underdarknl
- Bump pylint from 2.13 to 2.15.5 by @dependabot
- build octopoes in main image by @errieman
- bump octopoes to 2.6.2 by @noamblitz
- accept jpg from katalogus by @noamblitz
- Bump django-formtools from 2.3 to 2.4 by @dependabot
- Bump sqlparse from 0.4.2 to 0.4.3 by @dependabot
- Bump uwsgi from 2.0.20 to 2.0.21 by @dependabot
- Required/mandatory settings for Boefjes by @Rieven
- perf(octopoes): improve pagination by @Lisser
- Fix for getting consumable oois at plugin detail by @Rieven
- Change date to time for checking OOI in future by @Rieven
- Translations before release by @Rieven
- Bump soupsieve from 2.2.1 to 2.3.2.post1 by @dependabot
- Bump phonenumbers from 8.12.30 to 8.12.57 by @dependabot
- Bump xhtml2pdf from 0.2.5 to 0.2.8 by @dependabot
- Fix toggle not working at ooi detail by @Rieven
- Bump toolz from 0.11.1 to 0.12.0 by @dependabot
- Bump psycopg2-binary from 2.8.6 to 2.9.5 by @dependabot
- Bump reportlab from 3.6.6 to 3.6.12 by @dependabot
- Bump colorama from 0.4.5 to 0.4.6 by @dependabot
- Bump requests from 2.26.0 to 2.28.1 by @dependabot
- Health link in footer by @Rieven
- Add total objects at all objects list by @Rieven
- Adds setting on organization member to assign and acknowledge OOI clearance levels on redteamers by @TwistMeister
- Bump urllib3 from 1.26.5 to 1.26.12 by @dependabot
- Bump python-dotenv from 0.19.0 to 0.21.0 by @dependabot
- Bump mypy from 0.971 to 0.991 by @dependabot
- Bump idna from 3.3 to 3.4 by @dependabot
- Bump qrcode from 6.1 to 7.3.1 by @dependabot
- Merge release back to develop by @dekkers
- Bump certifi from 2022.6.15 to 2022.9.24 by @dependabot
- Add scan levels to dev account in setup by @TwistMeister
- Improve onboarding "setup accounts" flow by @TwistMeister
- chore:upgraded to manon v1.1.0 by @HeleenSG
- Fixes task list download URL so it downloads zip with meta and RAW file by @TwistMeister
- Update stand alone checkboxes in forms in onboarding flow, to comply with new manon version by @TwistMeister
- Improve text "Got it, generate my report" to "start scanning" by @TwistMeister
- Bump vulture from 2.5 to 2.6 by @dependabot
- Bump djlint from 1.12.1 to 1.19.7 by @dependabot
- Revert "Bump djlint from 1.12.1 to 1.19.7" by @ammar92
- Bump pytz from 2021.1 to 2022.6 by @dependabot
- Set button states and heading styles by @TwistMeister
- Fix broken dashboard by @Rieven
- Move templates and locale directory inside rocky dir by @errieman
- Rocky deb package with venv by @errieman
- Changes to support new scheduler setup by @jpbruinsslot
- Feature/support download multiple raw files by @Donnype
- fix: Updated to manon v1.1.1 which includes the footer fix by @HeleenSG
- Fix normalizers task list by @noamblitz
- Bump certifi from 2022.9.24 to 2022.12.7 by @dependabot
- Fix: re-use uuid by @noamblitz
- bump octopoes to 2.8.0 by @noamblitz
- Upgraded manon version by @HeleenSG
- Fixed: qr-code alignment and form message alignment by @HeleenSG
- Upgraded to manon v1.1.3 by @HeleenSG
New Contributors
- @jpbruinsslot made their first contribution
Full Changelog: minvws/nl-kat-rocky@v1.3.0...v1.4.0rc1
Boefjes
What's Changed
- fix env file by @underdarknl in minvws/nl-kat-boefjes#16
- LeakIX handle HTTP 204 response by @zcrt in minvws/nl-kat-boefjes#19
- Censys by @ammar92 in minvws/nl-kat-boefjes#20
- [log4shell] [#14] make REPLY_FQDN variable by @zcrt in minvws/nl-kat-boefjes#25
- [bug] [shodan] log no info instead of failing boefje by @zcrt in minvws/nl-kat-boefjes#22
- RE: [log4shell] [#14] [#25] make REPLY_FQDN variable by @zcrt in minvws/nl-kat-boefjes#28
- fixes, including formatting by @Lisser in minvws/nl-kat-boefjes#31
- Implement pre-commit hooks and some improvements by @ammar92
- Bump docker from 5.0.2 to 6.0.1 by @dependabot
- Bump jsonschema from 4.16.0 to 4.17.0 by @dependabot
- Bump sqlalchemy from 1.4.42 to 1.4.43 by @dependabot
- Bump click from 8.0.3 to 8.1.3 by @dependabot
- Bump fastapi from 0.85.1 to 0.86.0 by @dependabot
- Fix fierce error handling by @Lisser
- Bump dnspython from 2.1.0 to 2.2.1 by @dependabot
- Bump sqlalchemy from 1.4.43 to 1.4.44 by @dependabot
- Bump fastapi from 0.86.0 to 0.87.0 by @dependabot
- Add covers for: webpage analysis, greenhosting, leakIX and many new ports open by @TwistMeister
- Fix requirements by @dekkers
- Merge release back to develop by @dekkers
- Updates pre-commit pipeline by @ammar92
- Debian package with venv by @errieman
- Changes to support new scheduler setup by @jpbruinsslot
- Fix alembic invocation by @dekkers
- Feature/multiple raw returns by @Donnype
- Feature/multiple raw returns by @noamblitz
- Refactor: prepare-for-containerization support by @Donnype
New Contributors
- @TwistMeister made their first contribution
- @jpbruinsslot made their first contribution
- @underdarknl made their first contribution in minvws/nl-kat-boefjes#16
- @zcrt made their first contribution in minvws/nl-kat-boefjes#19
- @ammar92 made their first contribution in minvws/nl-kat-boefjes#20
- @Lisser made their first contribution in minvws/nl-kat-boefjes#31
Full Changelog: minvws/nl-kat-boefjes@v1.3.0...v1.4.0rc1
Octopoes
What's Changed
- Add pytest, show test coverage and add more tests by @dekkers
- Replace the CI pipeline with
pre-commit
by @Darwinkel - Feature/robot improvements by @Lisser
- chore(robot): explicit host mapping in docker compose by @Lisser
- Bump setuptools from 65.5.0 to 65.5.1 by @dependabot
- Bump fastapi from 0.85.1 to 0.86.0 by @dependabot
- chore(dependencies): fix dependencies by @Lisser
- Feature/pre commit compliance by @Lisser
- refactor(*): comply to vulture by @Lisser
- Fix pushing the scan profile increments events to the scheduler by @Lisser
- Updates pre-commit pipeline by @ammar92
- Merge release back to develop by @dekkers
- Add mypy pre-commit stubs by @Darwinkel
- Run unit tests with supported Python versions by @Darwinkel
- feat(scan_profiles): send all mutation events to scheduler by @Lisser
- feat(api): implement object filtering based on scan level by @Lisser
- fix(types): implement some missing types by @Lisser
- chore(deps): update by @Lisser
- Initial Sphinx/autodoc support by @Darwinkel
- Potential fix for Pages deploy by @Darwinkel
- Fix trailing spaces in diagrams by @Darwinkel
- v2 architectural design document by @Lisser
- v2 -> v3 by @Lisser
- Ship venv with debian package by @errieman
- Remove redundant artifact upload by @Darwinkel
- default 0 for filtering by scan level by @Lisser
- Bump certifi from 2022.9.24 to 2022.12.7 by @dependabot
New Contributors
- @Darwinkel made their first contribution
- @dependabot made their first contribution
Full Changelog: https://github.com/minvws/nl-kat-octopoes/co...
v1.3.0
Summary
The 1.3.0 release of OpenKAT introduces new features and small fixes. The most interesting new elements are the way boefjes and normalizers are structured and the way settings are handled. Boefjes and normalizers are decoupled, to allow more matches between boefjes, normalizers and objects. Settings for boefjes have become available through the front end of OpenKAT. Also, OpenKAT has become easier to package and install.
IMPORTANT
Celery beat had been added to the Octopoes worker and the command used to start the Octopoes worker has been changed because of this. There needs to be one Octopoes worker that runs Celery beat to schedule tasks and there can also be only one worker with celery beat. For containers we added entrypoints that can handle any future changes, see https://github.com/minvws/nl-kat-coordination/blob/main/docs/containers.md#container-commands and https://github.com/minvws/nl-kat-coordination/blob/main/docker-compose.release-example.yml
Settings for boefjes
Building on the changes that were made in the previous release, from now on boefjes can have mandatory settings. For example, before enabling Shodan, you must provide the API key in the katalogus settings. This removes the need to place all boefje settings in a config file.
Plugin directory restructure
Boefjes and normalizers are now fully decoupled. This means that a plugin directory can now have as many subdirectories with boefjes and normalizers. For a boefjes or normalizer to be found, it has to have an __init__.py
, a boefje.json
or normalizer.json
and a file with a run method. This sets a big step towards "package-ability" of plugins. On the downside, your previously created boefjes and normalizers need a minor update. The manual on the wiki has been adapted to reflect these changes and will help you make the transition.
Pagination of Octopoes
Before, when loading the objects list, all OOIs were returned. This created performance issues, and therefore we added API side pagination to octopoes.
Full Changelog
Coordination
What's Changed
- apply changes to octopoes worker by @Lisser
Rocky
What's Changed
- Download PDF Robot Flow by @reincode050
- Unify and update linter configs by @reincode050
- fix element not found by @Rieven
- Fix/styling checkboxes treeview by @Rieven
- add groups field to admin by @noamblitz
- support octopoes SCM versioning in docker dev setup by @errieman
- Improve User views in admin by @dekkers
- make rocky compatible with new scan profile model by @Lisser
- change name to id in normalizer task list by @noamblitz
- Create dependabot.yml by @sigio
- Update requirements.txt by @underdarknl
- Bump pylint from 2.13 to 2.15.5 by @dependabot
- build octopoes in main image by @errieman
- bump octopoes to 2.6.2 by @noamblitz
- accept jpg from katalogus by @noamblitz
- Bump django-formtools from 2.3 to 2.4 by @dependabot
- Bump sqlparse from 0.4.2 to 0.4.3 by @dependabot
- Bump uwsgi from 2.0.20 to 2.0.21 by @dependabot
- Required/mandatory settings for Boefjes by @Rieven
- perf(octopoes): improve pagination by @Lisser
- Fix for getting consumable oois at plugin detail by @Rieven
- Change date to time for checking OOI in future by @Rieven
- Translations before release by @Rieven
- Bump soupsieve from 2.2.1 to 2.3.2.post1 by @dependabot
- Bump phonenumbers from 8.12.30 to 8.12.57 by @dependabot
- Bump xhtml2pdf from 0.2.5 to 0.2.8 by @dependabot
- update yarn packages by @Rieven
Boefjes
What's Changed
- update instructions for running boefjes by @errieman
- Feature/json boefje definitions by @Donnype
- support octopoes SCM versioning in docker dev setup by @errieman
- Expose environment keys (mandatory fields) by @Donnype
- Feature/recursively find plugins by @Donnype
- Feature/module to path by @Donnype
- Discovery fix by @Donnype
- add correct port and address to cert boefje by @noamblitz
- Remove redundant API key notices from description by @noamblitz
- Create dependabot.yml by @sigio
- build octopoes in main image by @errieman
- fix(binaryedge): rename ooi_type to object_type by @Lisser
- bump octopoes to 2.6.2 by @noamblitz
- optimize katalogus images by @noamblitz
- Env vars globally by @sigio
- Bump sqlalchemy from 1.4.32 to 1.4.42 by @dependabot
- Bump fastapi from 0.75.0 to 0.85.1 by @dependabot
- Bump uvicorn from 0.17.5 to 0.19.0 by @dependabot
- Bump pytest from 6.2.5 to 7.2.0 by @dependabot
- revert id to name change by @noamblitz
- get api keys from katalogus api plugin settings by @noamblitz
- Bump dnspython from 2.1.0 to 2.2.1 by @dependabot
- Bump black from 21.9b0 to 22.10.0 by @dependabot
- Bump psycopg2-binary from 2.9.3 to 2.9.5 by @dependabot
- Bump alembic from 1.7.7 to 1.8.1 by @dependabot
- Bump requests from 2.27.1 to 2.28.1 by @dependabot
- Fix requirements for requests and dispython by @ammar92
- Automated update to workflows: use checkout@v3 by @sigio
New Contributors
- @dependabot made their first contribution
Octopoes
What's Changed
- fix(retirejs): avoid indexerror by @Lisser
- Minor improvements and optimalisation for scanprofile recalculate by @dekkers
- Release 2.6.0 by @Lisser
- refactor(list_ooi): add count to result by @Lisser
- Create pull_request_template.md by @Lisser
- Automated update to workflows: use checkout@v3 by @sigio
- fix packaging requirement by @noamblitz
Mula
What's changed
- Merge pull request #130 from minvws/feat/monitor-org-timeout
- Access queues directly from schedulers dict
Bytes
What's Changed
- Bump mako from 1.2.0 to 1.2.2 by @dependabot
- Hotfix/rename name to by @Donnype
- Create dependabot.yml by @sigio
- Automated update to workflows: use checkout@v3 by @sigio
New Contributors
- @dependabot made their first contribution
v1.2.4
v1.2.3
What's Changed
- Update docs to build with pre built containers by @noamblitz in #46
Full Changelog: v1.2.0...v1.2.3
v1.2.0
Introduction
This new release of OpenKAT includes many small and large updates: OpenKAT has become easier to install, use and maintain. One of the main novelties is Keiko, the reporting module. We also introduced container images based on a GitHUB action, introduced a settings scheme for Boefjes and removed Flower from the system. Read the full changelog to see all the work that has been done. We look forward to comments on this release, here on GitHUB or by email @ [email protected].
IMPORTANT
Before using this update, delete your .env file in the main directory and prune docker, or make sure that your .env contains all new variables (KEIKO_API is new and required to make reports, also, USE_SCHEDULER=1 should be added).
Summary
Keiko
Keiko is a new module added to KAT responsible for creating informative reports in LateX. In this version, you can check out some of Keiko's capabilities by going to a findings report and clicking on the generate PDF report button. By using Keiko, we will be able to create different and more versatile reports in the future which will become one of KATs most important features.
Containers
In this version, people that want to use KAT without actively developing it can use pre-built containers which are built using GitHub workflows. This saves a lot of time and risks of compiling errors.
Celery and Flower
As Flower is not actively maintained we decided to remove KATs dependency on it. Previously, jobs were placed in a celery queue by the scheduler and the Boefjes would use that queue to know what to do. Since this release, the Boefje runner pops directly from the scheduler's queue. Not only is this method more secure (flower contained some vulnerabilities), but it also opens up the possibility for Boefjes to pop only jobs that they are capable of running. Think of two Boefje runners of which only one has access to IPv6.
Boefje settings
In this version, we laid the foundation of settings for Boefjes. For example, API keys, endpoints, but also "how many ports should NMAP scan?". Settings can be set per boefje, per organisation. In the next version, those settings will be fetched by the runner and injected into the Boefje job. Also, minimal settings for a Boefje to run will be added.
Model changes
For ease-of-use purposes we added a URL discovery bit, enabling users to also choose Hostnames as "starting points" instead of only URLs. Also, Subject Alternative Names were added to the Certificate object removing false positives that we were made aware of by the community.
Full Changelog
Coordination
What's Changed
- Update boefje entrypoints by @Donnype
- Add
make checkout branch=x
command by @reincode050 - feat(keiko): implement keiko in env, docker and makefile by @Lisser
- feature(keiko): update entrypoint by @Lisser
- Build production suitable container images in CI by @dekkers
- Run build-rocky-frontend outside docker by @dekkers
- refactor(keiko): change keiko api invocation by @Lisser
- Remove celery by @Rieven
- Use local octopoes when developing by @dekkers
New Contributors
- @dekkers made their first contribution
Rocky
What's Changed
- Robot framework implementation by @Reinaard
- Add note for SMTP by @Rieven
- Bug fix by @Rieven
- Implement PR and issue templates by @reincode050 in
- Implement baseline Python linters on Rocky by @reincode050
- feat(): Remove inline JS and add separate script to handle this by @TwistMeister
- Filter boefjes on object detail when scan level exceeds objects clearance level by @TwistMeister
- Change django password requirements to allow rdo-default by @sigio
- More functional Robot tests by @reincode050
- Remove 90% confidence lines by @reincode050
- Initial GA translation check by @reincode050
- Celery to scheduler by @Rieven
- Send Content-Security-Policy header using django-csp by @dekkers
- Add autocomplete to token field of form by @Rieven
- General settings for KAT-alogus by @Rieven
- Beautified Health Checks by @Rieven
- feat(keiko): add option to generate pdf report by @Lisser
- Refactoring CSV upload for Hostname, IPAddressV4, IPAddressV6 by @Rieven
- Password fixes for Robot tests by @reincode050
- Build production suitable container images in CI by @dekkers
- Fix collectstatic and by @dekkers
- Fix upgrading deb package by @errieman
- Enable uwsgi thunder lock to workaround bug by @dekkers
- Make password settings configurable using env variables by @dekkers
- Use local boefjes and octopoes when developing by @dekkers
- More features added to Task List by @Rieven
- Feature/settings per boefje by @Rieven
- remove inline JS by @Rieven
- Feature/translations by @Rieven
New Contributors
- @Reinaard made their first contribution
Mula
What's Changed
- Fix pylint suggestions by @jpbruinsslot
- Add commit to update by @jpbruinsslot
- Start database session for every method by @jpbruinsslot
- Remove obsolete sesssion by @jpbruinsslot
- Build production suitable container images in CI by @dekkers
- Support using Postgres as database by @dekkers
- Fix upgrading deb package by @errieman
- Remove references to dispatcher by @jpbruinsslot
New Contributors
- @dekkers made their first contribution
Bytes
What's Changed
- Build production suitable container images in CI by @dekkers
- Add log statements in log manager by @Donnype
- templated repos url in changelog by @errieman
- Requirements bump by @Donnype
- Fix debian package upgrade by @errieman
- Reconnect and retry basic_publish on pika.exceptions.ConnectionClosed by @Donnype
- Longer plugin_id, normalizer_name and boefje_id character fields by @Donnype
New Contributors
- @dekkers made their first contribution
Boefjes
What's Changed
- Build production suitable container images in CI by @dekkers
- Remove removed boefje requirements.txt from Dockerfile by @dekkers
- Update requirements.txt by @underdarknl
- Deb upgrade fix by @errieman
- Feature/create org on request by @noamblitz
- Feature/pop from scheduler pq worker update by @Donnype
- Add plugin_id parameter and filter on it for the all() method. by @Donnype
- Cherry picked local octopoes by @Donnype
- Add certificate subject alternative names to certificate boefje by @noamblitz
- Longer plugin_id, normalizer_name and boefje_id character fields by @Donnype
New Contributors
- @dekkers made their first contribution
Octopoes
What's Changed
- Release 2.4.0 by @noamblitz
- Fix debian package upgrade by @errieman
- fix(netblock model): fix human-readable by @Lisser
- fix path to debian build script by @errieman
- Main by @noamblitz
- fix netblock human_readable by @Lisser
- Fix openapi schema endpoint by @dekkers
- Build production suitable container images in CI by @dekkers
- Add makefile and debian package target by @errieman
- remove nginx dep from deb by @errieman
- URL discovery bit by @noamblitz
- Add certificatealternativenames and make bits work by @noamblitz
New Contributors
- @dekkers made their first contribution
v1.1.0
Introduction
Welcome to the first release of OpenKAT after we let her play out in the real world under the EU PL 1.2 license. The response has been fantastic, many thanks for this.
The goal of this release is to give OpenKAT nice and round edges in many areas that were a bit rough, reduce dependencies, make Octopoes persistent, introduce the debian packages etc: basically to improve OpenKAT in all corners. Also, OpenKAT now uses Manon open for the front end design.
It also includes some fixes that should warrant a more smooth installation. Please share your experience @ [email protected]
Before you upgrade your current installation please follow the advice below:
IMPORTANT
Make sure that your DB has no users with the same email address before migrating.
Delete your .env file in the main directory before making, or make sure that your .env contains all new variables. SCHEDULER_DB_DSN and SCHEDULER_API are new, rockydb credentials are renamed, see .env-dist.
To use email password recovery, make sure to set the SMTP env variables.
Summary
Debian installer and security improvements
Debian packages arrived! Some notes:
Rocky works out of the box due to a self-signed certificate. This certificate does have to be replaced by the user. Secondly, the default configuration of Nginx is strict. For example it sets the client body limits to protect against DoS attacks and only allows strong TLS ciphers and protocols. The installer also generates a random password for the RabbitMQ user.
Login has been changed internally. Instead of using the Django user model with a username field, we now use the email field for logging in. Also first and last name are changed to full name. The database will automatically enforce these changes while migrating. Therefore, the migration will fail when the database contains two users with the same email address.
When using docker containers, the Rocky user now owns the application directory so that it is able to write yarn error logs.
The UID and GID of the host in the containers is used to resolve permission errors caused by mounting the application code in the containers, which changes permission levels to the host UID and GID.
For end-users
Octopoes is now persistent, which means that data will not be lost after an update or reboot. To get a new version of KAT without cleaning all data, "make update" is now available which skips the cleaning step. This will pull new versions, do all necessary database migrations and spin all containers back up.
As usual, Rocky got a lot of small UI improvements, but most noticeably, Rocky now forces users to set the correct clearance level before running a boefje. Before, on an object without a clearance level when running a boefje, a clearance level would be set. Now, this is not possible anymore. Rocky also migrated to use open Manon which was previously open-sourced.
Flower and celery are now not dependencies of Rocky anymore. This means that when manually running a boefje, a call will be made to a new scheduler API which will schedule that job with a high priority. All jobs (boefjes and normalizers) are now shown in Rocky with that same scheduler API, not only those that are run manually.
Full Changelog
Coordination
What's Changed
- Update README.adoc by @ring-ring-ring
- Pinned RabbitMQ version by @ammar92
- .dockerignore by @Donnype
- use persistent xtdb by @noamblitz
- Correct LinkedIn url in link to openKAT by @reincode050
- Provide current user id to docker builds, defaulting to 1000 by @Donnype
- make update by @noamblitz
- Add SCHEDULER_DB_DSN by @jpbruinsslot
- fix makefile for macos by @noamblitz
- add scheduler api endpoint in env-dist by @noamblitz
New Contributors
- @reincode050 made their first contribution
Rocky
What's Changed
- Github workflow for creating .deb installer by @errieman
- Add robots.txt by @Rieven
- Upgrade requirements to use django 3.2.14 by @TwistMeister
- fix description kat-581 by @noamblitz
- Hide CVSS link in report when 0 findings by @TwistMeister
- Change crisis room total findings list to table by @TwistMeister
- Add temporary classes to fix recommendation labels by @TwistMeister
- Fix/graph ooi by @Rieven
- Bump lxml from 4.6.5 to 4.9.1 by @dependabot
- Delete ro-logo.svg by @TwistMeister
- Update manon-dev.css, by removing the reference to deleted icons by @TwistMeister
- Remove hyperlink on bit name on object detail by @TwistMeister
- Add formatter by @ppvg
- Temporarily hide add indemnification button by @TwistMeister
- Clearance level form initial value for declared levels by @ammar92
- Hide "scan object" form from boefje detail when no scannable objects by @TwistMeister
- Fix exported migrations to match migrate by @dekkers
- Feature/user model and auth by @Rieven
- Bump terser from 5.14.1 to 5.14.2 by @dependabot
- Fix shebang in run_rock.sh CI script by @dekkers
- Move mixins by @Rieven
- Set permissions for organization view and members by @Rieven
- Chown app dir to rocky user by @Donnype
- Add .editorconfig by @ppvg
- Fix for make build by @Rieven
- Configure rabbitmq user and pass by @errieman
- Fix/django bump by @underdarknl
- Increase items per page for oois and findings lists by @TwistMeister
- use repository name in deb changelog by @errieman
- Login and recovery by @Rieven
- Feature/objects filter on boefje detail jesse by @Lisser
- Provide current user id to docker builds and bump node version by @Donnype
- generate self-signed cert on install by @errieman
- Use manon from npm by @ppvg
- NL + PAP translations before release by @Rieven
- Feature/scheduler client by @Lisser
New Contributors
- @errieman made their first contribution
- @ppvg made their first contribution
- @dekkers made their first contribution
Mula
What's Changed
- Boefje error handling by @jpbruinsslot
- Update docs for job status endpoints by @jpbruinsslot
- Feature/error handling by @jpbruinsslot
Bytes
What's Changed
- Debian installer for bytes by @errieman
- Small docs update by @Donnype
- configure rabbitmq user on deb install by @errieman
- Add event for received normalizer_meta and fix Makefile issue by @Donnype
- Provide current user id to docker builds, defaulting to 1000 by @Donnype
New Contributors
- @errieman made their first contribution
Boefjes
What's Changed
- remove SPF boefje tests by @errieman
- Fix/better caching of boefjes requirements by @Donnype
- Debian installer by @errieman
- add manual trigger to deb build by @errieman
- fix version number on manual build by @errieman
- Catch boefje errors by @ammar92
Octopoes
What's Changed
- Debian installer for octopoes by @errieman
- temp fix for hostname objects from server headers by @noamblitz
- Configure rabbitmq user and pass on install by @errieman
- templated repos url in changelog by @errieman
- Provide current user id to docker builds, defaulting to 1000 by @Donnype
New Contributors
- @errieman made their first contribution