Refactor KATalogus client in Rocky #3717

Donnype · 2024-10-23T09:35:43Z

Changes

Handles client errors generically in a base client that can be organization-agnostic for some methods.
Introduces a new intermediate "adapter" called KATalogus that forces you to set an organization.

Issue link

Demo

QA notes

Please verify that we can still properly create organizations, add plugins, settings and enable/disable them from Rocky!

Code Checklist

All the commits in this PR are properly PGP-signed and verified.
This PR only contains functionality relevant to the issue.
I have written unit tests for the changes or fixes I made.
I have checked the documentation and made changes where necessary.
I have performed a self-review of my code and refactored it to the best of my abilities.
For any non-trivial functionality, I have added integration and/or end-to-end tests.
I have included comments in the code to elaborate on what is not self-evident from the code itself, including references to issues and discussions online, or implicit behavior of an interface.

Checklist for code reviewers:

Copy-paste the checklist from the docs/source/templates folder into your comment.

Checklist for QA:

Copy-paste the checklist from the docs/source/templates folder into your comment.

…on the KATalogusClient

Call methods with organization code

Some refactoring around the client usage

rocky/katalogus/client.py

rocky/katalogus/utils.py

sonarcloud · 2024-10-24T12:39:25Z

Quality Gate failed

Failed conditions
78.4% Coverage on New Code (required ≥ 80%)

See analysis details on SonarCloud

WIP fix unit tests

stephanie0x00 · 2024-10-28T12:18:20Z

Checklist for QA:

I have checked out this branch, and successfully ran a fresh make reset.
I confirmed that there are no unintended functional regressions in this branch:
- I have managed to pass the onboarding flow
- Objects and Findings are created properly
- Tasks are created and completed properly
I confirmed that the PR's advertised feature or hotfix works as intended.
I checked the logs for errors and/or warnings and made issues where necessary

What works:

Adding multiple organisations, creating a plugin variant works.

What doesn't work:

See below.

Bug or feature?:

Disabling a plugin gives the following error in the logs. The UI says the plugin is disabled.

rocky-1  | [28/Oct/2024 12:04:52] "GET /static/dist/tabler-icons.c16e726d.woff2 HTTP/1.1" 304 0
rocky-1  | [28/Oct/2024 12:04:52] "GET /static/img/favicon.svg HTTP/1.1" 200 819
rocky-1  | Traceback (most recent call last):
rocky-1  |   File "/app/rocky/katalogus/client.py", line 152, in verify_response
rocky-1  |     response.raise_for_status()
rocky-1  |   File "/usr/local/lib/python3.11/site-packages/httpx/_models.py", line 763, in raise_for_status
rocky-1  |     raise HTTPStatusError(message, request=request, response=self)
rocky-1  | httpx.HTTPStatusError: Client error '404 Not Found' for url 'http://katalogus:8000/v1/organisations/aa/plugins?ids=%7B%27dns-records%27%2C%20%27dns-sec%27%7D'
rocky-1  | For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/404
rocky-1  | 
rocky-1  | During handling of the above exception, another exception occurred:
rocky-1  | 
rocky-1  | Traceback (most recent call last):
rocky-1  |   File "/usr/local/lib/python3.11/site-packages/django/core/handlers/exception.py", line 55, in inner
rocky-1  |     response = get_response(request)
rocky-1  |                ^^^^^^^^^^^^^^^^^^^^^
rocky-1  |   File "/usr/local/lib/python3.11/site-packages/django/core/handlers/base.py", line 197, in _get_response
rocky-1  |     response = wrapped_callback(request, *callback_args, **callback_kwargs)
rocky-1  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
rocky-1  |   File "/usr/local/lib/python3.11/site-packages/django/views/generic/base.py", line 104, in view
rocky-1  |     return self.dispatch(request, *args, **kwargs)
rocky-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
rocky-1  |   File "/usr/local/lib/python3.11/site-packages/django/contrib/auth/mixins.py", line 109, in dispatch
rocky-1  |     return super().dispatch(request, *args, **kwargs)
rocky-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
rocky-1  |   File "/usr/local/lib/python3.11/site-packages/django/views/generic/base.py", line 143, in dispatch
rocky-1  |     return handler(request, *args, **kwargs)
rocky-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
rocky-1  |   File "/usr/local/lib/python3.11/site-packages/django/views/generic/base.py", line 226, in get
rocky-1  |     context = self.get_context_data(**kwargs)
rocky-1  |               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
rocky-1  |   File "/app/rocky/onboarding/views.py", line 307, in get_context_data
rocky-1  |     context["plugins"] = self.get_plugins()
rocky-1  |                          ^^^^^^^^^^^^^^^^^^
rocky-1  |   File "/app/rocky/onboarding/views.py", line 273, in get_plugins
rocky-1  |     plugins = katalogus.get_plugins(ids=plugin_ids)  # type: ignore
rocky-1  |               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
rocky-1  |   File "/app/rocky/katalogus/client.py", line 300, in get_plugins
rocky-1  |     return self._katalogus_client.get_plugins(self._member.organization.code, **params)
rocky-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
rocky-1  |   File "/app/rocky/katalogus/client.py", line 197, in get_plugins
rocky-1  |     response = self.session.get(f"/v1/organisations/{organization_code}/plugins", params=params)
rocky-1  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
rocky-1  |   File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1066, in get
rocky-1  |     return self.request(
rocky-1  |            ^^^^^^^^^^^^^
rocky-1  |   File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 837, in request
rocky-1  |     return self.send(request, auth=auth, follow_redirects=follow_redirects)
rocky-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
rocky-1  |   File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 926, in send
rocky-1  |     response = self._send_handling_auth(
rocky-1  |                ^^^^^^^^^^^^^^^^^^^^^^^^^
rocky-1  |   File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 954, in _send_handling_auth
rocky-1  |     response = self._send_handling_redirects(
rocky-1  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
rocky-1  |   File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1011, in _send_handling_redirects
rocky-1  |     raise exc
rocky-1  |   File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 994, in _send_handling_redirects
rocky-1  |     hook(response)
rocky-1  |   File "/app/rocky/katalogus/client.py", line 164, in verify_response
rocky-1  |     raise KATalogusNotAllowedError("Access to resource not allowed")
rocky-1  | katalogus.client.KATalogusNotAllowedError: <exception str() failed>
rocky-1  | 
rocky-1  | During handling of the above exception, another exception occurred:
rocky-1  | 
rocky-1  | Traceback (most recent call last):
rocky-1  |   File "/usr/local/lib/python3.11/site-packages/django/core/handlers/exception.py", line 55, in inner
rocky-1  |     response = get_response(request)
rocky-1  |                ^^^^^^^^^^^^^^^^^^^^^
rocky-1  |   File "/usr/local/lib/python3.11/site-packages/django/utils/deprecation.py", line 134, in __call__
rocky-1  |     response = response or self.get_response(request)
rocky-1  |                            ^^^^^^^^^^^^^^^^^^^^^^^^^^
rocky-1  |   File "/usr/local/lib/python3.11/site-packages/django/core/handlers/exception.py", line 57, in inner
rocky-1  |     response = response_for_exception(request, exc)
rocky-1  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
rocky-1  |   File "/usr/local/lib/python3.11/site-packages/django/core/handlers/exception.py", line 140, in response_for_exception
rocky-1  |     response = handle_uncaught_exception(
rocky-1  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^
rocky-1  |   File "/usr/local/lib/python3.11/site-packages/django/core/handlers/exception.py", line 181, in handle_uncaught_exception
rocky-1  |     return debug.technical_500_response(request, *exc_info)
rocky-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
rocky-1  |   File "/usr/local/lib/python3.11/site-packages/django/views/debug.py", line 69, in technical_500_response
rocky-1  |     html = reporter.get_traceback_html()
rocky-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
rocky-1  |   File "/usr/local/lib/python3.11/site-packages/django/views/debug.py", line 428, in get_traceback_html
rocky-1  |     c = Context(self.get_traceback_data(), use_l10n=False)
rocky-1  |                 ^^^^^^^^^^^^^^^^^^^^^^^^^
rocky-1  |   File "/usr/local/lib/python3.11/site-packages/django/views/debug.py", line 417, in get_traceback_data
rocky-1  |     c["exception_value"] = str(self.exc_value)
rocky-1  |                            ^^^^^^^^^^^^^^^^^^^
rocky-1  | TypeError: __str__ returned non-string (type __proxy__)
rocky-1  | 
rocky-1  | During handling of the above exception, another exception occurred:
......
rocky-1  |   File "/usr/local/lib/python3.11/site-packages/django/template/base.py", line 961, in render_annotated
rocky-1  |     return self.render(context)
rocky-1  |            ^^^^^^^^^^^^^^^^^^^^
rocky-1  |   File "/usr/local/lib/python3.11/site-packages/django/template/defaulttags.py", line 326, in render
rocky-1  |     return nodelist.render(context)
rocky-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^
rocky-1  |   File "/usr/local/lib/python3.11/site-packages/django/template/base.py", line 1000, in render
rocky-1  |     return SafeString("".join([node.render_annotated(context) for node in self]))
rocky-1  |                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
rocky-1  |   File "/usr/local/lib/python3.11/site-packages/django/template/base.py", line 1000, in <listcomp>
rocky-1  |     return SafeString("".join([node.render_annotated(context) for node in self]))
rocky-1  |                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
rocky-1  |   File "/usr/local/lib/python3.11/site-packages/django/template/base.py", line 961, in render_annotated
rocky-1  |     return self.render(context)
rocky-1  |            ^^^^^^^^^^^^^^^^^^^^
rocky-1  |   File "/usr/local/lib/python3.11/site-packages/django/template/defaulttags.py", line 326, in render
rocky-1  |     return nodelist.render(context)
rocky-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^
rocky-1  |   File "/usr/local/lib/python3.11/site-packages/django/template/base.py", line 1000, in render
rocky-1  |     return SafeString("".join([node.render_annotated(context) for node in self]))
rocky-1  |                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
rocky-1  |   File "/usr/local/lib/python3.11/site-packages/django/template/base.py", line 1000, in <listcomp>
rocky-1  |     return SafeString("".join([node.render_annotated(context) for node in self]))
rocky-1  |                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
rocky-1  |   File "/usr/local/lib/python3.11/site-packages/django/template/base.py", line 961, in render_annotated
rocky-1  |     return self.render(context)
rocky-1  |            ^^^^^^^^^^^^^^^^^^^^
rocky-1  |   File "/usr/local/lib/python3.11/site-packages/django/template/base.py", line 1059, in render
rocky-1  |     output = self.filter_expression.resolve(context)
rocky-1  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
rocky-1  |   File "/usr/local/lib/python3.11/site-packages/django/template/base.py", line 737, in resolve
rocky-1  |     new_obj = func(obj, *arg_vals)
rocky-1  |               ^^^^^^^^^^^^^^^^^^^^
rocky-1  |   File "/usr/local/lib/python3.11/site-packages/django/template/defaultfilters.py", line 48, in _dec
rocky-1  |     first = str(first)
rocky-1  |             ^^^^^^^^^^
rocky-1  | TypeError: __str__ returned non-string (type __proxy__)
rocky-1  | [28/Oct/2024 12:05:09] "GET /en/onboarding/aa/step/setup-scan/select-plugins/?ooi=URL%7Cinternet%7Chttps%3A%2F%2Fmispo.es%
2F&report_type=dns-report HTTP/1.1" 500 59
rocky-1  | {"request": "GET /en/aa/", "user_agent": "Mozilla/5.0 (X11; Linux x86_64; rv:128.0) Gecko/20100101 Firefox/128.0", "event"
: "request_started", "user_id": 1, "ip": "172.30.0.1", "request_id": "38ea8f69-e524-42b1-8222-1f026fc22dca", "level": "info", "timest
amp": "2024-10-28T12:05:09.616660"}

Donnype added 12 commits October 18, 2024 10:35

Rename KATalogusClientV1 to KATalogusClient

01fb72c

Add organization as an argument to all organization specific methods …

43da7b1

…on the KATalogusClient

Update all method calls with the organization_code argument

0bc7425

Create katalogus without organization code

b899d7e

Call methods with organization code

Style fixes

f3b49bd

Generic status error handling

368bfe0

First version of organization-specific katalogus client

e0ddbea

Some refactoring around the client usage

Remove generated normalizer name workaround

35fa19f

Use KATalogus in most places (again)

0ea91fa

More fixes

ab6c2e8

Some test fixes

2c1cbef

Final test fixes: handle organization existence properly

1bb0a43

Donnype requested a review from a team as a code owner October 23, 2024 09:35

Donnype changed the title ~~Chore/refactor rocky org clients~~ Refactor KATalogus client in Rocky Oct 23, 2024

Merge branch 'main' into chore/refactor-rocky-org-clients

610ec7e