update requirements #222

vincentsarago · 2024-06-17T13:02:48Z

Description

This is the first PR which will update the requirements for the different PC services.

The test should fail right now, but I wanted to see if they were passing for pccommon

Type of change

Please delete options that are not relevant.

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
This change requires a documentation update

How Has This Been Tested?

Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

Checklist:

Please delete options that are not relevant.

I have performed a self-review
Changelog has been updated
Documentation has been updated
Unit tests pass locally (./scripts/test)
Code is linted and styled (./scripts/format)

docker-compose.dev.yml

vincentsarago · 2024-06-17T13:03:21Z

pccommon/pccommon/config/collections.py

+        # https://docs.pydantic.dev/latest/api/config/#pydantic.alias_generators.to_camel
+        "alias_generator": camelize,
+        "populate_by_name": True,
+    }


vincentsarago · 2024-06-17T13:04:12Z

pccommon/pccommon/config/containers.py

+    # https://github.com/pydantic/pydantic/discussions/6388
+    # class Config:
+    #     json_loads = orjson.loads
+    #     json_dumps = orjson_dumps


we can't customize this with pydantic 2 but it seems that pydantic 2 is faster anyway

vincentsarago · 2024-06-17T16:39:21Z

pcstac/pcstac/client.py

-
-        base_conformance_classes.extend(self.extra_conformance_classes)
-
-        return sorted(list(set(base_conformance_classes)))


this was the same as https://github.com/stac-utils/stac-fastapi/blob/80064f77739ddc1acaafa78cb0d8a11f77ecd33e/stac_fastapi/types/stac_fastapi/types/core.py#L348-L357

vincentsarago · 2024-06-17T16:40:43Z

pcstac/pcstac/filter.py

@@ -34,6 +33,4 @@ async def _fetch() -> Dict:
                return queryables

        cache_key = f"{CACHE_KEY_QUERYABLES}:{collection_id}"
-        queryables = await cached_result(_fetch, cache_key, request)
-        headers = {"Content-Type": "application/schema+json"}
-        return JSONResponse(queryables, headers=headers)


should be handled by https://github.com/stac-utils/stac-fastapi/blob/main/stac_fastapi/extensions/stac_fastapi/extensions/core/filter/filter.py#L96-L110

vincentsarago · 2024-06-17T16:41:26Z

pcstac/pcstac/main.py

+    await connect_to_db(app)
+    await connect_to_redis(app)
+    yield
+    await close_db_connection(app)


startup/shutdown events are deprecated and lifespan is the new way to do

vincentsarago · 2024-06-17T16:42:09Z

pcstac/pcstac/main.py

    search_get_request_model=search_get_request_model,
    search_post_request_model=search_post_request_model,
    response_class=ORJSONResponse,
    exceptions={**DEFAULT_STATUS_CODES, **PC_DEFAULT_STATUS_CODES},
+    middleware=[
+        Middleware(BrotliMiddleware),
+        Middleware(ProxyHeaderMiddleware),


both are defined in the default stac-fastapi so they should have been present in the application before

vincentsarago · 2024-06-17T16:44:52Z

I don't seem to be able to run the tests locally:

scripts/test --common
WARN[0000] The "APP_INSIGHTS_INSTRUMENTATION_KEY" variable is not set. Defaulting to a blank string. 
WARN[0000] The "APP_INSIGHTS_INSTRUMENTATION_KEY" variable is not set. Defaulting to a blank string. 
[+] Running 4/0
 ⠿ Container pc-stac-db                       Running                                                                                                                                                                                                                                                                                0.0s
 ⠿ Container planetary-computer-apis-redis-1  Running                                                                                                                                                                                                                                                                                0.0s
 ⠿ Container pcapis-azurite                   Running                                                                                                                                                                                                                                                                                0.0s
 ⠿ Container planetary-computer-apis-stac-1   Running                                                                                                                                                                                                                                                                                0.0s
Running mypy for common...
Success: no issues found in 29 source files
Running black for common...
All done! ✨ 🍰 ✨
29 files would be left unchanged.
Running flake8 for common...
Running unit tests for common...
========================================================================================================================================================== test session starts ===========================================================================================================================================================
platform linux -- Python 3.9.19, pytest-7.4.4, pluggy-1.5.0
rootdir: /opt/src
configfile: pytest.ini
plugins: asyncio-0.18.3, anyio-3.7.1
asyncio: mode=auto
collected 3 items / 3 errors                                                                                                                                                                                                                                                                                                             

================================================================================================================================================================= ERRORS =================================================================================================================================================================
____________________________________________________________________________________________________________________________________________ ERROR collecting pccommon/tests/test_timeouts.py ____________________________________________________________________________________________________________________________________________
ImportError while importing test module '/opt/src/pccommon/tests/test_timeouts.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/lib/python3.9/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
pccommon/tests/test_timeouts.py:11: in <module>
    from pccommon.middleware import add_timeout
pccommon/pccommon/middleware.py:19: in <module>
    from pccommon.tracing import trace_request
pccommon/pccommon/tracing.py:14: in <module>
    from pccommon.config import get_apis_config
pccommon/pccommon/config/__init__.py:4: in <module>
    from pccommon.config.core import PCAPIsConfig
pccommon/pccommon/config/core.py:8: in <module>
    from pydantic_settings import BaseSettings
/usr/lib/python3.9/site-packages/pydantic_settings/__init__.py:1: in <module>
    from .main import BaseSettings, SettingsConfigDict
/usr/lib/python3.9/site-packages/pydantic_settings/main.py:7: in <module>
    from pydantic._internal._config import config_keys
E   ModuleNotFoundError: No module named 'pydantic._internal'
____________________________________________________________________________________________________________________________________________ ERROR collecting pccommon/tests/test_tracing.py _____________________________________________________________________________________________________________________________________________
ImportError while importing test module '/opt/src/pccommon/tests/test_tracing.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/lib/python3.9/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
pccommon/tests/test_tracing.py:1: in <module>
    from pccommon.tracing import _parse_cqljson
pccommon/pccommon/tracing.py:14: in <module>
    from pccommon.config import get_apis_config
pccommon/pccommon/config/__init__.py:4: in <module>
    from pccommon.config.core import PCAPIsConfig
pccommon/pccommon/config/core.py:8: in <module>
    from pydantic_settings import BaseSettings
/usr/lib/python3.9/site-packages/pydantic_settings/__init__.py:1: in <module>
    from .main import BaseSettings, SettingsConfigDict
/usr/lib/python3.9/site-packages/pydantic_settings/main.py:7: in <module>
    from pydantic._internal._config import config_keys
E   ModuleNotFoundError: No module named 'pydantic._internal'
______________________________________________________________________________________________________________________________________ ERROR collecting pccommon/tests/config/test_render_config.py ______________________________________________________________________________________________________________________________________
ImportError while importing test module '/opt/src/pccommon/tests/config/test_render_config.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/lib/python3.9/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
pccommon/tests/config/test_render_config.py:3: in <module>
    from pccommon.config import get_render_config
pccommon/pccommon/config/__init__.py:4: in <module>
    from pccommon.config.core import PCAPIsConfig
pccommon/pccommon/config/core.py:8: in <module>
    from pydantic_settings import BaseSettings
/usr/lib/python3.9/site-packages/pydantic_settings/__init__.py:1: in <module>
    from .main import BaseSettings, SettingsConfigDict
/usr/lib/python3.9/site-packages/pydantic_settings/main.py:7: in <module>
    from pydantic._internal._config import config_keys
E   ModuleNotFoundError: No module named 'pydantic._internal'
======================================================================================================================================================== short test summary info =========================================================================================================================================================
ERROR pccommon/tests/test_timeouts.py
ERROR pccommon/tests/test_tracing.py
ERROR pccommon/tests/config/test_render_config.py

I cannot replicate outside the docker env, this might be an issue when compiling pydantic rust code 🤷

vincentsarago · 2024-06-18T19:32:26Z

pccommon/pccommon/config/core.py

+        # Mypi is complaining about this with
+        # error: Incompatible types (expression has type "str",
+        # TypedDict item "extra" has type "Extra")
+        "extra": "ignore",  # type: ignore


todo: remove type: ignore and see if this pass in the CI

vincentsarago · 2024-06-18T19:34:06Z

pccommon/tests/config/test_render_config.py

@@ -74,12 +74,12 @@ def test_get_render_config() -> None:

 def test_render_config_parse_max_items() -> None:
    config = {
-        "render_params": [],
+        "render_params": {},


not sure how this passed before

vincentsarago · 2024-06-18T19:34:41Z

pcfuncs/Dockerfile

@@ -1,4 +1,4 @@
-FROM mcr.microsoft.com/azure-functions/python:4-python3.8
+FROM mcr.microsoft.com/azure-functions/python:4-python3.10


can revert to 3.8 if needed

vincentsarago · 2024-06-18T19:35:38Z

pcfuncs/animation/settings.py

-        env_nested_delimiter = "__"
+    model_config = {
+        "env_prefix": ANIMATION_SETTINGS_PREFIX,
+        "env_nested_delimiter": "__",  # type: ignore


same issue locally with mypy complaining about error: Incompatible types (expression has type "str",

vincentsarago · 2024-06-18T19:37:10Z

pcfuncs/image/settings.py

+    settings = ImageSettings()  # type: ignore
+    logger.info(f"API URL: {settings.api_root_url}")
+    logger.info(f"Concurrency limit: {settings.tile_request_concurrency}")
+    return settings


replaced ImageSettings.get with an external function which should gives the same result.

FYI: Pydantic was complaining about the .get method

vincentsarago · 2024-06-18T19:48:27Z

pcstac/tests/resources/test_item.py


 from pcstac.config import get_settings

+# Use a TypeAdapter to parse any datetime strings in a consistent manner
+UtcDatetimeAdapter = TypeAdapter(UtcDatetime)


stac-pydantic now use pydantic Datetime parsing to validate datetime string

pcstac/tests/resources/test_item.py

vincentsarago · 2024-06-19T19:36:08Z

@mmcfarland I'll all done with the dependency updates!

Sadly the pctiler app is not really well tested so the CI won't show if there are breaking changes 😬

I'm not quite aware of the interaction pattern between the Client and the API so I'm just going to 🤞 😅

mmcfarland · 2024-06-19T19:46:42Z

Great! I'll pull this down and do some testing and let you know how things are looking!

vincentsarago · 2024-06-26T10:15:41Z

FYI: there are some breaking changes that needs to happen in stac-fastapi (ref: stac-utils/stac-fastapi#713) so we might need to update this PR

vincentsarago · 2024-07-10T09:53:00Z

pcstac/pcstac/main.py

@@ -50,6 +55,15 @@

 app_settings = get_settings()

+items_get_request_model = PCItemCollectionUri


we can now directly define the GET - /items model. This wasn't used previously

mmcfarland · 2024-07-16T18:58:57Z

Superseded by #233 -- thanks for this baseline @vincentsarago!

update pccommon requirements

784ca8f

vincentsarago commented Jun 17, 2024

View reviewed changes

docker-compose.dev.yml Show resolved Hide resolved

vincentsarago commented Jun 17, 2024

View reviewed changes

update pcstac requirements

85b9352

vincentsarago changed the title ~~[DRAFT] update pccommon requirements~~ [DRAFT] update requirements Jun 17, 2024

vincentsarago commented Jun 17, 2024

View reviewed changes

vincentsarago closed this Jun 17, 2024

vincentsarago reopened this Jun 17, 2024

vincentsarago added 2 commits June 17, 2024 22:07

update pcfuncs

d8bde91

update pcfuncs/pcstac/pccommon

52d0345

vincentsarago commented Jun 18, 2024

View reviewed changes

pcstac/tests/resources/test_item.py Outdated Show resolved Hide resolved

vincentsarago added 2 commits June 19, 2024 10:57

allow YYYY-MM-DD date format in search

262dc53

update pctiler

e05619b

vincentsarago changed the title ~~[DRAFT] update requirements~~ update requirements Jun 26, 2024

vincentsarago added 2 commits July 10, 2024 11:38

merge from main

c389a88

update stac-fastapi.*

307b93f

vincentsarago commented Jul 10, 2024

View reviewed changes

mmcfarland closed this Jul 16, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

update requirements #222

update requirements #222

vincentsarago commented Jun 17, 2024

vincentsarago Jun 17, 2024

vincentsarago Jun 17, 2024

vincentsarago Jun 17, 2024

vincentsarago Jun 17, 2024

vincentsarago Jun 17, 2024

vincentsarago Jun 17, 2024

vincentsarago commented Jun 17, 2024

vincentsarago Jun 18, 2024

vincentsarago Jun 18, 2024

vincentsarago Jun 18, 2024

vincentsarago Jun 18, 2024

vincentsarago Jun 18, 2024

vincentsarago Jun 18, 2024

vincentsarago commented Jun 19, 2024

mmcfarland commented Jun 19, 2024

vincentsarago commented Jun 26, 2024

vincentsarago Jul 10, 2024

mmcfarland commented Jul 16, 2024


		base_conformance_classes.extend(self.extra_conformance_classes)

		return sorted(list(set(base_conformance_classes)))

		@@ -1,4 +1,4 @@
		FROM mcr.microsoft.com/azure-functions/python:4-python3.8
		FROM mcr.microsoft.com/azure-functions/python:4-python3.10

		@@ -50,6 +55,15 @@

		app_settings = get_settings()

		items_get_request_model = PCItemCollectionUri

update requirements #222

update requirements #222

Conversation

vincentsarago commented Jun 17, 2024

Description

Type of change

How Has This Been Tested?

Checklist:

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vincentsarago commented Jun 17, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vincentsarago commented Jun 19, 2024

mmcfarland commented Jun 19, 2024

vincentsarago commented Jun 26, 2024

Choose a reason for hiding this comment

mmcfarland commented Jul 16, 2024