mypy implementation for adapters in tiled #700

skarakuzu · 2024-03-28T17:21:59Z

beginning of mypy implementation in tiled. It is not ready to merge.

danielballan · 2024-04-01T20:21:58Z

Protocols that we should write up together:

metadata, which is like dict[str, Any] but more specific than Any because we know which types are JSON-serializable, and metadata must be JSON-serializable
AccessPolicy
BaseAdapter (general interface, includes things like metadata(), specs, structure_family, structure
Separate Protocol for each structure family: ArrayAdapter, AwkwardAdapter, ContainerAdapter, SparseAdapter, TableAdapter

danielballan · 2024-04-11T17:06:58Z

tiled/adapters/array.py

-    ):
+        metadata: Optional[JSON] = None,
+        specs: Optional[List[Spec]] = None,
+        access_policy: Optional[Union[SimpleAccessPolicy, DummyAccessPolicy]] = None,


I think this is a good place to use the AccessPolicy protocol. Same for everywhere else this is used.

danielballan · 2024-04-11T17:07:44Z

tiled/adapters/array.py

        array = self._array
-        if slice is not None:
-            array = array[slice]
+        array = array[slice]


Might as well combine this into a single statement.

danielballan · 2024-04-11T17:12:24Z

tiled/adapters/array.py

+    def read(
+        self,
+        slice: Union[
+            int, slice, Tuple[Union[int, slice, EllipsisType], ...], EllipsisType


I seems worthwhile to make a "type alias" for this, like you did for JSON. I suggest the name NDSlice.

danielballan · 2024-04-11T17:13:13Z

tiled/adapters/array.py

-def slice_and_shape_from_block_and_chunks(block, chunks):
+def slice_and_shape_from_block_and_chunks(
+    block: Tuple[int, ...], chunks: Tuple[Tuple[int, ...], ...]
+) -> Tuple[Tuple[slice, ...], Tuple[int, ...]]:


Suggested change

) -> Tuple[Tuple[slice, ...], Tuple[int, ...]]:

) -> Tuple[NDSlice, Tuple[int, ...]]:

danielballan · 2024-04-11T17:13:51Z

tiled/adapters/awkward.py

        return self._metadata

-    def read_buffers(self, form_keys=None):
+    def read_buffers(self, form_keys: Optional[List[str]] = None) -> Dict[str, Any]:


Suggested change

def read_buffers(self, form_keys: Optional[List[str]] = None) -> Dict[str, Any]:

def read_buffers(self, form_keys: Optional[List[str]] = None) -> Dict[str, bytes]:

danielballan · 2024-04-11T17:30:45Z

tiled/adapters/sparse_blocks_parquet.py

        if len(self.blocks) > 1:
            raise NotImplementedError
        uri = self.blocks[(0,) * len(self._structure.shape)]
        data.to_parquet(path_from_uri(uri))

-    def read(self, slice=...):
+    def read(self, slice: Optional[Union[int, slice]]) -> NDArray[Any]:


Suggested change

def read(self, slice: Optional[Union[int, slice]]) -> NDArray[Any]:

def read(self, slice: Optional[Union[int, slice]]) -> sparse.COO:

danielballan · 2024-04-11T17:51:02Z

tiled/adapters/tiff.py

@@ -22,13 +26,23 @@ class TiffAdapter:

    def __init__(
        self,
-        data_uri,
+        data_uri: Union[str, List[str]],


Suggested change

data_uri: Union[str, List[str]],

data_uri: str,

danielballan · 2024-04-11T17:51:57Z

tiled/adapters/zarr.py


 INLINED_DEPTH = int(os.getenv("TILED_HDF5_INLINED_CONTENTS_MAX_DEPTH", "7"))


-def read_zarr(data_uri, structure=None, **kwargs):
+def read_zarr(
+    data_uri: Union[str, List[str]],


Suggested change

data_uri: Union[str, List[str]],

data_uri: str,

danielballan · 2024-04-11T17:54:04Z

tiled/adapters/sparse.py

-    def read_block(self, block, slice=None):
+    def read_block(
+        self, block: Tuple[int, ...], slice: Optional[Union[int, slice]] = None
+    ) -> NDArray[Any]:


Suggested change

) -> NDArray[Any]:

) -> sparse.COO:

danielballan · 2024-04-11T17:54:14Z

tiled/adapters/sparse.py

        coords, data = self.blocks[block]
        _, shape = slice_and_shape_from_block_and_chunks(block, self._structure.chunks)
        arr = sparse.COO(data=data[:], coords=coords[:], shape=shape)
        if slice:
            arr = arr[slice]
        return arr

-    def read(self, slice=None):
+    def read(self, slice: Optional[Union[int, slice]] = None) -> NDArray[Any]:


Suggested change

def read(self, slice: Optional[Union[int, slice]] = None) -> NDArray[Any]:

def read(self, slice: Optional[Union[int, slice]] = None) -> sparse.COO:

…typing

dylanmcreynolds · 2024-04-26T20:31:07Z

tiled/adapters/protocols.py

+        pass
+
+
+if sys.version_info < (3, 9):


Is this PR the time to upgrade Tiled's minimum supported python version to SPEC0, which at present would be 3.10 or greater? We could then remove this Mapping stuff.

One or two users out there have asked if we can give them a 3.8 (or even a 3.7) compatible release up through the first beta, so they have at least a basically usable client available in legacy environments.

I figure we'll fast-forward to 3.10 shortly after tagging beta.

skarakuzu force-pushed the mypy_for_tiled branch 2 times, most recently from e317207 to 03b2408 Compare April 5, 2024 18:44

danielballan reviewed Apr 11, 2024

View reviewed changes

danielballan force-pushed the mypy_for_tiled branch from 15ebb1b to 3f57d4c Compare April 23, 2024 15:49

skarakuzu changed the title ~~started mypy implementation for tiled: Note lots of mypy errors!~~ mypy implementation for adapters in tiled Apr 26, 2024

skarakuzu marked this pull request as ready for review April 26, 2024 15:36

Seher Karakuzu and others added 23 commits April 26, 2024 16:22

some more changes

fe79576

typed zarr.py

1038553

typed tiff.py

7595639

xarray.py typed

06ac19c

some more typing added

a80caba

some more typing. still few problems in mapping.py

786a275

more typing

82a3d01

still 1 mypy problem left

c5ac22b

docstring skeletons added

1a0b167

some more improvement

2265002

solved some docs build problems

c6358fd

few more changes

2777e3c

change type_alliases.py

84e64ac

some more changes

9a9486f

some more changes

64a3fdb

update python to 3.11 in doc compilation

8c32d0b

fixed failing tests

090eda3

Add protocols

1f4536a

change docs python to 3.10

6a20cb9

preliminary addressing to the comments

be14c77

some tests for protocols

8509083

some corrections in protocols tests

1be2018

fix small bug in test_protocols.py

9331763

Seher Karakuzu and others added 21 commits April 26, 2024 16:22

try to solve EllipsisType that only exists in python 3.10

edf4e1f

change python version check

9c7aa41

some more fix

b066f41

Bump Python version of docs build.

5c05e13

tests

ed2a731

some more unit tests for protocols

95ffb51

add accesspolicy protocol tests

4d1f6b0

few more fixes

bb0f108

python 3.8 MutableMapping problem: try to replacse with the one from …

809089c

…typing

try to change collections.abc.MApping with typing.Mapping if python<3.8

126eed2

small fixes for typing errors appeared in python 3.8

b268500

one more typing fix for python 3.8

e9dfa6d

some more fix python3.8

07f717a

Type awkward buffers dict more strictly

6f4e773

Tighten typing on data_uris.

7038db5

Fix typing of partition parameter in read_partition.

9f10eb9

Zarr accepts array data, not dataframe/tabular.

d923f4f

Remove commented unused code

557e32f

Fix typo

7d4a34e

Fix type of partition

d000794

added a changelog entry

453b807

danielballan force-pushed the mypy_for_tiled branch from 1d4b7a1 to 453b807 Compare April 26, 2024 20:24

Make python-version consistent for docs

e863422

dylanmcreynolds reviewed Apr 26, 2024

View reviewed changes

danielballan merged commit 143dd22 into bluesky:main Apr 26, 2024
9 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

mypy implementation for adapters in tiled #700

mypy implementation for adapters in tiled #700

skarakuzu commented Mar 28, 2024

danielballan commented Apr 1, 2024

danielballan Apr 11, 2024

danielballan Apr 11, 2024

danielballan Apr 11, 2024

danielballan Apr 11, 2024

danielballan Apr 11, 2024

danielballan Apr 11, 2024

danielballan Apr 11, 2024

danielballan Apr 11, 2024

danielballan Apr 11, 2024

danielballan Apr 11, 2024

dylanmcreynolds Apr 26, 2024

danielballan Apr 26, 2024

	) -> Tuple[Tuple[slice, ...], Tuple[int, ...]]:
	) -> Tuple[NDSlice, Tuple[int, ...]]:

	def read_buffers(self, form_keys: Optional[List[str]] = None) -> Dict[str, Any]:
	def read_buffers(self, form_keys: Optional[List[str]] = None) -> Dict[str, bytes]:

	def read(self, slice: Optional[Union[int, slice]]) -> NDArray[Any]:
	def read(self, slice: Optional[Union[int, slice]]) -> sparse.COO:

mypy implementation for adapters in tiled #700

mypy implementation for adapters in tiled #700

Conversation

skarakuzu commented Mar 28, 2024

danielballan commented Apr 1, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment