Skip to content

Commit

Permalink
Rearrange SPEC sections
Browse files Browse the repository at this point in the history
As discussed in the last steering committee meeting.
See scientific-python#247.
  • Loading branch information
jarrodmillman committed Aug 2, 2023
1 parent d0f436a commit c83a089
Show file tree
Hide file tree
Showing 7 changed files with 126 additions and 128 deletions.
18 changes: 9 additions & 9 deletions quickstart.py
Original file line number Diff line number Diff line change
Expand Up @@ -34,27 +34,27 @@
<!--
Briefly and clearly describe the proposal.
Explain the general need and the advantages of this specific proposal.
If relevant, include examples of how the new functionality would be used,
intended use-cases, and pseudo-code illustrating its use.
-->
## Implementation
### Core Project Endorsement
<!--
Discuss how this would be implemented.
Briefly discuss what it means for a core project to endorse this SPEC.
-->
### Core Project Endorsement
### Ecosystem Adoption
<!--
Discuss what it means for a core project to endorse this SPEC.
Briefly discuss what it means for a project to adopt this SPEC.
-->
### Ecosystem Adoption
## Implementation
<!--
Discuss what it means for a project to adopt this SPEC.
Discuss how this would be implemented.
Explain the general need and the advantages of this specific proposal.
If relevant, include examples of how the new functionality would be used,
intended use-cases, and pseudo-code illustrating its use.
-->
## Notes
Expand Down
40 changes: 19 additions & 21 deletions spec-0000/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,9 +32,19 @@ Specifically, we recommend that:
1. Support for a given version of Python be dropped **3 years** after its initial release.
2. Support for a given version of other core packages be dropped **2 years** after their initial release.

{{< mermaid >}}
{{< include-raw "chart.md" >}}
{{< /mermaid >}}
### Core Project Endorsement

<!--
Discuss what it means for a core project to endorse this SPEC.
-->

### Ecosystem Adoption

<!--
Discuss what it means for a project to adopt this SPEC.
-->

## Implementation

### Motivation

Expand All @@ -52,27 +62,15 @@ In the past, longer support cycles were common.
There were several reasons for this, including the Python 2 / 3 transition, difficulties installing packages, and users needing to use old, operating-system provided versions of Python.
The situation has since improved due to improved installations via binary wheels, virtual environments becoming commonplace, and support for Python 2 being dropped.

### Drop Schedule

{{< include-md "schedule.md" >}}

## Implementation

<!--
Discuss how this would be implemented.
-->

<!--
### Core Project Endorsement
### Support Window

Discuss what it means for a core project to endorse this SPEC.
-->
<!--
{{< mermaid >}}
{{< include-raw "chart.md" >}}
{{< /mermaid >}}

### Ecosystem Adoption
### Drop Schedule

Discuss what it means for a project to adopt this SPEC.
-->
{{< include-md "schedule.md" >}}

## Notes

Expand Down
119 changes: 62 additions & 57 deletions spec-0001/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,59 @@ shortcutDepth: 3

## Description

This SPEC recommends a lazy loading mechanism—targeted at libraries—that avoids import slowdowns
and provides explicit submodule exports.

For example, it allows the following behavior:

```python
import skimage as ski # cheap operation; does not load submodules

ski.filters # cheap operation; loads the filters submodule, but not
# any of its submodules or functions

ski.filters.gaussian(...) # loads the file in which gaussian is implemented
# and calls that function
```

This has several advantages:

1. It exposes a **nested namespace that behaves as a flat namespace**.
This avoids carefully having to import exactly the right combination of submodules, and allows interactive exploration of the namespace in an interactive terminal.

2. It **avoids having to optimize for import cost**.
Currently, developers often move imports inside of functions to avoid slowing down importing their module.
Lazy importing makes imports at any depth in the hierarchy cheap.

3. It provides **direct access to submodules**, avoiding local namespace conflicts.
Instead of doing `import scipy.linalg as sla` to avoid clobbering a local `linalg`, one can now import each library and access its members directly: `import scipy; scipy.linalg`.


### Core Project Endorsement

<!--
Discuss what it means for a core project to endorse this SPEC.
-->

### Ecosystem Adoption

Lazy loading has been adopted by
[scikit-image](https://github.com/scikit-image/scikit-image/pull/5101)
and [NetworkX](https://github.com/networkx/networkx/pull/4909).
SciPy implements a [subset of lazy
loading](https://github.com/scipy/scipy/pull/15230) which exposes only
subpackages lazily.
A prototype implementation of `lazy_loader` was adapted for
[napari](https://github.com/napari/napari/pull/2816).

<!--
Discuss what it means for a project to adopt this SPEC.
-->

## Implementation

### Background

Early on, most scientific Python packages explicitly imported their submodules.
For example, you would be able to do:

Expand Down Expand Up @@ -56,39 +109,17 @@ from scipy.linalg import eig
eig(...)
```

This SPEC proposes a lazy loading mechanism—targeted at libraries—that avoids import slowdowns and brings back explicit submodule exports, but without slowing down imports.

For example, it allows the following behavior:

```python
import skimage as ski # cheap operation; does not load submodules

ski.filters # cheap operation; loads the filters submodule, but not
# any of its submodules or functions

ski.filters.gaussian(...) # loads the file in which gaussian is implemented
# and calls that function

```

This has several advantages:

1. It exposes a **nested namespace that behaves as a flat namespace**.
This avoids carefully having to import exactly the right combination of submodules, and allows interactive exploration of the namespace in an interactive terminal.

2. It **avoids having to optimize for import cost**.
Currently, developers often move imports inside of functions to avoid slowing down importing their module.
Lazy importing makes imports at any depth in the hierarchy cheap.

3. It provides **direct access to submodules**, avoiding local namespace conflicts.
Instead of doing `import scipy.linalg as sla` to avoid clobbering a local `linalg`, one can now import each library and access its members directly: `import scipy; scipy.linalg`.

### Usage

Python 3.7, with [PEP 562](https://www.python.org/dev/peps/pep-0562/), introduces the ability to override module `__getattr__` and `__dir__`.
In combination, these features make it possible to again provide access to submodules, but without incurring performance penalties.

We propose a [utility library](https://pypi.org/project/lazy_loader/) for easily setting up so-called "lazy imports" so that submodules are only loaded upon accessing them.
### lazy_loader

Lazy loading is implemented at
https://github.com/scientific-python/lazy_loader and is
pip-installable as
[lazy_loader](https://pypi.org/project/lazy_loader/).

#### Usage

As an example, we will show how to set up lazy importing for `skimage.filters`.
In the library's main `__init__.py`, specify which submodules are lazily loaded:
Expand Down Expand Up @@ -243,12 +274,7 @@ This is when you define a lazily loaded function, say `my_func`, in a file of th
Somehow, the doctest collector modifies the parent module's `__dict__` to include `my_func` (the module, not the function), essentially short circuiting the lazy loader and its ability to provide `my_module.my_func` (the function).
Fortunately, there is an easy way to address this that already aligns with common practice: define `my_func` inside `_my_func.py` instead (note the underscore).

## Implementation

Lazy loading is implemented at
https://github.com/scientific-python/lazy_loader and is
pip-installable as
[lazy_loader](https://pypi.org/project/lazy_loader/).
#### YAML files

Once a lazy import interface is implemented, other interesting options
become available (but is not implemented in `lazy_loader`).
Expand Down Expand Up @@ -278,27 +304,6 @@ In the mean time, we now have the necessary mechanisms to implement it ourselves

[^cannon]: Cannon B., personal communication, 7 January 2021.

### Core Project Endorsement

<!--
Discuss what it means for a core project to endorse this SPEC.
-->

### Ecosystem Adoption

Lazy loading has been adopted by
[scikit-image](https://github.com/scikit-image/scikit-image/pull/5101)
and [NetworkX](https://github.com/networkx/networkx/pull/4909).
SciPy implements a [subset of lazy
loading](https://github.com/scipy/scipy/pull/15230) which exposes only
subpackages lazily.
A prototype implementation of `lazy_loader` was adapted for
[napari](https://github.com/napari/napari/pull/2816).

<!--
Discuss what it means for a project to adopt this SPEC.
-->

## Notes

- The [lazy loading blog post by Brett Cannon](https://snarky.ca/lazy-importing-in-python-3-7/) showed the feasibility of the concept, and informed our design.
Expand Down
17 changes: 0 additions & 17 deletions spec-0002/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,23 +37,6 @@ without rewriting code.
This SPEC focuses on the rationale for these mechanisms, and provides
links to implementations related technical discussions.

## Implementation

<!--
Discuss how this would be implemented.
-->

### Core Project Endorsement

<!--
Discuss what it means for a core project to endorse this SPEC.
-->

### Ecosystem Adoption

<!--
Discuss what it means for a project to adopt this SPEC.
-->

## Notes

Expand Down
12 changes: 12 additions & 0 deletions spec-0003/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,18 @@ As active members of the scientific Python and open-source software (OSS) commun

It is important to note that accessibility is an ongoing journey, and you need not be overwhelmed by the many recommendations outlined in the provided resources. Taking an incremental approach allows for continuous improvement, ensuring that each enhancement makes technology more accessible and user-friendly.

### Core Project Endorsement

<!--
Discuss what it means for a core project to endorse this SPEC.
-->

### Ecosystem Adoption

<!--
Discuss what it means for a project to adopt this SPEC.
-->

## Implementation

### 1. Alt text
Expand Down
24 changes: 12 additions & 12 deletions spec-0004/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,18 @@ you to give feedback about upcoming changes. As with testing against nightlies o
your dependencies this gives your dependents a chance to report problems before they
find their way into a release.

### Core Project Endorsement

<!--
Discuss what it means for a core project to endorse this SPEC.
-->

### Ecosystem Adoption

<!--
Discuss what it means for a project to adopt this SPEC.
-->

## Implementation

This section outlines how to implement using and building nightly wheels. We assume your
Expand Down Expand Up @@ -135,18 +147,6 @@ At that point, let the user know that they have been added and that they can cre
access token (as outlined above.) They can now upload new wheels and perform maitenance
actions on their project.

## Core Project Endorsement

<!--
Discuss what it means for a core project to endorse this SPEC.
-->

## Ecosystem Adoption

<!--
Discuss what it means for a project to adopt this SPEC.
-->

## Notes

<!--
Expand Down
24 changes: 12 additions & 12 deletions spec-0005/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,18 @@ If relevant, include examples of how the new functionality would be used,
intended use-cases, and pseudo-code illustrating its use.
-->

### Core Project Endorsement

<!--
Discuss what it means for a core project to endorse this SPEC.
-->

### Ecosystem Adoption

<!--
Discuss what it means for a project to adopt this SPEC.
-->

## Implementation

<!--
Expand Down Expand Up @@ -56,18 +68,6 @@ workflow" add the following lines as an additional step:
anaconda_nightly_upload_token: ${{ secrets.UPLOAD_TOKEN }}
```

### Core Project Endorsement

<!--
Discuss what it means for a core project to endorse this SPEC.
-->

### Ecosystem Adoption

<!--
Discuss what it means for a project to adopt this SPEC.
-->

## Notes

<!--
Expand Down

0 comments on commit c83a089

Please sign in to comment.