Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rearrange SPEC sections #256

Merged
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 11 additions & 11 deletions quickstart.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
SPEC Quickstart
===============

Quickly setup stub for new SPEC proposal.
Quickly setup stub for new SPEC.
"""

from datetime import datetime
Expand Down Expand Up @@ -33,28 +33,28 @@
## Description

<!--
Briefly and clearly describe the proposal.
Explain the general need and the advantages of this specific proposal.
If relevant, include examples of how the new functionality would be used,
intended use-cases, and pseudo-code illustrating its use.
Briefly and clearly describe the recommendation.
-->

## Implementation
### Core Project Endorsement

<!--
Discuss how this would be implemented.
Briefly discuss what it means for a core project to endorse this SPEC.
-->

### Core Project Endorsement
### Ecosystem Adoption

<!--
Discuss what it means for a core project to endorse this SPEC.
Briefly discuss what it means for a project to adopt this SPEC.
-->

### Ecosystem Adoption
## Implementation

<!--
Discuss what it means for a project to adopt this SPEC.
Discuss how this would be implemented.
Explain the general need and the advantages of this specific recommendation.
If relevant, include examples of how the new functionality would be used,
intended use-cases, and pseudo-code illustrating its use.
-->

## Notes
Expand Down
40 changes: 19 additions & 21 deletions spec-0000/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,9 +32,19 @@ Specifically, we recommend that:
1. Support for a given version of Python be dropped **3 years** after its initial release.
2. Support for a given version of other core packages be dropped **2 years** after their initial release.

{{< mermaid >}}
{{< include-raw "chart.md" >}}
{{< /mermaid >}}
### Core Project Endorsement

<!--
Briefly discuss what it means for a core project to endorse this SPEC.
-->

### Ecosystem Adoption

<!--
Briefly discuss what it means for a project to adopt this SPEC.
-->

## Implementation

### Motivation

Expand All @@ -52,27 +62,15 @@ In the past, longer support cycles were common.
There were several reasons for this, including the Python 2 / 3 transition, difficulties installing packages, and users needing to use old, operating-system provided versions of Python.
The situation has since improved due to improved installations via binary wheels, virtual environments becoming commonplace, and support for Python 2 being dropped.

### Drop Schedule

{{< include-md "schedule.md" >}}

## Implementation

<!--
Discuss how this would be implemented.
-->
### Support Window

<!--
### Core Project Endorsement

Discuss what it means for a core project to endorse this SPEC.
-->
<!--
{{< mermaid >}}
{{< include-raw "chart.md" >}}
{{< /mermaid >}}

### Ecosystem Adoption
### Drop Schedule

Discuss what it means for a project to adopt this SPEC.
-->
{{< include-md "schedule.md" >}}

## Notes

Expand Down
118 changes: 61 additions & 57 deletions spec-0001/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,58 @@ shortcutDepth: 3

## Description

This SPEC recommends a lazy loading mechanism—targeted at libraries—that avoids import slowdowns
and provides explicit submodule exports.

For example, it allows the following behavior:

```python
import skimage as ski # cheap operation; does not load submodules

ski.filters # cheap operation; loads the filters submodule, but not
# any of its submodules or functions

ski.filters.gaussian(...) # loads the file in which gaussian is implemented
# and calls that function
```

This has several advantages:

1. It exposes a **nested namespace that behaves as a flat namespace**.
This avoids carefully having to import exactly the right combination of submodules, and allows interactive exploration of the namespace in an interactive terminal.

2. It **avoids having to optimize for import cost**.
Currently, developers often move imports inside of functions to avoid slowing down importing their module.
Lazy importing, when implemented through out a library, makes all imports cheap.

3. It provides **direct access to submodules**, avoiding local namespace conflicts.
Instead of doing `import scipy.linalg as sla` to avoid clobbering a local `linalg`, one can now import each library and access its members directly: `import scipy; scipy.linalg`.

### Core Project Endorsement

Endorsing this SPEC means agreeing, in principle, with the advantages of lazy loading described above.

### Ecosystem Adoption

jarrodmillman marked this conversation as resolved.
Show resolved Hide resolved
Adopting this SPEC means implementing, using the `lazy_loader` package or any other mechanism (such as module `__getattr__`), lazy loading of subpackages and, if desired, subpackage attributes.

Lazy loading has been adopted by
[scikit-image](https://github.com/scikit-image/scikit-image/pull/5101)
and [NetworkX](https://github.com/networkx/networkx/pull/4909).
SciPy implements a [subset of lazy
loading](https://github.com/scipy/scipy/pull/15230) which exposes only
subpackages lazily.
A prototype implementation of `lazy_loader` was adapted for
[napari](https://github.com/napari/napari/pull/2816).

<!--
Discuss what it means for a project to adopt this SPEC.
-->

## Implementation

### Background

Early on, most scientific Python packages explicitly imported their submodules.
For example, you would be able to do:

Expand Down Expand Up @@ -56,39 +108,17 @@ from scipy.linalg import eig
eig(...)
```

This SPEC proposes a lazy loading mechanism—targeted at libraries—that avoids import slowdowns and brings back explicit submodule exports, but without slowing down imports.

For example, it allows the following behavior:

```python
import skimage as ski # cheap operation; does not load submodules

ski.filters # cheap operation; loads the filters submodule, but not
# any of its submodules or functions

ski.filters.gaussian(...) # loads the file in which gaussian is implemented
# and calls that function

```

This has several advantages:

1. It exposes a **nested namespace that behaves as a flat namespace**.
This avoids carefully having to import exactly the right combination of submodules, and allows interactive exploration of the namespace in an interactive terminal.

2. It **avoids having to optimize for import cost**.
Currently, developers often move imports inside of functions to avoid slowing down importing their module.
Lazy importing makes imports at any depth in the hierarchy cheap.

3. It provides **direct access to submodules**, avoiding local namespace conflicts.
Instead of doing `import scipy.linalg as sla` to avoid clobbering a local `linalg`, one can now import each library and access its members directly: `import scipy; scipy.linalg`.

### Usage

Python 3.7, with [PEP 562](https://www.python.org/dev/peps/pep-0562/), introduces the ability to override module `__getattr__` and `__dir__`.
In combination, these features make it possible to again provide access to submodules, but without incurring performance penalties.

We propose a [utility library](https://pypi.org/project/lazy_loader/) for easily setting up so-called "lazy imports" so that submodules are only loaded upon accessing them.
jarrodmillman marked this conversation as resolved.
Show resolved Hide resolved
### `lazy_loader`

To make it easier for projects to implement lazy loading of submodules and functions, we provide a utility
library, called `lazy_loader`. It is implemented at
https://github.com/scientific-python/lazy_loader and is
installable from [pypi](https://pypi.org/project/lazy_loader/) and [conda-forge](https://anaconda.org/conda-forge/lazy_loader).

#### Usage

As an example, we will show how to set up lazy importing for `skimage.filters`.
In the library's main `__init__.py`, specify which submodules are lazily loaded:
Expand Down Expand Up @@ -243,12 +273,7 @@ This is when you define a lazily loaded function, say `my_func`, in a file of th
Somehow, the doctest collector modifies the parent module's `__dict__` to include `my_func` (the module, not the function), essentially short circuiting the lazy loader and its ability to provide `my_module.my_func` (the function).
Fortunately, there is an easy way to address this that already aligns with common practice: define `my_func` inside `_my_func.py` instead (note the underscore).

## Implementation

Lazy loading is implemented at
https://github.com/scientific-python/lazy_loader and is
pip-installable as
[lazy_loader](https://pypi.org/project/lazy_loader/).
#### YAML files

Once a lazy import interface is implemented, other interesting options
become available (but is not implemented in `lazy_loader`).
Expand Down Expand Up @@ -278,27 +303,6 @@ In the mean time, we now have the necessary mechanisms to implement it ourselves

[^cannon]: Cannon B., personal communication, 7 January 2021.

### Core Project Endorsement

<!--
Discuss what it means for a core project to endorse this SPEC.
-->

### Ecosystem Adoption

Lazy loading has been adopted by
[scikit-image](https://github.com/scikit-image/scikit-image/pull/5101)
and [NetworkX](https://github.com/networkx/networkx/pull/4909).
SciPy implements a [subset of lazy
loading](https://github.com/scipy/scipy/pull/15230) which exposes only
subpackages lazily.
A prototype implementation of `lazy_loader` was adapted for
[napari](https://github.com/napari/napari/pull/2816).

<!--
Discuss what it means for a project to adopt this SPEC.
-->

## Notes

- The [lazy loading blog post by Brett Cannon](https://snarky.ca/lazy-importing-in-python-3-7/) showed the feasibility of the concept, and informed our design.
Expand Down
22 changes: 2 additions & 20 deletions spec-0002/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,8 @@ endorsed-by:
## Description

<!--
Briefly and clearly describe the proposal.
Explain the general need and the advantages of this specific proposal.
Briefly and clearly describe the recommendation.
Explain the general need and the advantages of this specific recommendation.
If relevant, include examples of how the new functionality would be used,
intended use-cases, and pseudo-code illustrating its use.
-->
Expand All @@ -37,24 +37,6 @@ without rewriting code.
This SPEC focuses on the rationale for these mechanisms, and provides
links to implementations related technical discussions.

## Implementation

<!--
Discuss how this would be implemented.
-->

### Core Project Endorsement

<!--
Discuss what it means for a core project to endorse this SPEC.
-->

### Ecosystem Adoption

<!--
Discuss what it means for a project to adopt this SPEC.
-->

## Notes

<!--
Expand Down
12 changes: 12 additions & 0 deletions spec-0003/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,18 @@ As active members of the scientific Python and open-source software (OSS) commun

It is important to note that accessibility is an ongoing journey, and you need not be overwhelmed by the many recommendations outlined in the provided resources. Taking an incremental approach allows for continuous improvement, ensuring that each enhancement makes technology more accessible and user-friendly.

### Core Project Endorsement

<!--
Briefly discuss what it means for a core project to endorse this SPEC.
-->

### Ecosystem Adoption

<!--
Briefly discuss what it means for a project to adopt this SPEC.
-->

## Implementation

### 1. Alt text
Expand Down
33 changes: 13 additions & 20 deletions spec-0004/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,14 +20,7 @@ endorsed-by:

## Description

<!--
Briefly and clearly describe the proposal.
Explain the general need and the advantages of this specific proposal.
If relevant, include examples of how the new functionality would be used,
intended use-cases, and pseudo-code illustrating its use.
-->

This SPEC describes how to test against nightly wheels of several widely used
This SPEC recommends how to test against nightly wheels of several widely used
projects and how to create nightly wheels for your project. The document use the word
_nightly_ to refer to some semi regular interval, like daily, weekly or every three days.

Expand All @@ -42,6 +35,18 @@ you to give feedback about upcoming changes. As with testing against nightlies o
your dependencies this gives your dependents a chance to report problems before they
find their way into a release.

### Core Project Endorsement

<!--
Discuss what it means for a core project to endorse this SPEC.
-->

### Ecosystem Adoption

<!--
Discuss what it means for a project to adopt this SPEC.
-->

## Implementation

This section outlines how to implement using and building nightly wheels. We assume your
Expand Down Expand Up @@ -135,18 +140,6 @@ At that point, let the user know that they have been added and that they can cre
access token (as outlined above.) They can now upload new wheels and perform maitenance
actions on their project.

## Core Project Endorsement

<!--
Discuss what it means for a core project to endorse this SPEC.
-->

## Ecosystem Adoption

<!--
Discuss what it means for a project to adopt this SPEC.
-->

## Notes

<!--
Expand Down
Loading