Question about accepted names for type stub–only packages

[Sachaa-Thanasius]

Forgive me if I ramble a bit; I want to trace my thought process while trying to figure this out.

I'm currently making a package only with type stubs for another library and by default assumed the name of it should be libraryname-stubs. Deciding to see where that assumption comes from, I looked at the official typing docs and see what it says about stub-only package names:

typing docs:

By convention, a stub package for a library called widget would be named types-widget.

No mention there of the -stubs convention that I could tell. I wanted to see how that tracks with the type checker I use and whether it would automatically detect stub-only packages prefixed with types- and -stubs. That's when I found PEP 561, which specifies that the names for these types of packages must end in -stubs. Okay, types- isn't mentioned in there; maybe the type checkers do in their docs.

pyright docs:

For a given package, try to resolve first using a stub package. Stub packages, as defined in PEP 561, are named the same as the original package but with “-stubs” appended.

mypy docs:

If the package is stub-only (not imported at runtime), the package should have a prefix of the runtime package name and a suffix of -stubs.

No mention of types-.

I guess my ultimate questions from all this are the following:

1. Do type checkers implement logic to automatically search for stubs in packages prefixed with types-?
2. Should they, i.e. are they specced to automatically find stubs in such packages with that prefix (maybe in another pep that I missed)?
3. If the answers to the above are no, should the name recommendation line in the typing docs be fixed?

Sorry for the long-winded post about such a small thing, but I'd appreciate any clarification for what seems like a small inconsistency.

[JelleZijlstra]

One point to understand here is the difference between distributions (the things you install) and packages (the things you import, or the names of directories in site-packages). PEP 561 is clear that the package must be named widget-stubs, and that's what type checkers will look for on the file system. However, the distribution, the thing you install, is conventionally called types-widget. It's unfortunate that the two names don't match, but perhaps difficult to fix now.

[Sachaa-Thanasius]

Ah, that's my mistake. Thank you for explaining; I'll do some more research on the difference. Do you think it would be worth adding a note about the naming convention for stub-only packages in the typing docs somewhere? Or maybe if it already exists, directing me to it, in case I forget about this in the future? Might just be because I'm a noob, but I'd appreciate a clear point of reference.

[JelleZijlstra]

Yes, I think the docs could be improved here.

[Sachaa-Thanasius]

That would be fantastic. Maybe someone could make an issue/accompanying PR for that. I'm no expert, but if I can help the process, I'd love to.

[JelleZijlstra]

Yes, it's here: https://github.com/python/typing/blob/master/docs/source/stubs.rst#distribution

Thank you!

[hauntsaninja]

Here's another part of the history. Once upon a time, most stub distributions (and of course packages therein) were named widget-stubs.

Then at some point, typeshed wanted to ship stubs for its third party packages. Because this is sort of "official", we wanted a way to convey that, ideally by a separate PyPI namespace, which at the time looked like it might have gotten implemented. But we settled for a prefix, in the hope that types-widget would one day become types/widget.

This then turned into user-facing advice to look for types-widget (in the typing docs), since typeshed is a common source of stubs. And now because namespace never materialised, it's just proven confusing because other people than typeshed also now name their distributions types-*.

See also:
python/typeshed#2491 (comment)
python/typeshed#7434

[Sachaa-Thanasius]

Interesting. I appreciate the additional context!

[Josverl]

This is information that I would have liked to find 4 years ago when I started to create and publish stubs for MicroPython.

Also I wonder why stubgen foo does create a foo.pyi rather than a foo-stubs.pyi which to my understanding is clearly in conflict with the above. (Same with -m or -p )

Now I find that while I apparently got the name conventions the wrong-way around, everything does work. So should I morph and upset the current users ?

[henribru]

I don't think you got the convention wrong at the time, I'm pretty sure the -stubs suffix was the common choice for the distribution name back then. The types- prefix comes from when typeshed started publishing their stubs on PyPi. This didn't occur until February of 2021. Reading through python/typeshed#2491 it looks like there was consensus for the prefix in 2020, but I don't think anyone would know that unless they were an active participant in that discussion. And at the time it was also natural in my mind to assume that this prefix was only for the typeshed packages and not for stub packages in general. I still kind of think like that for my packages, keeping them with the -stubs suffix makes it simpler in the future if I decide I want to get my stubs into typeshed since I won't have to transfer the permissions and there won't be any version clashes and such.

Anyway, if you look on PyPi there are a lot of packages with the -stubs suffix, so I wouldn't worry too much about it.

[srittau]

Also, for security reasons, it was always the plan that the types- namespace was reserved for typeshed. Unfortunately, reserved namespaces are still not a thing on PyPI. See pypi/warehouse#4967.

[Josverl]

@JelleZijlstra , can you please check my understanding of your explanation.
form your explanation on distribution and package naming conventions for type-only stubs I extrapolate the following:

Lets assume a distribution petstore with the following structure:

petstore
├── __init__.py
├── Parrot.py
└── Hamster.py

To create a type-only distribution for petstore I see four options:

Name: either types-petstore or petstore-stubs

Internal structure:

1) Single file	2) -stub file	3) folder	4) -stub folder
petstore.pyi	petstore-stubs.pyi	petstore	petstore-stubs
		├── __init__.pyi	├── __init__.pyi
		├── Parrot.pyi	├── Parrot.pyi
		└── Hamster.pyi	└── Hamster.pyi

1. Single file
   PEP 561: "It is preferred that the *.pyi stubs are alongside the corresponding *.py files."
   This is the current approach. It has the advantage of being simple and easy to use.
   The majority of stubs on typeshed are single file stubs.

2. -stub file
   Neither mypy nor pyright seem to support this approach.

3. folder
   This is a popular approach for stubs for the larger or more complex distibutions in the typeshed repository.

4. -stub folder
   PEP 561: "stubs can also be put in a separate package and distributed separately. Third parties can also find this method useful if they wish to distribute stub files. The name of the stub package MUST follow the scheme foopkg-stubs for type stubs for the package named foopkg."
   I have not been able to find any stubs in the typeshed repository that use this approach.

   Based on a few simple tests

   • it is supported by pyright
   • it does not seem to be supported by mypy ( I suppose Im missing something here)

So my questions are:

• Is my understanding correct?
• Is there a reason why the typeshed repository does not use the -stub folder approach?
• Is there a reason why mypy does not support the -stub folder approach,I think am I missing something here?

[hauntsaninja]

There are a few misconceptions here. Do not look at the layout of the source in typeshed, look at what you actually get when you run pip install types-requests.

• The distribution name (aka the name on PyPI) is types-requests
• The folder that gets installed in site-packages is requests-stubs (as mandated by PEP 561)

Out of the things you've listed, 2 is something that will never work. 1 is not part of PEP 561 and is something that would likely only work via type checker specific mechanisms like MYPYPATH. 3 needs a py.typed file to be valid. 4 is the only thing you have that will work as you've mentioned (and yes, it is supported by mypy).

[Josverl]

On 42nd reading of the PEP, a Package is indeed explicitly defined as a directory containing modules , (
so 2) is indeed not an option for this reason.
I Agree option 3) is not according to the PEP - but I do note that it works for a lot of cases. but then option 4) is to be preferred.

What I find misleading/unclear is that PEP 561 does explicitly mentions 1) as the preferred method for stub only packages; At least that has been my understanding of the first sentence of : https://peps.python.org/pep-0561/#stub-only-packages, and I find that it works quite well with both PyRight and MyPy

Now based on this conversation it appears that first sentence does actually not refer to stub-only packages ( as stub-only distributions or packages will not contain any .py files )

For me this underlines that English is not a strongly typed language either,
and that an example of the recommended and supported structures and naming conventions would help prevent misconceptions and improve the overall ecosystem.

So my final understanding is this:

Distribution : types-petstore
contains the stub-only package : petstore-stubs , with the following structure

petstore-stubs
├── __init__.pyi
├── Parrot.pyi
└──  Hamster.pyi

In addition stub.pyi files should be either

• alongside the .py file ( 1st party distributions )
• inside a type-* distribution, and inside a stub-only package ( 1st or 3rd party distributions)

I assume that my current (different) 3rd party distribution and package format happens to place the distributed stub.pyi files in the same location where a 1st party package might place them, and hence they get used by the typecheckers.

So thanks all for the clarification and helping me understand the intended structure better.
I do have some changes to implement across all published distributions, but with a clear rationale.