Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve API parameter types #41

Draft
wants to merge 21 commits into
base: main
Choose a base branch
from

Conversation

Derugon
Copy link
Collaborator

@Derugon Derugon commented Mar 18, 2024

Improve the types of API parameters, by adding types for more modules and parameters, infering stricter types, and fixing some type issues.

To simplify the implementation of most of these changes, the api_params type generator has been rewritten. The changes proposed in this PR are mostly backwards-compatible (using some deprecation annotations), but some type fixes may break existing stuff. All changes are introduced below, potentially breaking changes are annotated with ⚠.

Proposed changes

1. Extract modules from multiple APIs

Use multiple site APIs to generate types, allowing to generate types for extensions or modules that are not available on Wikipedia (including Flow and WikiBase).

This also provides a generic way for the type generator to detect which enum parameters are wiki-dependant (e.g. roles, namespaces, or languages). If incompatible types are detected, their type can be automatically generalized to string, so types are not over-specific. The proposed type generator queries module data on all sites independently, then merges the generated API trees together.

The set of queried APIs can be extended with the SOURCES dict. In this PR it includes most english (and international) MediaWiki sites: MediaWiki, Wikipedia, Wikidata, Wikifunctions, Wikibooks, Wikimedia API Portal, Wikimedia Commons, Meta-Wiki, Wikinews, Wikiquote, Wikisource, Wikiversity, Wikivoyage, Wiktionary, and some test sites.

2. Expose types

Expose all interface and enumeration types within the mw.Api/mw.Api.Params namespaces.

This part follows PR #40, in which API parameter types were left unchanged to prevent changing them twice with this proposal.

3. Change type names

Change the interface naming scheme to be based on the actual API path, and not on PHP class names.

This prevents the interface names from being changed when PHP files are renamed or moved (which will happen with MW 1.43, where various API files have changed name and been moved to the MediaWiki\Api namespace), and fixes some parameter interfaces sharing the same name and being merged together. In practice, the same API module can appear multiple times with different parameter prefixes, depending on the modules it is used with. By using API paths, a single module can generate multiple interfaces, preventing this issue.

API paths are not extension-dependant, which allows to search for specific type interfaces more easily with auto-completion. For example:

  • all format=… interfaces are in the mw.Api.Params.Format namespace,
  • all action=query&list=… interfaces in the mw.Api.Params.Action.Query.List namespace.

Note that this makes most interfaces have longer names (e.g. ApiQueryAllLinksParams fully qualified becomes mw.Api.Params.Action.Query.List.AllLinks), which may be mitigated by using namespace aliases.

In this PR, current type names are still accepted and deprecated, using a (hardcoded) list of replacements.

4. Generate types for templated parameters ⚠

Various interfaces use templated parameters to let users specify a variable number of structured arguments. These are generally defined by using a slots parameter to specify a set of identifiers, then by using parameters multiple times, suffixed by one of the previously defined identifiers. For example, the action=compare API uses 2 set of templated parameters (from… and to…) to specify additional properties of one or multiples sources and targets.

As a first approach, templated parameters are generated using template literal types in this PR. This only ensures properly-prefixed parameters have the correct types. For example, the following parameters are generated for the action=compare interface:

interface Compare extends Params {
    ...
    fromslots?: string | string[];
    [k: `fromtext-${string}`]: string;
    [k: `fromsection-${string}`]: string;
    ...
    toslots?: string | string[];
    [k: `totext-${string}`]: string;
    [k: `tosection-${string}`]: string;
    ...
}

This could be improved further, as this approach does not ensure parameter name suffixes are synchronized with each other.

5. Make API parameters stricter in types

Make parameters required, when mentioned on the API module declaration.

action, format, and similar parameter types are narrowed in sub-module interfaces. For example, using mw.Api.Params.Action.Block | mw.Api.Params.Action.Unblock only allows action: "block" | "unblock". ⚠

For compatibility, exported (and deprecated) type aliases still have all their parameters optional.

6. Include a basic JSdoc

Generate JSdoc comments along with the module interfaces and parameters, to include information that can not be expressed with TS types.

In this PR, this includes:

  • default parameter values,
  • sensitive parameters,
  • deprecated modules and parameters,
  • online documentation links.

This does not include:

  • which parameters can be used with GET and which ones with POST,
  • required read/write rights,
  • deprecated or internal parameter values,
  • value bounds, like byte limits for strings or min/max values for numbers.

Other changes

  • Remove JSON-format specific parameters from the root Params interface. They may not be used with other formats, so Params.Format.Json can be used instead. If a user wants to typecheck Params.Action.Move with JSON-specific parameters, Params.Action.Move & Params.Format.Json can be used instead.
  • Sort interfaces. The proposed system builds a type hierarchy, then folds it in pre-order.
  • Disable tslint rules on specific lines and not for the entire file, so unintended error cas still be detected.

Adrien LESÉNÉCHAL added 4 commits March 18, 2024 15:38
to split data extraction from TS code formatting, and allow further improvements
- make some API parameters required, based on module information,
- infer `action` and `format` parameter value depending on the params interface used
- disable `tslint` rules in specific lines and not globally, so unintended error cas still be detected
- add missing `fm` formats
- fix some interfaces sharing the same name
- sort interface to not mix up "action" and "format" interfaces
- use `ApiParams` instead of `UnknownApiParams` on functions doing an actual API query, since parameters need to be correct here,
- do not auto-complete API params that API functions already specify
- force some API params to be specified, when the functions throw an error if missing
- remove JSON-format specific params, use `ApiFormatJsonParams` instead
@Derugon Derugon changed the title Improve API params types Improve API parameter types Mar 18, 2024
AnYiEE added a commit to AnYiEE/types-mediawiki-renovate that referenced this pull request Mar 20, 2024
feat: add more MediaWiki modules check other authors/commits at wikimedia-gadgets#41
@AnYiEE
Copy link
Contributor

AnYiEE commented Mar 20, 2024

Some methods of mwn (e.g. parseWikitext) provide a default value for action, but ApiParseParams treats action as a required property, which is an incompatible issue.

E.g. AnYiEE/AwesomeGadgets@edc450c#diff-e5e7386f4d0511dd5de28802496acf9bdf305877daae5388f87443c1f8450779R492

@Derugon Derugon marked this pull request as draft March 20, 2024 10:56
@AnYiEE
Copy link
Contributor

AnYiEE commented Mar 21, 2024

At the same time, the "variant" property is missing. qiuwenbaike/QiuwenGadgets@6bd8000#diff-ad3705a25402295b5651e9eb78f318e7c91737361988b391165eeff64948fb32R12

Rewrite API queries to recursively retrieve all submodules, not only main modules and action=query submodules

Not that a submodule hierarchy is built, so each submodule remembers where it comes from. This is not used for now, but will be used to fix some perfix & interface name issues in another commit.
@AnYiEE
Copy link
Contributor

AnYiEE commented Mar 23, 2024

Some websites may still be using Flow, but all types related to Flow have been removed in this change (including optional values ​​of the *contentmodel property of some interfaces). Is it necessary to keep them?

The three interfaces ReadingListsApi* extend from ReadingListsApiQueryReadingListsParams -> ApiQueryParams, but their list property type is number, which is incompatible with OneOrMore<string>, causing lint errors.

Some properties should not strictly limit optional values (such as useskin) because some websites do not use certain skins.

- make interface order deterministic, so alternatives are sorted alphabetically and submodules are next to their root module
- fix action/format modules not being stored correctly in cache
AnYiEE added a commit to AnYiEE/types-mediawiki-renovate that referenced this pull request Mar 30, 2024
Adrien LESÉNÉCHAL added 6 commits October 12, 2024 13:01
These changes try to fix a lot of issues with the previously proposed changes. Note that there are still issues remaining, so
There are still issues, so this is still a work in progress.

API modules:
- extract all sub-module interfaces, not only action/command/format/list/meta/prop/submodule
- change interface naming scheme to be closer to the actual API path and minimize conflicts

Sub-modules:
- duplicate interfaces, when they are used with different sub-modules (and property name prefixes)
- add type mapping interfaces, so the list can be extended with other sub-modules
- auto-complete sub-module name properties with mapping interface keys
- allow unknown sub-modules to be specified again

Parameters:
- detect and generalize "slot"-type arguments, to allow other values than `main` and `*`
- add types for templated parameters

JSdoc:
- add online doc links when some are specified
- add `@deprecated` annotations to API parameters
- fix type names in `mw.Api`
- fix module parameters being mixed up in some cases
- fix some incorrect prefixes being added to sub-module parameter names
- remove unused code
As @AnYiEE reported, this causes a lot of issues with existing projects.
This would require a more robust design and a lot more work to be of any use, so I remove it from this PR
Query API module data from multiple APIs at once, then merge the results. Queried APIs are determined by `SOURCES`.
This allows to generate types for more diverse modules, including ones that were removed from wikipedia recently (e.g. Flow modules).
On top of that, we can use the differences between APIs to detect enumerated parameters that are wiki-dependant (e.g. groups, skins).

Other changes:
- fix the script sometimes generating empty types, now it uses `never` in these cases, until a better solution is found.
- generalize any enumerated parameter type with more than 100 possible values, to prevent generating lists of rights and languages.
- change module name capitalization algorithm, to search for pattern combinations in the name, instead of matching the full name once
- add various basic patterns to fix (I hope all) badly capitalized interface names
- fix some `undefined` errors when trying to merge modules from outdated APIs, which allows us to also use test wikis.
- remove `NAME_TYPE_GENERALIZE` and `format` sub-module special cases, since the generator can now handle both of these properly
Derugon pushed a commit to Derugon/types-mediawiki that referenced this pull request Oct 14, 2024
PR wikimedia-gadgets#41 also changes API parameter type names, to fix various name conflicts and parameter prefix issues, so I move it from this PR to PR wikimedia-gadgets#41.
Adrien LESÉNÉCHAL added 6 commits October 17, 2024 14:26
- expose all types within `mw.Api`
- split interface paths into sub-namespaces, allowing to use type aliases to reduce type names when desired
- add deprecated type aliases, mapping old naming scheme to exposed (and renamed) types
- fix handling of `required` values: if a site says a parameter is required and another one says it is optional, then we want to make it optional
- cleaning up some stuff, started to document some things
- move types to a separate declaration file to use the (easier to read) TS syntax
- rewrite ModuleMerger to make it easier to follow
- add more strategies to handle incompatibilities between sites
- fix ModuleLoader breaking the whole process when at least one site is unavailable (e.g. under maintenance, which appened a few days ago with labtestwikitech.wikimedia.org)
- add missing API module parameter properties to the TS declaration file, not impacting the generated TS code for now
so we do not generate more than expected
Change parameter type system, to make better deductions & generate better TS code. Related changes:
- detect and remove useless type enum members, that are already specified behind a type alias
- fix enum type values wrongly disappearing when the value is generalized (e.g. string literals disappearing when the type is generalized to number)
- fix some enum type members not being ordered properly
Other changes:
- use the default value of a module parameter to detect whether narrowed optional parameters should still be optional or not (i.e. if narrowed in a range that does not include the default values, then it no longer is optional)
- show default values in JSdoc
- show which parameters are sensible in JSdoc
Show numbers as numbers, fix empty strings by putting a text for now, and show arrays as a list of strings
Add a basic type replacement system to replace common parameter types.
Currently used with namespaces and token types, will be expanded later.
@Derugon Derugon mentioned this pull request Oct 23, 2024
Adrien LESÉNÉCHAL added 3 commits October 24, 2024 17:38
- add `mw.Api.Toggle`, to simplify some enum types
- expose, rename, and deprecate `ApiAssert` and `ApiLegacyTokenType`
- rename `mw.Api.Token` to `mw.Api.TokenType`
Revert parameter strictness changes.
This would require more work and is not a priority, so I remove it from this PR.

Also remove the use of deprecated type aliases added in the last commits.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants