plugin: new interface

[Exirel]

I was playing for a long time with the idea of a unified Sopel's Plugin interface: a way to abstract how plugins are loaded from the core, so that you can implement various way of finding and loading them without having to change how the bot behave and interact with commands, rules, events, etc.

This PR slightly changes how callables are registered, and totally change how they are reloaded.

Plugin Interface

The Plugin Interface is represented by the sopel.plugins.handlers.AbstractPluginHandler:

• plugin.load(): load the plugin in sys.modules
• plugin.reload(): reload the plugin
• plugin.configure(settings): handle the configuration wizard for the plugin (if necessary)
• plugin.setup(bot): setup the plugin (if necessary)
• plugin.register(bot): register the plugin with the bot
• plugin.shutdown(bot): take action on shutdown (if necessary)
• plugin.is_loaded(): tell if the plugin is loaded
• plugin.has_*() (configure, setup, shutdown): tell if the plugin has this particular method

Under the hood, I kept a lot of what is in sopel.loader, so it isn't a change that big!

What's next?

I would like to add this kind of plugin:

• Python setup.py Entry Point based plugins: a Python package could define an entry point, and that would be used to discover plugins much more efficiently than the old technique of package namespace (as used by sopel_modules.*),
• Python object plugins: an handler that would not take a python module, but an object, to gather its callable, its configure, setup, and shutdown methods,

but that'll be for another day!

Related issues, PR, and future works

• Rework internal APIs for Command Line Tools #1471 is about sopel-* CLI tools: this interface will be very useful to inspect plugins, show their details, configure and enable/disable them (as in Enhancement: Configuring individual modules only? #1519)
• Let's stop using sleep() #1543 is about time.sleep(): with the rework of the JobScheduler and the new cycle of "shutdown, reload, register" that is better, it makes the JobScheduler and the interval decorator a good option for some cases (as requested by remind: use module.interval instead of custom monitoring #1581 or seen in Jobs weird on reload? #831)

[Exirel]

This PR is kind of related to #1471, as it modifies Sopel's internals, and it will definitively makes CLI tools easier to make.

For instance, it'll be much easier to implement a "list all plugins" CLI tool, or "configure this module in particular", etc.

[Exirel]

Edit: fixed!

---

Meh. Travis found a kind of cyclic import. Fine. I'll have to fix that one way or another. That'll be for tomorrow!

[Exirel]

As said on IRC:

11:16:32 Yup, indeed, I forgot about config.core.extra. x)

So I need to update enumerate_plugins to load plugins from these directories ("extra" is a list).

[Exirel]

As said on IRC: I merged #1490 in, so this PR will handle everything related to plugin's loading & reloading.

Also: so many things to fix, make, improve, and else. 😄

[Exirel]

I've figure out a way to:

• remove a circular import problem between sopel.loader, sopel.config, and sopel.plugins
• reduce the usage of sopel.loader to a very minimal set of functions
• put everything into sopel.cli.run and sopel.plugins instead

but I didn't add tests for sopel.plugins, which is not good, and I need to fix that.

And then, eventually, I'll ask for a review!

[dgw]

Not dug into this yet because it's still tagged WIP, but does this basically replace #990, #1053, and possibly other PRs that deal with module loading/unloading/reloading?

[Exirel]

Yup, it does. It's not perfect, because I do not handle (yet) sub-module reloading, but that'll come in later.

[Exirel]

So, for testing purpose, I need #1557 to be merged: I want to test the startup routine of the bot, when it loads plugins. And for that, I need to be able to stop the JobScheduler, otherwise the test suite never stop running. That could be annoying.

[dgw]

Honestly, I didn't go through the whole diff again, just a patchwork of changes made since my last review. But this is "core" enough to Sopel's operation that, once merged, any issues will probably bubble up very quickly as we all work on other stuff.

I'd like a review from one other @sopel-irc/rockstars before merging, so hopefully someone else is up to the task! 😅

[HumorBaby]

I ended up coming across some issues while testing this PR, but it's all related to the CLI (I'll put these in another issue). The plugin interface, however, looks good to me 🚀 (reload not testing given all the issues it has now anyway... can't be worse 😅, so it may be better to get this in, then re-frame the reload issues in the context of this new interface).

[Exirel]

I ended up coming across some issues while testing this PR, but it's all related to the CLI

What kind of issues? 😮

[dgw]

I trust @HumorBaby to open that other issue when he's ready. Unlike some of us (*cough* me *cough*), he's good at remembering what he promised to do. 😁