plugin: reload using new interface

[Exirel]

Based on #1479 - this PR starts at the plugin: add AbstractPluginHandler.reload commit.

Using the new plugin interface defined in sopel.plugins, I replaced how plugins are reloaded in sopel.modules.reload:

• sopel.bot.Sopel now exposes methods to add, remove, and reload plugins
• sopel.modules.reload uses the bot's method to reload plugins (all of them, or only one)
• sopel.plugin.handlers.AbstractPluginHandler defines a new reload method
• sopel.plugin.handlers.AbstractPluginHandler's implementations use the bot's new methods to add or remove the plugin

It Is Not Perfect And Here Is Why

There are problems unsolved and questions unanswered here:

1. Problem: module tree is not reloaded, only the main file (can and must be fixed),
2. Problem: Python's imp.reload and importlib.reload do not remove names undefined in a new version of a reloaded module (fix possible only with Py3.4+),
3. Problem: Job's management is still a mess (same as before),
4. Question: should we use (updated) configuration?

Problem 1: Reload Module Tree

This one is a big problem to solve, because there is more than meets the eye:

• reloading the module tree must not reload built-in or externals modules,
• but because of that a reload may fail if an external module has been changed in a way that it requires a reload...

This leads me to conclude that .reload should not be used in production in these cases. And they are hard as hell to fix, or even to warn the user about.

So it's half a problem, half a question. And I don't have the answer either way.

Problem 2: Python's reload Is Weird

Note: this behavior is the one currently in place in Sopel. This is not new.

Both imp.reload and importlib.reload have this weird behavior, explained by this line from the documentation (python2 reload / python3 importlib.reload):

If the new version of a module does not define a name that was defined by the old version, the old definition remains.

Meaning that:

1. a module has a function something, is loaded at Sopel's startup
2. this module is modified, and something does not exist anymore
3. .reload <this-module> command is used
4. something still exists and is seen by Sopel, no matter what we do,

However, this can be fixed... with Python 3.4's public internals (namely, importlib.util and importlib.machinery) (see a snippet of code). By the way, using Py3.4's public machinery would fix Problem 1 too...

Problem 3: Job Are Weird Too

Note: this behavior is the one currently in place in Sopel. This is not new.

I didn't change any of the "unregister a job" behavior, so this is something that requires more work. The good part is: this can be done in a future PR, without any consequences so far, because it's already like that right now.

Question: Does .reload Should Use Updated Configuration?

In Sopel's current version, .reload uses the current configuration (potentially modified since startup), to gather a list of plugins:

• if a plugin has been disabled, it might not be unloaded properly,
• if a plugin has been enabled, it will load it, somehow

And that is a problem for me, because it means inconsistency and unexpected behavior. I would rather have something stable, that always work, in a predicable way, than the current behavior.

That's why this new version reloads the list of plugins as it exists at the moment in the bot's memory (ie. in bot._plugins). We could, if we want, change this behavior to unload a plugin (enabled or not), and to load only the plugin if it's still enabled, and enable all new enabled plugins.

And that's the question: what should we do?

[Exirel]

Looking at issues:

• .reload doesn't run shutdown routines #1395 fixed: calling shutdown on reload
• reload: Reloading packaged modules doesn't work correctly #1056 half fixed: the job/interval command is not fixed (Jobs weird on reload? #831 can be considered a duplicate)
• .reload for modules with subclasses that call super leads to TypeError #1401 not fixed: can not be done with Python 2
• Automatic reloading of PyFilePlugins #871 not fixed: it would make it easier, though

Overall, "reload" in Python is very hard, and I'm not sure we can really fix it. There are too many use cases and corner cases where we don't have the right tools for that.

[Exirel]

So, after further consideration, I've some issues to fix first.

[Exirel]

So, I decided to merge this branch into #1479 because it makes my job way easier, and I think it does not make sense anymore to separate the new loading from reloading using new loading.

So, I keep that around for history (I keep the branch as-is, too), but everything will happen in #1479 now!

[dgw]

So, I decided to merge this branch into #1479 because it makes my job way easier, and I think it does not make sense anymore to separate the new loading from reloading using new loading.

Makes sense to me. :)