Un- and reload plugins with Deferreds

[s-ol]

I started working on all of this, no doubt the twisted guide was useful. Error handling needs to be improved but the actual unloading part is failsafe now (I think). Ref #58

[johnmaguire]

Closing this in lieu of #87

[s-ol]

hm, it seems like the new commit doesn't show up because you closed this @johnmaguire (don't know if you get notified of this either); you can check it out at afed1bb

[johnmaguire]

This looks pretty good. I made some comments now, and I'll play around with a little tonight. :)

Not sure I want to merge this directly into master when we have this in a good state. I want to investigate what sort of work would need to be done to let plugin commands / event handlers return Deferreds, and maybe do that at the same time.

If we implement Deferred support throughout Cardinal, I think it might justify a 2.1 release, but in that case, there's a couple other things I'd probably like to get done before releasing 2.1 (#46 really shouldn't be too hard, I just haven't gotten around to it, and it lays the groundwork for #61 and thus being able to implement other protocols). And since we're tying Cardinal to Twisted pretty heavily with this change (which I'm still a little uneasy about because as I've previously said, I was kinda going the other direction), it might cause me to re-examine the method in which I was going to implement secondary protocols.

For example, theoretically a Slack protocol driver could be implemented as a Twisted protocol. But it also means I am set on Cardinal being a Twisted application, through-and-through which means other decisions:

1. Should I write a TAP (Twisted Application) file?
2. Should I try to implement Twisted logging (I'd prefer to stick with the built-in logging, but they seem to really want me to use it!)
3. What's the best method for unit testing the Twisted components (they have a stdlib unittest replacement library, but I'm not sure it's compatible with py.test)

I'm not terribly fond of all the pieces of stdlib Twisted wants you to forego.

The alternative option (yes, even after we got all this working), is to focus instead on #46 so we can implement a different IRC engine, and get Python 3 support, and implement async through await / async. I think that's eventually where I'd like to get anyway (Py3 support for sure.) I've been reading up on Twisted and it seems like they are toying with the idea (after Py3 port is done for real) of replacing reactor with async / await.

[s-ol]

I never looked into async/await, but python3 definetely sounds good aswell. I think using twisted for unit tests is pretty much a requirement when the application is an asynchronous twisted application, at least some guide I read left that impression.

I never liked these standard-library pressures of bigger frameworks either, I'll have a look at async and await and see how much twisted is really doing for us at the moment.

[johnmaguire]

@S0lll0s The one other option we might have as a stop-gap is to implement threading (multiprocessing?) instead of async. It might be a little more complex to implement, but is probably also a bigger win overall.

[johnmaguire]

If each thread can run its own reactor loop (I think it should even?) then the web UI implementation could still be done with Twisted, it'd just have to run the loop internally somehow.

[johnmaguire]

By the way, do you idle on any IRC networks? It might be nice to have that as a method for discussing some of these changes. I did setup #cardinal on irc.darkscience.net, but am not opposed to adding another server to my IRC client. I also have some questions about themer. ;) Curious if I can implement awesome-wm in it.

[s-ol]

Sure, shoot me a message on freenode or OFTC (nick like on gh). Probably makes more sense for the small details.

[s-ol]

Oh and I'm doing my testing in #checkit on Freenode

[johnmaguire]

This is the message I got with the old log_failure command:

2015-12-30 18:31:41,964 - cardinal.plugins - ERROR - Could not load plugin module: notes
Traceback (most recent call last):
  File "/home/jmaguire/repos/Cardinal/cardinal/plugins.py", line 359, in errback
    failure.raiseException()
  File "<string>", line 2, in raiseException
  File "/home/jmaguire/repos/Cardinal/plugins/notes/plugin.py", line 8
    asdf;!
         ^
SyntaxError: invalid syntax

Unfortunately failure.raiseException() kind of obliterates the stack it seems.

[johnmaguire]

It's too bad exception chaining isn't available untili Py3.

[johnmaguire]

Closing this in lieu of #87 again. I think we'll probably end up using pplugins instead to do concurrency instead of just asynchronous requests. If we don't, #87 is the most up-to-date currently.