Testing against actual VCS binaries

[chintal]

Is it possible to modify the test environment to also include the VCS binaries?

If the binaries can be guaranteed to exist, a small handful of test repositories can be hosted somewhere and the tests can check the code that actually interacts with the VCSs.

[reinout]

At least svn, mercurial and git are available, I'm pretty sure bzr is too.

We don't need to actually have a remote host, for testing purposes local folders are OK. At least, that's how zest.releaser tests it. The tests for zest.releaser have a test setup that sets up those local folders.

Works quite well, but we keep having to compensate for small changes in the commands' output.

[chintal]

See PR #17

[reinout]

I think copying the test setup from zest.releaser is, I think, easier. Most of the setup then happens outside of the test in python code instead of making the doctest harder to read.

Additional comment: it is totally fine (and perhaps even better) to write regular tests instead of doctests for this. Might make it easier to do.

[chintal]

Perhaps so. For the moment it's stable enough.

Later on the setup like bits can be moved out of the doctests and into the setup python code, perhaps, when it becomes clear which parts are 'common' to it all. As it is, it seems there are a few minor tweaks each VCS needs, so doing it separately for each is probably necessary.

Ultimately, we'll have to see which approach would be easier to maintain.

[reinout]

Having python code in a test setup function is preferable to having a doctest that is twice as long.

"Later on it can be moved" isn't really a very good reason if it has no place to be there anyway. Test setup belongs as much as possible in a test setup method, not in the actual documentation.

For zest.releaser I'm doing the test setup out of the doctest, so I'm 90% sure I'm going to complain if it is done differently in checkoutmanager.

I'm not trying to discourage you, but I am worried a bit :-)

But, I just re-looked at your now-closed pull request and there I see about 25% test setup and 75% actual tests in a bzr doctest. Actually, that's not bad at all. And the setup is pretty clear and fits into the narrative.

Well: if technically possible, I'd prefer the test setup to happen in test setup code.