Support for shell completion

[andreypopp]

This PR adds support for shell completion (as requested by #1).

Opening it early and as a draft to ask for feedback on the approach.

How it works, the gist

We define a shell completion protocol between a cmdliner-based program
and a shell.

On the shell side we expect to have a completion function which:

1. invokes a cmdliner-based program with a special completion token injected
   into the argv line

2. interprets the output of the program completion and presents it to the user

We add a helper cmdliner tool (to be distributed with the library) which is
used to generate shell completion scripts for a cmdliner-based program for
either bash or zsh (as requested).

How to test

In this PR I've configured dune to build example programs as executables, it's
convenient to have them on $PATH for testing, we also want cmdliner tool to
be available:

export PATH="${PWD}/_build/default/example:$PATH"
export PATH="${PWD}/_build/install/default/bin:$PATH"

First we need to install completion handlers for our programs. Let's do that
for a few of example programs:

eval "$(cmdliner completion-script --shell zsh tail_ex.exe)"
eval "$(cmdliner completion-script --shell zsh cp_ex.exe)"
eval "$(cmdliner completion-script --shell zsh darcs_ex.exe)"

The supported values for --shell are bash and zsh as for now.

Now I suggest to try the following completions (I'll paste the output I get as
well with <TAB> token showing the place where I've requested completions).

Completion for top level options and subcommands:

; darcs_ex.exe <TAB>
--debug        -- Give only debug output.
--prehook      -- Specify command to run before this $(mname) command.
--quiet    -q  -- Suppress informational output.
--verbose  -v  -- Give verbose output.
initialize  -- make the current directory a repository
record      -- create a patch from unrecorded changes
help        -- display help about darcs and darcs commands

Completion for options of a subcommand:

; darcs_ex.exe initialize <TAB>
--debug        -- Give only debug output.
--prehook      -- Specify command to run before this $(mname) command.
--quiet    -q  -- Suppress informational output.
--repodir      -- Run the program in repository directory $(docv).
--verbose  -v  -- Give verbose output.

Completion of an option name:

; darcs_ex.exe --<TAB>
--debug    -- Give only debug output.
--quiet    -- Suppress informational output.
--verbose  -- Give verbose output.
--prehook  -- Specify command to run before this $(mname) command.

Completion of an option value of enum type (see tail_ex.ml):

; tail_ex.exe --follow <TAB>
name        -- name
descriptor  -- descriptor

Completion of an argument value and option names:

; cp_ex.exe <TAB>
--force      -f      -- If a destination file cannot be opened, remove it and try again.
--recursive  -R  -r  -- Copy directories recursively.
--verbose    -v      -- Print file names as they are copied.
B0.ml           _build/         cmdliner.opam   dune            LICENSE.md      pkg/            _tags
bin/            build.ml*       completion/     dune-project    Makefile        README.md       test/
BRZO            CHANGES.md      doc/            example/        _opam/          src/

Completion of an argument value (configured to filename completion):

; cp_ex.exe ./<TAB>
B0.ml          build.ml*      completion/    dune-project   Makefile       README.md      test/
BRZO           CHANGES.md     doc/           example/       _opam/         src/
_build/        cmdliner.opam  dune           LICENSE.md     pkg/           _tags

Completion of only option names:

; cp_ex.exe -<TAB>
--force      -f      -- If a destination file cannot be opened, remove it and try again.
--recursive  -R  -r  -- Copy directories recursively.
--verbose    -v      -- Print file names as they are copied.

TODO

• complete subcommands
  • complete parent command options in subcommand position
• complete option names
  • do not suggest completions for command line flags (they have no value)
• allow to specify completions for option values
  • automatically configure completions for enums
• complete values for positional arguments
• support zsh shell
  • separate groups for subcommands vs options
  • fix sort order
• support bash shell
• support fish shell
• figure out how to write tests
• get rid of custom dune setup (as I understand dune is not the main build system in the project)
• decide what to do with eval_peek_opts
• process doc directives when emitting completions
• do not suggets option names in position after --

[dbuenzli]

Thanks for getting your hands dirty with this @andreypopp. It is extremely appreciated.

I won't have the time to look at this closely for some time. But since you seem to be in the mood, early feedback:

1. I'd like to avoid hardcoding shell support in executables (and hence the library) itself. This doesn't really scale. I'd rather have a well designed generic completion mecanism for cmdliner that is drivable by the various shell completion insanities. Then we can devise hopefully generic completion scripts for the myriad of shells that we maintain on the side (versus embedding the support in executables). We can add a cmdliner tool distributed with the library if needed to help with installing completions given a shell and an executable name.

2. I'd like to avoid driving anything via environment variables. Options that start with --cmdliner are reserved I guess this should be enough to do the job. I suspect you can look into the short-circuiting behaviour of --help for inspiration.

3. There was also the idea of dumping all the cli metadata in machine readable form via --help=json. While this wouldn't cater for dynamic completions it's also perhaps something to keep in mind Export options information #186

4. Be careful, we need to support 4.08. I saw a String.starts_with in passing which is 4.13, you can add anything that may be missing in Cmdliner_base

[andreypopp]

Thanks for the feedback!

I've pushed few commits to address some of the feedback items and updated the
PR description accordingly.

More specifically:

1. I'd like to avoid hardcoding shell support in executables (and hence the
   library) itself. This doesn't really scale. I'd rather have a well
   designed generic completion mecanism for cmdliner that is drivable by the
   various shell completion insanities. Then we can devise hopefully generic
   completion scripts for the myriad of shells that we maintain on the side
   (versus embedding the support in executables). We can add a cmdliner tool
   distributed with the library if needed to help with installing completions
   given a shell and an executable name.

I've added a cmdliner helper tool that either can be used as

eval "$(cmdliner completion-script --shell zsh tail_ex.exe)"

or to generate completion scripts to be distributed with cmdliner based programs if needed.

2. I'd like to avoid driving anything via environment variables. Options that
   start with --cmdliner are reserved I guess this should be enough to do the
   job. I suspect you can look into the short-circuiting behaviour of --help
   for inspiration.

No environment variables are needed now. But the completion mode is activated
with +cmdliner-complete:PREFIX token now. Maybe we can treat this as reserved
as well? Activating with --cmdliner is possible too but will probably require
some more code to treat this option name specially.

3. There was also the idea of dumping all the cli metadata in machine readable
   form via --help=json. While this wouldn't cater for dynamic completions it's
   also perhaps something to keep in mind Export options information Export options information #186

Noted. But I think this is a separate feature to be honest. For a good
completion UX context sensitive dynamic completion is needed.

4. Be careful, we need to support 4.08. I saw a String.starts_with in passing
   which is 4.13, you can add anything that may be missing in Cmdliner_base

Done.

[dbuenzli]

We will need a proper type name for that. Also I don't think this belongs to info it belongs to the 'a Arg.conv type. E.g. if I use Arg.file converter I shouldn't have to repeat `Complete_file, (same for Arg.enum).

Currently this type is not abstract but it has been scheduled to become since 2017. So probably it's now time to act and make 'a conv into a proper abstract record.

Does that still fit your approach ?

[andreypopp]

We will need a proper type name for that. Also I don't think this belongs to
info it belongs to the 'a Arg.conv type. E.g. if I use Arg.file converter I
shouldn't have to repeat Complete_file, (same for Arg.enum).
Currently this type is not abstract but it has been scheduled to become since
2017. So probably it's now time to act and make 'a conv into a proper
abstract record.

I've made the _ conv type abstract.

Then moved the completion specification to a converter:

• Had to change term type to keep a map from args to completion specs instead of
  a set, so I can retrieve the completion spec later. I've kept the Arg.Set name though...

• Instead of exposing completion spec as a separate type, for now, I've opted
  out to add few arguments to conv-family of functions. I think it's simpler
  that way. Let me know what you think.

Then there's another commit which changes the order we process pos args
when parsing a command line: previously it was right-to-left, now it's
left-to-right. This shouldn't affect actual evaluation, as I understand (as
each pos arg looks only at its own range), but affects completion as the
completion token is now attached to the left most pos arg in "ambiguous" cases
like in cp_ex.ml SRC... DST.

[andreypopp]

There's one other idea, I think, worth exploring. to make the completion
mechanism more useful - an ability to eval terms in a completion context.

That would allow to support cases where completion might depend on some
configuration, for example a database connection string:

migrate --db "postgres://localhost:5432/mydb" --migration <TAB>

The code would look like:

let db_arg =
  Arg.(value & opt (some string) None & info ["db"])

let migration_arg =
  let complete prefix ctx =
    match Cmdliner.Term.try_eval_for_completion ctx db_arg with
    | None -> (* cannot eval db_arg *) []
    | Some db -> Db.get_unapplied_migrations db
  in
  Arg.(value & opt (some string) None & info ~complete ["migration"])