add support for constraint handlers

[rschwarz]

Adresses part of #94.

[rschwarz]

Very early sketches, still.

First idea is to have an AbstractConstraintHandler for which user define a subtype, which will also play the role of SCIP_CONSHDLRDATA.

I wanted to define methods (with nice Julia signatures) for the callbacks. But it's not quite clear yet how I will get a @cfunction pointer of these (after dispatch!). So, it might be better to use a setcallback!(model, fun) approach, like in MPB.

[mlubin]

but it's not quite clear yet how I will get a @cfunction pointer of these (after dispatch!).

You don't. You provide a top-level function that redirects to the user methods. See, e.g., https://github.com/JuliaOpt/Ipopt.jl/blob/7aa1943f263a9b5ea29cf5e66c85c1c6e3aede49/src/Ipopt.jl#L195.

[rschwarz]

You provide a top-level function that redirects to the user methods.

Yeah, we did something similar in the old SCIP.jl (lazycb_wrapper).

In the IPOPT example, the user-defined Julia function is stored in the prob type, and then called as prob.userfun(...). What I had imagined was that the user provides the methods that are then selected via dispatch, as in userfun(prob, ...), using the specific subtype of AbstractConstraintHandler. But that approach might not actually have any benefits and might lead to errors with incomplete implementation of the callback interface.

[rschwarz]

I've also wondered whether to use MOI.set and get for access to the callbacks. But since this is SCIP specific, there is probably no benefit in that either.

[codecov]

Codecov Report

Merging #109 into master will increase coverage by 1.09%.
The diff coverage is 84.84%.

@@            Coverage Diff             @@
##           master     #109      +/-   ##
==========================================
+ Coverage   16.63%   17.72%   +1.09%     
==========================================
  Files         135      138       +3     
  Lines        4311     4377      +66     
==========================================
+ Hits          717      776      +59     
- Misses       3594     3601       +7

Impacted Files	Coverage Δ
src/SCIP.jl	100% <ø> (ø)	⬆️
src/MOI_wrapper/variable.jl	83.62% <0%> (-2.23%)	⬇️
src/MOI_wrapper/conshdlr.jl	100% <100%> (ø)
src/convenience.jl	100% <100%> (ø)
src/managed_scip.jl	94.28% <100%> (-1.37%)	⬇️
src/MOI_wrapper.jl	89.06% <100%> (-1.42%)	⬇️
src/conshdlr.jl	84% <84%> (ø)
src/MOI_wrapper/results.jl	97.14% <85.71%> (-2.86%)	⬇️
... and 7 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 670d761...b575101. Read the comment docs.

[rschwarz]

Have a bare-bones implementation, with a minimal (passing) test of a dummy constraint handler that does nothing (but the callbacks are being called).

Next step would be a more involved test/example that involves MOI, to see what kind of utility methods will be needed to make the mappings between SCIP_CONS* pointers and the related SCIP_CONSDATA* pointers, as well as conversions between C pointers and Julia objects.

I'm already worrying about using Julia objects (that can be GCed) as SCIP_CONSHDLRDATA and SCIP_CONSDATA and have the user be responsible to keeping them alive...

[mlubin]

s/is satisfies/satisfies/ (below as well)

[mlubin]

s/is violates/violates/

[mlubin]

I'm already worrying about using Julia objects (that can be GCed) as SCIP_CONSHDLRDATA and SCIP_CONSDATA and have the user be responsible to keeping them alive...

Do you mean for the pure SCIP interface or for MOI? For MOI, could you create (and then free) these during the optimize! call?

[rschwarz]

Do you mean for the pure SCIP interface or for MOI? For MOI, could you create (and then free) these during the optimize! call?

I feel that it can affect both. I imagine that these objects are really created by the (MOI) user, just like you would create a function, set pair when you add a constraint. I guess that we should keep a copy/reference to it inside SCIP.Optimizer, to be sure.

A related issue is that the callback (on the SCIP/C side) receive a SCIP*. So, semantically, it would make sense that the SCIP.jl and MOI users would receive a ManagedSCIP and SCIP.Optimizer, respectively. I'm not sure whether these can be injected automatically, or we should tell users to store these references in their ConstraintHandler object.

By the way, I appreciate that you read the updated code, but I don't think it's ready for review, yet. I have not figured out what level of detail to expose to the SCIP.jl users and the variable naming is all over the place. For instance, I would like to make it clear whether a an argument is of type Ptr{SCIP_CONS}, Ptr{SCIP_CONSDATA}, or the corresponding user object (e.g. DummyConstraint).

I already started working on the example of a lazy linear constraint handler and it's not so obvious how to give the user access to the solution values. A Dict{MOI.VI, Float64}? Or a closure where they can fetch values for individual MOI.VIs? I guess I will not develop any further convenience on the pure SCIP or ManagedSCIP side and focus on MOI directly.

[rschwarz]

OK, I have now done everything that I had planned to do, so I consider it ready for review.

Before merging it, I would like to test it with a real-word usage case (PipeLayout.jl) first, which will take some time.

[rschwarz]

I was pondering on changing the MOI-facing function include_conshdlr to use MOI.submit.
This would probably entail using another struct (a Submittable) that holds the actual AbstractConstraintHandler in addition to the keyword arguments of the current functions.

But it's not clear what the benefits are, for the user. Would it be easier to submit the plugin from JuMP? Also, it might be confusing, given that the discussion in jump-dev/MathOptInterface.jl#782 points to using SolverAttributes for callbacks instead, for most other MIP solvers out there.

[rschwarz]

So, this works well enough for my use-case (treetopohdlr.jl, semisubhdlr.jl, cbtopo.jl) that I would like to merge & release at this point, if there are no objections or questions.

[mlubin]

Seems reasonable to merge this and look at proper support from JuMP in a separate PR.

[rschwarz]

look at proper support from JuMP in a separate PR

You mean in a solver-independent way?

I find the way I create constraints from callbacks nice enough, based thinly on JuMP.@build_constraint.

[mlubin]

@rschwarz, just in a documented way that's usable without PipeLayout.jl, not solver-independent :)

[rschwarz]

just in a documented way that's usable without PipeLayout.jl

Good point. Maybe this calls for a tested example, as another JuMP-dependent test stage just like MINLPTests.

[matbesancon]

Maybe this calls for a tested example, as another JuMP-dependent test stage just like MINLPTests.

Or even in an example folder called from the normal travis build (if small enough that should be fine with build time)

[rschwarz]

Or even in an example folder called from the normal travis build (if small enough that should be fine with build time)

But then we have the problem again that we can not upgrade SCIP to a new version of MOI as long as a compatible JuMP is not released.