Finer control of solve and sample args via turing_inference args

[saumil-sh]

Checklist

• Appropriate tests were added
• Any code changes were done in a way that does not break public API
• All documentation related to code changes was updated
• The new code follows the
  contributor guidelines, in particular the SciML Style Guide and
  COLPRAC.
• Any new documentation only uses public API

Additional context

The turing_inference function interacts with solve and sample; however, the turing_inference keyword arguments are supplied only to the differential equation solver through solve. There is no way to fine-tune the hyper-parameters of the sampling algorithm. The only way is to modify the source. This may present beginners with a steeper learning curve. With this pull request, I change the turing_inference arguments to explicitly take solve_kwargs, sample_args, and sample_kwargs. Defaults are set for anyone wanting off-the-shelf inference, and power users can modify everything to their liking. This change comes at a cost: divergence from the stan_inference argument structure and a minimal breaking change for turing_inference. It could be an excellent time to add this now with MTK v9 when everyone updates their code.

[Vaibhavdixit02]

This is great @saumil-sh! I don't really understand why the stan_inference cannot also be made to move to this though, can you say more about that?

[Vaibhavdixit02]

Also if we are going to do a breaking release might be worth to consider if we want the SciML interface implemented here

[saumil-sh]

Boy, I am glad to hear that this doesn't make sense only in my head! That is correct; there is no reason why stan_inference cannot be moved to a similar argument structure.

What is your idea of a SciML interface here? like one inference function which interacts with turing, stan, etc.?

This discussion is fantastic, actually; a common interface could facilitate using stan and turing side-by-side, comparing results, and benchmarking.

[Vaibhavdixit02]

That is correct; there is no reason why stan_inference cannot be moved to a similar argument structure.

Yeah so we should update that as well, if you don't mind putting it in this PR!

What is your idea of a SciML interface here? like one inference function which interacts with turing, stan, etc.?

Yes, but more inline with the interface of other packages - we'd have structs TuringInference, StanInference etc and define solve(::ODEProblem, ::TuringInference, ...)

[saumil-sh]

Okay, I don't have stan installed at the moment. I will try to modify stan_inference relying on CI/CD checks for tests and report back. 😄

EDIT 1: Installing CmdStan 🚀

[saumil-sh]

Update: Which CmdStan versions are supported by DiffEqBayes/jl? production version didn't pass tests on my local setup.
It was because my CmdStan version was newer than what DiffEqBayes.jl currently supports, but the documentation doesn't mention this, perhaps related.
I found the updated syntax at DiffEqBayesStan.jl. Should the former be a submodule for this repo? How does this work since it is a Julia package?

EDIT 1: The CI/CD tests are run using CmdStan v2.29; therefore, that is the most recent supported version. Locally, I am running v2.34, and with updated syntax, the tests passed.

[Vaibhavdixit02]

Feel free to bump the CmdStand being used on CI in that case. Also now the julia package that serves as the interface to cmdstan is StanSample.jl not CmdStan.jl so just take a look at that as well

DifEqBayesStan was moved into a separate repository because Rob wanted to do some testing and benchmarking on his own. I had mostly kept up with the updates here so the syntax shouldn't need to be updated much?

[saumil-sh]

I copied over some lines from DiffEqBayesStan.jl for the recent version of CmdStan to work. Ideally, it is nice not to do it manually, so I suggested submodules/subpackages.

I have done everything I can here on this PR. I wonder why documentation and format checks are failing. I ran format checks from my VS code, which uses JuliaFormatter.jl 🤷

[Vaibhavdixit02]

This fell out of my attention, thanks for updating!