Z-expansion reweight

[mroda88]

An Issue has been reported by Kirk from NOvA during the forum (March 2020). They discovered that the z-expansion re-weight knobs which are supposed to take into account the correlations between the 4 parameters of the standard z-expansion are not taking into account the correlation at all.

[candreop]

This sounds like a misunderstanding of what the knobs are supposed to be doing. They are just the corresponding model params - they are not meant to be doing anything other than allowing the user to tweak the expansion. The user can use the corresponding covariance matrix and vary them in a way that respects the correlation and/or calculate penalty terms that include the correlation.

[kirkbays]

If I can push back on that a bit - it's difficult because there's also no accompanying documentation. Other knobs in GENIE reweight are 'ready to use', so to speak. These Z-expansion knobs, however, though provided along side all the other knobs that are meant to be used as is, should NOT be used as is. They are only really valid when properly correlated using a matrix that is not included, and that fact is not documented anywhere, nor are there any warnings. It's a recipe for disaster for the user.

NOvA came extremely close to using these incorrect uncertainties for our main analysis. Either the diagonalized, properly correlated knobs should be the ones provided, or the correlation matrix should somehow be provided, without which these knobs are effectively useless. At the least there should be documentation with large red warnings about what these are and aren't.

[kirkbays]

We're also unsure where the a0 uncertainty, which doesn't seem to be mentioned in the paper, comes from.

[candreop]

Hi Kirk - I am afraid I have to reject the comment that there is no documentation: This an implementation of the z-expansion model by the original authors. The covariance matrix for the z-expansion parameters is derived by the authors of this work and, I believe, it is given in arxiv:1603.03048.
I also warn against the comment that “other parameters can be used as is”. Not sure what “as is” means. They certainly have correlations between them, same as the z expansion parameters! We now extract some these correlations from our global fits to neutrino scattering data.

[kirkbays]

I mean GENIE documentation. How is it supposed to be clear to the user that these are the non-correlated uncertainty parameters so they even know they have more work to do before they can use them? Z expansion is basically never mentioned or described in the manual. How is the user supposed to even know that paper you refer to exists? And then if they do find it, how are they supposed to correlate the parameters themselves, when the software tools to do so don't easily exist inside GENIE as offered? I'm worried that if you don't realize there is an issue here, other experiments will fall prey to the same trap NOvA almost fell into

[candreop]

Re: “How is the user supposed to even know that paper you refer to exists?“ Didn’t you know you were using the z-expansion? Using GENIE is not a substitute for reading the relevant literature. Using a complex simulation and toolkit, such as GENIE, as a black box is discouraged. More documentation is always better, of course. We’re in the process of writing papers on the recent tunes and GENIE3, but this (same as your work) proceeds as fast as our resources allow. We also run a user forum that provides an opportunity to receive feedback: You did ask and we did provide feedback. Perhaps, if NOvA does not have the expertise within the collaboration to make correct use of the tools, you can ask for our feedback a bit earlier. We also provide example
code showing how to use those z-expansion parameters along with a correlation matrix, but I guess this was not seen either... I have also (on many occasions - see the front GENIE web page, if in doubt) emphasised in all ways possible that GENIE/ReWeight, as it stands currently, is just a collection of simple weight calculators and it is NOT a complete and consistent error evaluation for any of our tunes. This is something that will come with GENIE4 (see our published release schedule) when we have the ability to integrate ReWeight with Professor and release our computed response functions for non reweightable parameters. Currently, as is well explained to our users, while we work to upgrade ReWeight, not even our own global analysis results are implemented/integrated within ReWeight. So, on the face of these limitations, thinking that some third party partial tuning results from 2016 are magically integrated into ReWeight, is an error of judgement. We extended to NOvA a call to contribute to the upgrade of the ReWeight (especially since you use the currently non-reweightable hN) but, to date, there was no offer to help. Perhaps if you want to see faster ReWeight upgrades you can consider contributing.

[kirkbays]

Yes, we knew we were using the Z-expansion, what wasn't clear is that the provided knobs were the raw a1-a4 terms and not the already correlated 'eigen'-systs, which would be the more useful set of four knobs. As for expertise, we consulted both Jeremy and Hugh about this, who determined the correlation matrix was not included and there was no way to easily use it. Could you please point me towards the example code? I appreciate you taking the time to respond here.

[candreop]

Sorry @kirkbays, I missed the follow up comment (I somehow thought this issue was submitted in the Generator repository and not here) and my reply comes too late. There is some example code in ReWeight Apps (https://github.com/GENIE-MC/Reweight/blob/master/src/Apps/gRwghtNCorrelatedParams.cxx). Basically, this code shows how to do a Cholesky decomposition and move between a correlated/uncorrelated parameter basis. This is an example written for z expansion, in particular, by Aaron who coded up the z expansion model. If you want to generate parameters throws, the Cholesky decomposition of the covariance matrix can be used in a method that gives you systematic parameters that have the appropriate covariance. If you just use the the z expansion parameters in a fit, then you can let the fitter move them any way it likes, but you need to include the covariance in the penalty term. It was not possible for us to include the covariances in this version of the code: To do that we need it to evolve from a set of simple weight calculators, to a tool that can support different tunes (making appropriate choices for each tune / as the input covariances can be different for different tunes).