Formulate using Lark

[aryan26roy]

Hi @chrisburr @eduardo-rodrigues @jonas-eschle and @henryiii ,

Me and @jpivarski are working on a new version of Formulate on the main branch which has the following features:

• Uses Lark to parse TTree formulas
• Out-of-the-box compatible with Uproot
• Backwards incompatible with the current interface of Formulate

Our main goal is to be able to easily translate formulas to and from ROOT TTree, Numexpr and Uproot. GIven that the infrastructure for converting to and from Numexpr is not in place yet, if this branch is made the sole branch right now, we would loose that capability temporarily.

The reason for starting this discussion is to ask for any opinions on this and the possibility of making this the sole implementation for this library. Our main concerns are with the interface and how, if at all, making major API changes may have any downstream effects.

[eduardo-rodrigues]

Hello @aryan26roy and @jpivarski. I'm not the author of this package but from my perspective the package has effectively been dead since several years, with no real developments. As the main functionality is meant to be what you list, I would personally go ahead 👍. My 2 cents.

[aryan26roy]

Thanks!

[henryiii]

I'd say go ahead! I do plan to integrate it in the mid future to uproot-browser, and would love to see it get updated first. :)

[aryan26roy]

Makes sense to me. Thanks!

[jpivarski]

The formulate package on PyPI is owned by @chrisburr and the scikit-hep account. One of us will probably need to get access to that to be able to push a new release.

I still don't have access to the scikit-hep account since 2-factor authentication was added. (It has been on my to-do list for a while, and Chris and I never managed to have a one-on-one meeting to share the QR code.)

[jpivarski]

@jonas-eschle, does zfit depend on the formulate interface? What requirements do you have on it?

[jonas-eschle]

No, not at all, zfit is independent

[jonas-eschle]

Thanks for bringing this up! I've been digging into formulate a while ago to resurect it, but it needs indeed a major overhaul and incompatible changes that I started, yet I never implemented these as they grew too big (only minor changes).

Very welcome from my side to perform these changes, fully agree on the plan and I am also available to help with some stuff (reviews, design, code,...).

One of my plans was to use Sympy instead of numexpr, so effectively a ROOT <=> Sympy converter, since Sympy can convert to and from numexpr and more (but Sympy handles a few things, like the equal, a bit different).

The advantage would be that simplifications could be easily made (i.e. remove unnecessary braces), other backends can be supported (whatever sympy has) and that a latex representation would be readily available.
I tried this road, but as said, I got stuck with too many changes required to make formulate run and pass tests again.

So anything is fine with me, please go ahead with all the incompatible changes and see as it fits.

[aryan26roy]

Hello everyone,

I am happy to announce that the main branch as of today, is a subset (functionality wise) of the master branch built completely from scratch using Lark. The Python API remains the same except for the Expression object and the CLI.

Currently there are a bunch of unit tests that aim to pin down the behavior of the new formulate when it comes to parsing numexpr and TTreeFormula. The new formulate additionally adds support for TTreeFormula special functions ( The ones that end in dollar sign in this documentation). A few of the special functions could not implemented because they are not "functions" as we commonly think of them in the context of functional programming (What they do depends on what expression they are embedded within).

If there are no objections, we will switch the default branch of this repo to main and release the new formulate as v1.0.

Let us know if it's all okay.

Signed,
Aryan & Jim

[jonas-eschle]

Hey both, thanks a lot for the work, it looks great, and congratulations, the main purpose of speed is definitely achieved! Overall, no strong hold-backs from my side. Just to clarify a few things

• the readme and docs is still empty, while we can copy (most) of it from the main branch, at least the README and some primitive docs would be good to have filled?
• on the features: operations between AST objects are a non-goal or are they just not implemented yet? (It was possible before to use operations on them which simplifide the construction somewhat, but was also a necessity, as it was so slow. IMHO, it's not strictly needed, but nice to have. So any way is fine)

If it's the main branch, I'll gladly add some updates for the CI, pre-commit and stuff, and can extend the test-suite (I tried before to do some hypothesis tests using the invariant of transformations back and forth, can add that)

[jpivarski]

The parts that weren't reimplemented from old-formulate to new-formulate are negotiable non-goals. Do we know of a use-case for manipulating AST objects or a CLI?

For my part, I'm interested in this so that TTreeFormulas can be a second language in Uproot, and eventually the default one. This would affect the expressions and aliases of array-reading functions like uproot.TTree.arrays. In particular, aliases should have always been in TTreeFormula syntax, since aliases shipped with the ROOT file (TTree::fAliases) would be in TTreeFormula syntax, not Python.

This was an Uproot 4.0.0 goal, almost 4 years ago now. This language="py" default was supposed to be temporary.

[jonas-eschle]

The use-case (for me) so far was that I could concatenate conditions, i.e. ast1 & ast2, now I would need to do formulate.from_numexpr(ast1.to_numexpr() + " & " + ast2.to_numexpr()). That's not an impossibility, but the former is definitely neater and allows for a uniform handling. Required, no, but not hard to implement either I assume, as it's a one-off thing? (it could even be automatically overloaded by automating the above step (i.e. overload all necessary dunder methods dynamically).

I can see the uproot use-case, and it makes a lot of sense there, should definitely be the "main use-case" (my use-case with concatenating conditions is for reading files usually), and I am glad that this can finally go into it.

[jonas-eschle]

@aryan26roy @jpivarski I've found another use-case that seems missing (also note that the delivery failed again, see fix: #47): the variables attribute. One crucial thing for me in formulate was that it could extract the variables used, i.e. to load it from the file (and I presume this is no different in uproots case?)

[jpivarski]

Uproot might need that as well. I haven't looked into it yet, but we'll probably need to get a list of variables to know what to read from the file.