[DRAFT] - Test using re2 regex pkg in place of std regex

[ahrav]

Some benchmarks testing go std lib regex, vs re2 vs oniguruma.

re2 - https://github.com/wasilibs/go-re2
oniguruma - https://github.com/go-enry/go-oniguruma

old.txt -> go std lib
new.txt -> oniguruma
newest.txt -> re2 (too brain dead to name it better lol)

go vs oniguruma

go vs re2

oniguruma vs re2

[zricethezav]

@ahrav have you seen https://github.com/go-enry/go-oniguruma? I played around with introducing this into gitleaks once upon a time and the results were very impressive. This was before keywords were introduced into the detecting algorithm so the results were more dramatic. Still, go-onigurama might be worth investigating as it has better performance than re2 in most cases and can be dropped in iirc.

[ahrav]

Ohhhh, wow. I did not see this at all. Yea I can definitely give this a spin. Thanks for pointing it out.

[ahrav]

@zricethezav added some benchmarks after i did some testing w/ oniguruma and re2.

[zricethezav]

@zricethezav added some benchmarks after i did some testing w/ oniguruma and re2.

@ahrav Based on what I'm seeing looks like re2 wins out right? That's good news since onigurama would require some cgo flags if im not mistaken

[ahrav]

@zricethezav added some benchmarks after i did some testing w/ oniguruma and re2.

@ahrav Based on what I'm seeing looks like re2 wins out right? That's good news since onigurama would require some cgo flags if im not mistaken

Yep that's right. You are correct I had to set the CGO flags when testing oniguruma as well which wasn't great. The re2 change is a pretty nice and easy drop in and replace.

[dustin-decker]

Since it's API compatible w/ go's regexp I think we could just consider doing a go mod replace directive.

[ahrav]

Since it's API compatible w/ go's regexp I think we could just consider doing a go mod replace directive.

I don't think you can go mod replace a std lib pkg. I could be wrong though.

[ahrav]

I do wonder with this more efficient regex engine, could we explore a larger chunk size?

[dustin-decker]

Closing because #2324