feat(python,rust): add date pattern dd.mm.YYYY

[Julian-J-S]

close #14990

• add dd.mm.YYYY date pattern which is widely used.
• enables automatic date inference for to_date as well as in read_csv
• also added tests for all date patterns

pl.DataFrame({"D.M.Y": ["01.02.2020", "03.04.2020", "31.12.2020"]}).with_columns(
    pl.all().str.to_date()
)

shape: (3, 1)
┌────────────┐
│ D.M.Y      │
│ ---        │
│ date       │
╞════════════╡
│ 2020-02-01 │
│ 2020-04-03 │
│ 2020-12-31 │
└────────────┘


pl.read_csv(
    source="dates\n01.02.2020\n03.04.2020\n31.12.2020".encode(),
    try_parse_dates=True,
)

shape: (3, 1)
┌────────────┐
│ dates      │
│ ---        │
│ date       │
╞════════════╡
│ 2020-02-01 │
│ 2020-04-03 │
│ 2020-12-31 │
└────────────┘

[codspeed-hq]

CodSpeed Performance Report

Merging #16045 will improve performances by 27.87%

_{Comparing JulianCologne:feat-add-D-M-Y-dot-sepatared-date-pattern-inference (09a776b) with main (6730a72)}

Summary

⚡ 1 improvements
✅ 34 untouched benchmarks

Benchmarks breakdown

	Benchmark	main	JulianCologne:feat-add-D-M-Y-dot-sepatared-date-pattern-inference	Change
⚡	test_filter2	2.8 ms	2.2 ms	+27.87%

[alexander-beedie]

I think the patterns are assessed in priority order, and I doubt this is the most common form (I'd expect that to be the one separated by "-") 🤔 Can you check? (I'm on mobile at the moment, so can't confirm).

[MarcoGorelli]

thanks for your PR

if we go ahead with this, it should also be added to DATETIME_D_M_Y too (DATE_D_M_Y should be a subset of it)

[Julian-J-S]

I think the patterns are assessed in priority order, and I doubt this is the most common form (I'd expect that to be the one separated by "-") 🤔 Can you check? (I'm on mobile at the moment, so can't confirm).

@alexander-beedie
Afaik the order does NOT matter as is checks for all formats all the time.
Was thinking about the order anyway.
However, I did check the occurrences (#15949 (comment)) and the order by countries actually is:

• dd/mm/yyyy: 36x
• dd.mm.yyyy: 26x
• dd-mm-yyyy: 7x (least used)

if we go ahead with this, it should also be added to DATETIME_D_M_Y too (DATE_D_M_Y should be a subset of it)

@MarcoGorelli
Was thinking about this, too.
However, it looks like while dd.mm.YYYY is very common as date in many countries there does not seem to be an equivalent for the datetime. This is also my experience, as I have never seen a datetime using this date format. 🤔

[MarcoGorelli]

Afaik the order does NOT matter as is checks for all formats all the time.

IIRC it goes in order until it finds a match, then it keeps using the last successful one (until it may need to look at the next one in the group). maybe let's put them in order of popularity then?

However, it looks like while dd.mm.YYYY is very common as date in many countries there does not seem to be an equivalent for the datetime. This is also my experience, as I have never seen a datetime using this date format.

ok sure

[Julian-J-S]

maybe let's put them in order of popularity then?

sure! But how do we measure this?
I would suggest

• dd-mm-yyyy: ISO 8601 international standard; should be No 1
• dd/mm/yyyy: more countries than below according to wiki
• dd.mm.yyyy:

No sure what the performance impact really is but if we do this "micro optimizations" should we also adjust the DATE_Y_M_D, DATETIME_D_M_Y and DATETIME_Y_M_D patterns to put the ISO 8601 standard as the first format as these are used by ALL widespread data engineering/analytics/etl/... tools??

[MarcoGorelli]

thanks @JulianCologne - probably not worth micro-optimising the order, and as you noted, %d.%m.%Y is very common