YQ-3722 add simdjson parser into RD #10204

GrigoriyPA · 2024-10-08T10:19:01Z

Changelog entry

Added simdjson parser into RD

Changelog category

Performance improvement

Additional information

github-actions · 2024-10-08T10:22:48Z

⚪ 2024-10-08 10:22:48 UTC Pre-commit check linux-x86_64-release-asan for 3f1e08d has started.
⚪ 2024-10-08 10:22:58 UTC Artifacts will be uploaded here
⚪ 2024-10-08 10:25:55 UTC ya make is running...
🔴 2024-10-08 10:31:38 UTC Build failed, see the logs.

github-actions · 2024-10-08T10:24:30Z

⚪ 2024-10-08 10:24:29 UTC Pre-commit check linux-x86_64-relwithdebinfo for 3f1e08d has started.
⚪ 2024-10-08 10:24:40 UTC Artifacts will be uploaded here
⚪ 2024-10-08 10:27:33 UTC ya make is running...
🔴 2024-10-08 10:39:56 UTC Build failed, see the logs.

github-actions · 2024-10-08T11:03:57Z

⚪ 2024-10-08 11:03:56 UTC Pre-commit check linux-x86_64-relwithdebinfo for 3500381 has started.
⚪ 2024-10-08 11:04:07 UTC Artifacts will be uploaded here
⚪ 2024-10-08 11:07:06 UTC ya make is running...
🟡 2024-10-08 12:26:57 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
41287	35721	0	1	5460	105

⚪ 2024-10-08 12:30:33 UTC ya make is running... (failed tests rerun, try 2)
🟢 2024-10-08 12:41:48 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
105 (only retried tests)	4	0	0	1	100

🟢 2024-10-08 12:41:54 UTC Build successful.
🟡 2024-10-08 12:42:13 UTC ydbd size 2.8 GiB changed* by +579.2 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `8b39e36`	merge: `3500381`	diff	diff %
ydbd size	3 008 719 720 Bytes	3 009 312 784 Bytes	+579.2 KiB	+0.020%
ydbd stripped size	477 379 640 Bytes	477 554 168 Bytes	+170.4 KiB	+0.037%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-10-08T11:04:06Z

⚪ 2024-10-08 11:04:06 UTC Pre-commit check linux-x86_64-release-asan for 3500381 has started.
⚪ 2024-10-08 11:04:17 UTC Artifacts will be uploaded here
⚪ 2024-10-08 11:07:14 UTC ya make is running...
🟡 2024-10-08 12:56:01 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
11578	11354	0	64	54	106

🟢 2024-10-08 12:57:03 UTC Build successful.
🟡 2024-10-08 12:57:36 UTC ydbd size 5.7 GiB changed* by +990.9 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `8b39e36`	merge: `3500381`	diff	diff %
ydbd size	6 097 996 744 Bytes	6 099 011 472 Bytes	+990.9 KiB	+0.017%
ydbd stripped size	1 520 324 752 Bytes	1 520 792 048 Bytes	+456.3 KiB	+0.031%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-10-08T13:37:17Z

⚪ 2024-10-08 13:37:17 UTC Pre-commit check linux-x86_64-relwithdebinfo for bee4648 has started.
⚪ 2024-10-08 13:37:27 UTC Artifacts will be uploaded here
⚪ 2024-10-08 13:40:27 UTC ya make is running...
⚫ 2024-10-08 13:58:54 UTC Check cancelled

github-actions · 2024-10-08T13:39:44Z

⚪ 2024-10-08 13:39:43 UTC Pre-commit check linux-x86_64-release-asan for bee4648 has started.
⚪ 2024-10-08 13:39:55 UTC Artifacts will be uploaded here
⚪ 2024-10-08 13:42:49 UTC ya make is running...
⚫ 2024-10-08 13:59:00 UTC Check cancelled

github-actions · 2024-10-08T14:00:37Z

⚪ 2024-10-08 14:00:37 UTC Pre-commit check linux-x86_64-relwithdebinfo for f62080e has started.
⚪ 2024-10-08 14:03:02 UTC Artifacts will be uploaded here
⚪ 2024-10-08 14:06:33 UTC ya make is running...
🟡 2024-10-08 15:02:55 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
41287	35707	0	4	5462	114

⚪ 2024-10-08 15:06:45 UTC ya make is running... (failed tests rerun, try 2)
🟡 2024-10-08 15:17:56 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
118 (only retried tests)	15	0	1	0	102

⚪ 2024-10-08 15:18:05 UTC ya make is running... (failed tests rerun, try 3)
🟢 2024-10-08 15:29:28 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
103 (only retried tests)	1	0	0	0	102

🟢 2024-10-08 15:29:36 UTC Build successful.
🟡 2024-10-08 15:30:02 UTC ydbd size 2.8 GiB changed* by +543.0 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `b8ac272`	merge: `f62080e`	diff	diff %
ydbd size	3 008 719 776 Bytes	3 009 275 856 Bytes	+543.0 KiB	+0.018%
ydbd stripped size	477 379 640 Bytes	477 551 224 Bytes	+167.6 KiB	+0.036%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-10-08T14:03:07Z

⚪ 2024-10-08 14:03:06 UTC Pre-commit check linux-x86_64-release-asan for f62080e has started.
⚪ 2024-10-08 14:03:17 UTC Artifacts will be uploaded here
⚪ 2024-10-08 14:06:10 UTC ya make is running...
🟡 2024-10-08 15:37:58 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
11603	11325	0	97	72	109

🟢 2024-10-08 15:38:57 UTC Build successful.
🟡 2024-10-08 15:39:29 UTC ydbd size 5.7 GiB changed* by +936.7 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `b8ac272`	merge: `f62080e`	diff	diff %
ydbd size	6 097 996 944 Bytes	6 098 956 080 Bytes	+936.7 KiB	+0.016%
ydbd stripped size	1 520 324 880 Bytes	1 520 778 288 Bytes	+442.8 KiB	+0.030%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

ydb/core/fq/libs/row_dispatcher/topic_session.cpp

ydb/core/fq/libs/row_dispatcher/json_parser.cpp

Added simdjson parsing

8222d36

GrigoriyPA requested a review from a team as a code owner October 8, 2024 10:19

github-actions bot added the new-feature label Oct 8, 2024

GrigoriyPA requested review from kardymonds and uzhastik October 8, 2024 10:19

github-actions bot added performance and removed new-feature labels Oct 8, 2024

Removed debug counters

41d45d6

Removed TEvDataParsed

145a3c1

Changed json filter input type

4c1fcfc

kardymonds reviewed Oct 9, 2024

View reviewed changes

ydb/core/fq/libs/row_dispatcher/topic_session.cpp Show resolved Hide resolved

ydb/core/fq/libs/row_dispatcher/json_parser.cpp Show resolved Hide resolved

GrigoriyPA requested a review from kardymonds October 9, 2024 07:50

kardymonds approved these changes Oct 9, 2024

View reviewed changes

GrigoriyPA merged commit 8fe290e into ydb-platform:main Oct 9, 2024
10 checks passed

GrigoriyPA deleted the YQ-3722-add-simdjson-parser-into-RD branch October 9, 2024 09:05

kardymonds pushed a commit to kardymonds/ydb that referenced this pull request Oct 16, 2024

YQ-3722 add simdjson parser into RD (ydb-platform#10204)

a0656e0

This was referenced Oct 21, 2024

Fix kafka cdc read and login without @ #10678

Merged

Allow short auto partitioning strategies yql #10701

Merged

GrigoriyPA mentioned this pull request Oct 23, 2024

YQ-3784 support computation graph invalidation in purecalc #10799

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

YQ-3722 add simdjson parser into RD #10204

YQ-3722 add simdjson parser into RD #10204

GrigoriyPA commented Oct 8, 2024 •

edited

Loading

github-actions bot commented Oct 8, 2024 •

edited

Loading

github-actions bot commented Oct 8, 2024 •

edited

Loading

github-actions bot commented Oct 8, 2024 •

edited

Loading

github-actions bot commented Oct 8, 2024 •

edited

Loading

github-actions bot commented Oct 8, 2024 •

edited

Loading

github-actions bot commented Oct 8, 2024 •

edited

Loading

github-actions bot commented Oct 8, 2024 •

edited

Loading

github-actions bot commented Oct 8, 2024 •

edited

Loading

YQ-3722 add simdjson parser into RD #10204

YQ-3722 add simdjson parser into RD #10204

Conversation

GrigoriyPA commented Oct 8, 2024 • edited Loading

Changelog entry

Changelog category

Additional information

github-actions bot commented Oct 8, 2024 • edited Loading

github-actions bot commented Oct 8, 2024 • edited Loading

github-actions bot commented Oct 8, 2024 • edited Loading

github-actions bot commented Oct 8, 2024 • edited Loading

github-actions bot commented Oct 8, 2024 • edited Loading

github-actions bot commented Oct 8, 2024 • edited Loading

github-actions bot commented Oct 8, 2024 • edited Loading

github-actions bot commented Oct 8, 2024 • edited Loading

GrigoriyPA commented Oct 8, 2024 •

edited

Loading

github-actions bot commented Oct 8, 2024 •

edited

Loading

github-actions bot commented Oct 8, 2024 •

edited

Loading

github-actions bot commented Oct 8, 2024 •

edited

Loading

github-actions bot commented Oct 8, 2024 •

edited

Loading

github-actions bot commented Oct 8, 2024 •

edited

Loading

github-actions bot commented Oct 8, 2024 •

edited

Loading

github-actions bot commented Oct 8, 2024 •

edited

Loading

github-actions bot commented Oct 8, 2024 •

edited

Loading