[20291] Gather-send implementation #4537

cferreiragonz · 2024-03-08T12:25:33Z

Description

This PR refactors the creation and sending of RTPS Messages by utilizing a list of NetworkBuffers (pointers + sizes) as the data transmission object, thereby eliminating an additional data message copy in the RTPSMessageGroup.
This refactor impacts both the Transports and the RTPS Layer, involving updates to the sender resources lambda functions, transport interfaces, statistics module, and the logic implemented in RTPS message creation.

Two extra ResourceLimitedVectors have been added to RTPSMessageGroup_t to act as NetworkBuffers and to contain the metadata of the payloads used in the creation of the RTPS message. In this way mallocs can be reduced as much as possible during runtime. TCP transport has mallocs during the creation of the TCP message and during the asio write operation.

SendBuffersAllocationAttributes has been extended with a new attribute defining the allocation configuration of the NetworkBuffers.

This PR is on top of: [21121] IPayloadPool refactor #4892 and needs to be merged after it, in order to use the latest changes of payload_owner_

Opened as draft:

Refactor at Transport level
Refactor at Sender Resource level
Refactor at Participant level
Statistics
Redesign on how secure messages are managed
Doxygen
Tests. They have been updated with new API. Gather-send is implicitly tested in every other send.
Documentation

Contributor Checklist

Commit messages follow the project guidelines.
The code follows the style guidelines of this project.
[N/A] Tests that thoroughly check the new feature have been added/Regression tests checking the bug and its fix have been added; the added tests pass locally
Any new/modified methods have been properly documented using Doxygen.
❌ Changes are ABI compatible.
❌ Changes are API compatible.
New feature has been added to the versions.md file (if applicable).
New feature has been documented/Current behavior is correctly described in the documentation.
Related documentation PR: [20291] Gather-send implementation Docs Fast-DDS-docs#724
[N/A] Applicable backports have been included in the description.

Reviewer Checklist

The PR has a milestone assigned.
The title and description correctly express the PR's purpose.
Check contributor checklist is correct.
Check CI results: changes do not issue any warning.
Check CI results: failing tests are unrelated with the changes.

Signed-off-by: cferreiragonz <[email protected]>

MiguelCompany

A nit that does not require to pass CI again

test/blackbox/common/DDSBlackboxTestsBasic.cpp

MiguelCompany · 2024-06-13T06:02:52Z

@richiprosima Please test_3 discovery-server

Signed-off-by: cferreiragonz <[email protected]>

* Refs #20337: Buffer list in UDP sending function Signed-off-by: cferreiragonz <[email protected]> * Refs #20337: Buffer list in SHM sending function & Copy to shared buffer function Signed-off-by: cferreiragonz <[email protected]> * Refs #20337: Buffer list in TCP sending function Signed-off-by: cferreiragonz <[email protected]> * Refs #20342: Add new Buffer structure Signed-off-by: cferreiragonz <[email protected]> * Refs #20342: Add new lambda to send buffers into SenderResources Signed-off-by: cferreiragonz <[email protected]> * Refs #20342: Refactor on UDP transport Signed-off-by: cferreiragonz <[email protected]> * Refs #20342: Refactor on UDP test_transport Signed-off-by: cferreiragonz <[email protected]> * Refs #20342: Refactor on TCP Transport Signed-off-by: cferreiragonz <[email protected]> * Refs #20342: Refactor on TCPChannelResource Signed-off-by: cferreiragonz <[email protected]> * Refs #20342: Refactor on TCP tests Signed-off-by: cferreiragonz <[email protected]> * Refs #20342: Refactor on SHM transport & enable copying multiple buffers Signed-off-by: cferreiragonz <[email protected]> * Refs #20342: Refactor on SHM tests Signed-off-by: cferreiragonz <[email protected]> * Refs #20342: Fix mock tests after rebase Signed-off-by: cferreiragonz <[email protected]> * Refs #20352: Refactor on addSubmessageData/DataFrag Signed-off-by: cferreiragonz <[email protected]> * Refs #20352: Add constructor overloads to NetworkBuffers Signed-off-by: cferreiragonz <[email protected]> * Refs #20352: Add new attributes and methods to RTPSMessageGroup.h Signed-off-by: cferreiragonz <[email protected]> * Refs #20352: Refactor on add_data() Signed-off-by: cferreiragonz <[email protected]> * Refs #20352: Refactor on add_data_frag() Signed-off-by: cferreiragonz <[email protected]> * Refs #20352: Refactor on insert_submessage() Signed-off-by: cferreiragonz <[email protected]> * Refs #20352: Refactor on send() Signed-off-by: cferreiragonz <[email protected]> * Refs #20352: Minor changes in RTPSMessageGroup Signed-off-by: cferreiragonz <[email protected]> * Refs #20352: Refactor on RTPSMessageSenderInterface Signed-off-by: cferreiragonz <[email protected]> * Refs #20352: Refactor on ChaningTransport and ABI compatible send_lambda_ Signed-off-by: cferreiragonz <[email protected]> * Refs #20352: Refactor on Statistics module Signed-off-by: cferreiragonz <[email protected]> * Refs #20352: Add security support Signed-off-by: cferreiragonz <[email protected]> * Refs #20352: Minor fixes Signed-off-by: cferreiragonz <[email protected]> * Refs #20352: Doxygen Signed-off-by: cferreiragonz <[email protected]> * Refs #20352: Delete Sender's Resource deprecated API Signed-off-by: cferreiragonz <[email protected]> * Refs #20352: Uncrustify Signed-off-by: cferreiragonz <[email protected]> * Refs #20352: Fix Windows build Signed-off-by: cferreiragonz <[email protected]> * Refs #20352: Account for change of namespaces Signed-off-by: cferreiragonz <[email protected]> * Refs #20352: Add NetworkBuffer.cpp Signed-off-by: cferreiragonz <[email protected]> * Refs #20352: Revision minor changes Signed-off-by: cferreiragonz <[email protected]> * Refs #20352: Revision minor changes 2 Signed-off-by: cferreiragonz <[email protected]> * Refs #20291: Avoid stats_msg dynamic malloc Signed-off-by: cferreiragonz <[email protected]> * Refs #20291: Use vector instead of list Signed-off-by: cferreiragonz <[email protected]> * Refs #20291: Refactor RTPSMessageGroup to avoid Mallocs Signed-off-by: cferreiragonz <[email protected]> * Refs #20291: Use limited vector to avoid repeated mallocs Signed-off-by: cferreiragonz <[email protected]> * Refs #20291: Fix rebase Signed-off-by: cferreiragonz <[email protected]> * Refs #20291: Improve doxygen Signed-off-by: cferreiragonz <[email protected]> * Refs #20291: Revision Signed-off-by: cferreiragonz <[email protected]> * Refs #20291: Add ResourceLV config into QoS Signed-off-by: cferreiragonz <[email protected]> * Refs #20291: XML - New QoS added Signed-off-by: cferreiragonz <[email protected]> * Refs #20291: Get Payload in RTPSMessageGroup Signed-off-by: cferreiragonz <[email protected]> * Refs #20291: Uncrustify Signed-off-by: cferreiragonz <[email protected]> * Refs #20291: Revision - Use RLContainerConfig and minor changes Signed-off-by: cferreiragonz <[email protected]> * Refs #20291: Revision - Get payload after correct RTPSMsg creation Signed-off-by: cferreiragonz <[email protected]> * Refs #20291: Revision - Default values Signed-off-by: cferreiragonz <[email protected]> * Refs #20291: Update versions.md & CMakeLists Signed-off-by: cferreiragonz <[email protected]> * Refs #20291: Revision - Headers & versions.md Signed-off-by: cferreiragonz <[email protected]> * Refs #20291: Adjust payload_pool test Signed-off-by: cferreiragonz <[email protected]> * Refs #20291: Test comment Signed-off-by: cferreiragonz <[email protected]> --------- Signed-off-by: cferreiragonz <[email protected]>

cferreiragonz added this to the v3.0.0 milestone Mar 11, 2024

cferreiragonz force-pushed the poc/gather-send-implementation branch 2 times, most recently from aa6e6cc to 5de9ef4 Compare March 11, 2024 14:32

cferreiragonz marked this pull request as ready for review March 11, 2024 14:34

cferreiragonz force-pushed the poc/gather-send-implementation branch from 02e2919 to af3951c Compare March 12, 2024 07:12

cferreiragonz mentioned this pull request Mar 12, 2024

[20291] Gather-send implementation Docs eProsima/Fast-DDS-docs#724

Merged

7 tasks

cferreiragonz force-pushed the poc/gather-send-implementation branch 2 times, most recently from 376e650 to 261c9a0 Compare March 12, 2024 09:17

cferreiragonz changed the base branch from master to 3.0.x-devel March 12, 2024 10:17

cferreiragonz force-pushed the poc/gather-send-implementation branch from 261c9a0 to 3067287 Compare March 12, 2024 10:18

cferreiragonz added the needs-review PR that is ready to be reviewed label Mar 12, 2024

cferreiragonz force-pushed the poc/gather-send-implementation branch 3 times, most recently from 86c7d43 to c3741e5 Compare March 14, 2024 06:59

JLBuenoLopez force-pushed the 3.0.x-devel branch from 2de726b to 4a61143 Compare March 14, 2024 07:55

cferreiragonz force-pushed the poc/gather-send-implementation branch from c3741e5 to 6752776 Compare March 14, 2024 08:49

EduPonz force-pushed the 3.0.x-devel branch from 4a61143 to 39b2115 Compare March 20, 2024 09:15

cferreiragonz force-pushed the poc/gather-send-implementation branch from 6752776 to 9344ae3 Compare March 20, 2024 09:34

EduPonz force-pushed the 3.0.x-devel branch from 1f2e8b7 to 30d4fcd Compare March 21, 2024 07:18

cferreiragonz force-pushed the poc/gather-send-implementation branch from 9344ae3 to 3ff583d Compare March 22, 2024 07:39

EduPonz force-pushed the 3.0.x-devel branch 2 times, most recently from 8193fef to a364ed8 Compare March 27, 2024 14:54

github-actions bot force-pushed the 3.0.x-devel branch from 18d8c6e to f3b3af6 Compare April 3, 2024 00:04

richiware force-pushed the 3.0.x-devel branch from fb8e560 to 943b988 Compare April 11, 2024 13:01

JesusPoderoso force-pushed the 3.0.x-devel branch from 943b988 to 2c47b5e Compare April 12, 2024 05:50

github-actions bot force-pushed the 3.0.x-devel branch from 2c47b5e to c978d52 Compare April 16, 2024 08:56

EduPonz force-pushed the 3.0.x-devel branch 3 times, most recently from 5148f5d to bd83bb2 Compare April 23, 2024 05:52

cferreiragonz added the needs rebase label Apr 23, 2024

cferreiragonz added 20 commits June 13, 2024 07:58

Refs #20352: Add NetworkBuffer.cpp

d6a4146

Signed-off-by: cferreiragonz <[email protected]>

Refs #20352: Revision minor changes

6e83cc3

Signed-off-by: cferreiragonz <[email protected]>

Refs #20352: Revision minor changes 2

17cc6ec

Signed-off-by: cferreiragonz <[email protected]>

Refs #20291: Avoid stats_msg dynamic malloc

a748ae4

Signed-off-by: cferreiragonz <[email protected]>

Refs #20291: Use vector instead of list

809ae5d

Signed-off-by: cferreiragonz <[email protected]>

Refs #20291: Refactor RTPSMessageGroup to avoid Mallocs

557c2ca

Signed-off-by: cferreiragonz <[email protected]>

Refs #20291: Use limited vector to avoid repeated mallocs

153156b

Signed-off-by: cferreiragonz <[email protected]>

Refs #20291: Fix rebase

ffaced5

Signed-off-by: cferreiragonz <[email protected]>

Refs #20291: Improve doxygen

42a51dd

Signed-off-by: cferreiragonz <[email protected]>

Refs #20291: Revision

4d9addc

Signed-off-by: cferreiragonz <[email protected]>

Refs #20291: Add ResourceLV config into QoS

4134e23

Signed-off-by: cferreiragonz <[email protected]>

Refs #20291: XML - New QoS added

020b04d

Signed-off-by: cferreiragonz <[email protected]>

Refs #20291: Get Payload in RTPSMessageGroup

c7ce07d

Signed-off-by: cferreiragonz <[email protected]>

Refs #20291: Uncrustify

f84a62f

Signed-off-by: cferreiragonz <[email protected]>

Refs #20291: Revision - Use RLContainerConfig and minor changes

32158d3

Signed-off-by: cferreiragonz <[email protected]>

Refs #20291: Revision - Get payload after correct RTPSMsg creation

a5d7bee

Signed-off-by: cferreiragonz <[email protected]>

Refs #20291: Revision - Default values

fbcc6a7

Signed-off-by: cferreiragonz <[email protected]>

Refs #20291: Update versions.md & CMakeLists

efe7deb

Signed-off-by: cferreiragonz <[email protected]>

Refs #20291: Revision - Headers & versions.md

ac0f539

Signed-off-by: cferreiragonz <[email protected]>

Refs #20291: Adjust payload_pool test

8a9eede

Signed-off-by: cferreiragonz <[email protected]>

cferreiragonz force-pushed the poc/gather-send-implementation branch from 394adb8 to 8a9eede Compare June 13, 2024 05:58

MiguelCompany self-requested a review June 13, 2024 05:59

MiguelCompany requested changes Jun 13, 2024

View reviewed changes

test/blackbox/common/DDSBlackboxTestsBasic.cpp Outdated Show resolved Hide resolved

Refs #20291: Test comment

4cf3f20

Signed-off-by: cferreiragonz <[email protected]>

MiguelCompany approved these changes Jun 13, 2024

View reviewed changes

MiguelCompany merged commit 371286c into master Jun 13, 2024
3 checks passed

MiguelCompany deleted the poc/gather-send-implementation branch June 13, 2024 12:23

elianalf mentioned this pull request Jun 14, 2024

Fix compilation warning #4945

Merged

7 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[20291] Gather-send implementation #4537

[20291] Gather-send implementation #4537

cferreiragonz commented Mar 8, 2024 •

edited by MiguelCompany

Loading

MiguelCompany left a comment

MiguelCompany commented Jun 13, 2024

[20291] Gather-send implementation #4537

[20291] Gather-send implementation #4537

Conversation

cferreiragonz commented Mar 8, 2024 • edited by MiguelCompany Loading

Description

Contributor Checklist

Reviewer Checklist

MiguelCompany left a comment

Choose a reason for hiding this comment

MiguelCompany commented Jun 13, 2024

cferreiragonz commented Mar 8, 2024 •

edited by MiguelCompany

Loading