Unable to make connections between two LANs.

[srad0]

My application is very much like the HelloWorld example, all UDP traffic, and built upon Fast-DDS 2.8.0. In trying to connect two LANs separated by a VPN using the DDS-Router 1.1.0 image (no NAT involved). If I start the DDS-Router container using --net=host in the Docker CLI, and I have the following interfaces on nodeA ...

    br-2cdcb5ab1cf: ...
    docker0: ...
    eno0: ...
    lo: ... 
    mpbr0: ...
    tun0: 172.10.12.101 ...
    virbro: ...

NodeB has same with tun0 as:
tun0: 172.12.20.20 ...

I'm running a participant using defaults with the following config in each LAN with a different profile name ...

        <participant profile_name="node_a">
            <rtps>
                <builtin>
                    <discovery_config>
                        <discoveryProtocol>SIMPLE</discoveryProtocol>
                    </discovery_config>
                </builtin>
            </rtps>
        </participant>

For the router, I listen on the local VPN interface as the public facing IP and connect to the endpoint VPN interface.

participants:
  - name: lanA
    kind: local
    domain: 0

  - name: myWAN
    kind: wan
    domain: 0
    listening-addresses:
      - ip: 172.10.12.101
        port: 40777
    connection-addresses:
      - ip: 172.12.20.20
        port: 40666

For NodeB, listening and connection is swapped and local lan is named wanB.

Yet, I've not been successful in getting anything to connect or see messages of any sort. I'm wondering if there's something going on with the internal IP of the router and how my transports are set up by using default. Can anyone explain what's going on?

Is it possible IPv6 is what I should be using since I don't have NAT? I did try with IPv6 addresses, but that didn't work either. Also don't want to use a discovery server.

[jparisu]

Hi @srad0

Your configuration seems correct.
There are 2 possible issues here:

1. The DDS Routers are not discovering each other
2. The Endpoints are not discovering the DDS Router in their respective LANs.

Both can be due to the VPN capabilities. Check that your VPN allow multicast traffic in local, and check if you need to open some ports somehow to connect different LANs of the VPN. There are many other possibilities regarding the VPN: connection failures, firewalls, etc.

My suggestion is to use the Echo Participant. Check this example https://eprosima-dds-router.readthedocs.io/en/latest/rst/examples/echo_example.html .
Using this Participant you would be able to know if your DDS Routers discover each other and if they discover the endpoints in their LANs.

For NodeB, listening and connection is swapped and local lan is named wanB.

NOTE If you are under a VPN, you just need one of the DDS Routers to have a listening-addresses and the other connection-addresses.
Check also the Domain ID of the other DDS Router, to be the same as the endpoints in its LAN.

[srad0]

I'm now testing on two hosts on my local subnet in order to simplify things in case the VPN is interfering and have changed the domain id's to be unique on each host. I'm still not seeing any messages at all with the echo participant added. I've confirmed firewalls are down.

If I stop the dds-routers and change the domain id's to be the same, that tells me my participants are working from host-to-host. Here's my latest router and profile config.

participants:

  - name: lanA
    kind: local
    domain: 1

  - name: myWAN
    kind: wan
    domain: 1
    listening-addresses:
      - ip: 192.168.20.31
        port: 40777
    connection-addresses:
      - ip: 192.168.20.33
        port: 40666

  - name: echo_participant
    kind: echo
    discovery: true
    data: true
    verbose: true

participants:

  - name: lanB
    kind: local
    domain: 2

  - name: myWAN
    kind: wan
    domain: 2
    listening-addresses:
      - ip: 192.168.20.33
        port: 40666
    connection-addresses:
      - ip: 192.168.20.31
        port: 40777

  - name: echo_participant
    kind: echo
    discovery: true
    data: true
    verbose: true

Profile for nodeA.

        <participant profile_name="nodeA">
            <domainId>1</domainId>
            <rtps>
                <builtin>
                    <discovery_config>
                        <discoveryProtocol>SIMPLE</discoveryProtocol>
                    </discovery_config>
                </builtin>
            </rtps>
        </participant>

Profile for nodeB.

        <participant profile_name="nodeB">
            <domainId>2</domainId>
            <rtps>
                <builtin>
                    <discovery_config>
                        <discoveryProtocol>SIMPLE</discoveryProtocol>
                    </discovery_config>
                </builtin>
            </rtps>
        </participant>

Also, after staring routerA in lanA, should the router's echo participant print messages upon starting the local participant without even connecting to routerB? I'm just not seeing any messages at all. However, and I know this is wrong, but if I change both participants to the same domain and start a router, they of course talk and I see messages of "new endpoint discovered" in the router output.

[jparisu]

Hi @srad0 ,

Again, the configurations seems right. It seems like there must be some errors in the configuration or in the network that we are missing.

First of all, are you using docker for the DDS Participants and / or the DDS Router? This is important, and we would need to know how you run the processes in your hosts in order to help you further.

Then, are you sure you are using the correct domains? I have seen that you opened an issue in Fast (eProsima/Fast-DDS#3284), have you managed to fix it? It could be that your Participants in the DDS Router are not in the same domain for some reason that I do not totally get.
Are you loading some XML on the DDS Router process? If you have an XML with the default file name it could be loaded and break the internal entities creation.

Also, I suggest you to remove the connection-addresses from the first router, and the listening-addresses from the second. Only one way of communication is needed if there is no NAT in the middle. You can remove the domain tags also in the wan participants.

If nothing of this works, I suggest you to execute a Fast DDS Monitor https://fast-dds-monitor.readthedocs.io/en/latest/ and check if you can see the DDS Router entities.

[srad0]

As noted above, I am using Fast-DDS 2.8.0 for the participants and using the prebuilt 1.1.0 container for the routers.

As to the issue opened previously, you'll see theres a response that it has been fixed and how. Domains are working correctly.

As to the loading of XML, did you mean to say yaml? I've been loading yaml configs into the router, and had initially loaded using the default name as shown in the router documentation via docker volumes, but have since been passing yaml files with unique names in the same way.

I'm not clear on why I would remove the connection-address from one and the listening-address from the other. I can see doing that if one side is a publisher only, and the other is a subscriber only, but if I'm using a pub and sub on each end, would I then have connection and listening for both? I am now running a pub and sub on each end. Either way, I did as you suggest and noticed that the first router showed endpoint discovery and the second did not. Then I swapped with connection on one, and listening on the other and saw the same results. Then I put them both back as I previously had them and saw endpoint discovery on each, though no data passed. This is not consistent. It makes me question these routers. The host machines that I'm running this on are in a very simple configuration, and directly attached to a local switch.

Just so I understand the use of domain tags, if I have multiple domains on a LAN, is having a domain tag within their own wan sections not the way to select and pass that domains traffic through the router?

Have yet to try the DDS monitor because these machines are not local, but will try to work that.

If you have other ideas, please suggest.

[jparisu]

Hi @srad0

Troubleshooting

With this information I am almost sure you are encountering troubles with Shared Memory Transport. The main issue here is that your entities in the same machine are trying to communicate with each other via shared memory because they detect that they are in the same host. BUT due to docker configuration they cannot access to the same shared memory segment, and thus they are unable to communicate.

Solutions

There are multiple solutions for these. However neither of them depends on the DDS Router (we have in our road map to handle this specific issue from the application, but it is not ready yet).
Please, try any of these options and update us if your problem has disappear:

• Use flag --ipc=host in the docker run command, so the docker container has access to the shared memory segment of the host.
• Run your on-host entities as root so they have access to the same memory segment of the docker container.
• Set and run your docker container with the same user as the host running the DDS Entities.
• Compile Fast DDS with cmake option -DSHM_TRANSPORT_DEFAULT=OFF.

---

Technical answers

Regarding your doubts about DDS configurations, I will try to explain a bit the basics. However this is a very wide field and I cannot describe everything here. For further information, please relay to the DDS spec: https://www.omg.org/spec/DDS/

XML

Fast DDS has the capability to upload XML files with configurations for the DDS entities.
The DDS Router runs Fast DDS entities underneath to perform the communication. Thus, these entities can be configured by XML. However we do not encourage this because these entities are already configured correctly for their purpose
Apart from that, there is a YAML configuration file only for the DDS Router, but is not related with the XML one.

Initial-Peers

The configuration over wan using the DDS Router uses a discovery protocol called Initial-Peers. These protocol just requires for one of the DomainParticipants (A) to know where the other is (by setting a connection-address), and the other (B) to set where to listen (listening-address). Once A has contact B, B will know the location of A and will answer using the same channel and the communication would have been established. Thus, this mechanism will work as long as there is no NAT in the middle.
In the case there is a NAT, both must set their connection and listening addresses.

Fast DDS Monitor

The Fast DDS Monitor tool does not require to run in the same machine as the DDS network it is monitoring.

Domain

The domain set for simple/local participants in the DDS Router is to choose in which DDS DomainId the DomainParticipant will communicate. This uses underneath a specific multicast port to discover automatically every other Participant in the same LAN.
When using Initial-Peers (wan) the domain tag has no effect, and thus it is not needed.