How to use the destination ip of the incoming packet as the packet return source ip

[yudexiaozhou]

• in this img
  • client ip: *.53
  • vip ip: *.42
  • server ip: *.200
• and we are using lvs and using tun mode

[yudexiaozhou]

In QuicheQuicServerCodec.handleServer(), can I append the recipient address to the DatagramPacket that is being written and flushed to the channel? For example, would this code work:

ctx.writeAndFlush(new DatagramPacket(out.writerIndex(outWriterIndex + res), sender, recipient));

[normanmaurer]

No it will not... as far as I know it's not possible as the sender address is the one that the socket is "bound to". What you can do is to explicit bind a socket to a specify address when you bootstrap it.

[yudexiaozhou]

I'm really sorry, but I didn't quite understand how I was supposed to go about it

[normanmaurer]

What I was saying is that you can specify the address when creating the DatagramChannel. Something like this:

try {
            InetSocketAddress addressToBindTo = new InetSocketAddress(....);
            Bootstrap bs = new Bootstrap();
            Channel channel = bs.group(group)
                    .channel(NioDatagramChannel.class)
                    .handler(codec)
                    .bind(addressToBindTo).sync().channel();
            channel.closeFuture().sync();
        } finally {
            group.shutdownGracefully();
        }

[yudexiaozhou]

Is that right? I did try it, but in my scenario, the destination ip of the packet can't be fixed, so it's harder to fulfill the requirement this way.
I see nginx has this logic:
nginx inside ngx_sendmsg_vec, ngx_set_srcaddr_cmsg, if the server is listening to 0.0.0.0, the socket is not bound to a specific ip address, back to the packet is to be through this interface to specify the destination ip for the packet back to the source ip of the interface, and ultimately sent out through the sendmsg. The receiving packet uses recvmsg to get the source destination ip of the packet. otherwise the kernel sends out the packet by routing to select srcip as the source ip and sends it out.
What I'm trying to say is whether I can find an implementation of this in netty

[normanmaurer]

Can you point me to the nginx impl of it ?

[normanmaurer]

Ok I found it... I suspect we could support it in netty but its not there atm:

https://github.com/nginx/nginx/blob/master/src/os/unix/ngx_udp_sendmsg_chain.c#L246

[yudexiaozhou]

The relevant code is here

static ssize_t
ngx_sendmsg_vec(ngx_connection_t *c, ngx_iovec_t *vec)
{
    struct msghdr    msg;

#if (NGX_HAVE_ADDRINFO_CMSG)
    struct cmsghdr  *cmsg;
    u_char           msg_control[CMSG_SPACE(sizeof(ngx_addrinfo_t))];
#endif

    // Initializes a 'struct msghdr' structure. This structure is used to control how messages are sent.
    ngx_memzero(&msg, sizeof(struct msghdr));

    // If the connection has an address, set it to the 'msg.msg_name' field.
    if (c->socklen) {
        msg.msg_name = c->sockaddr;
        msg.msg_namelen = c->socklen;
    }

    // The data to be sent is set to the 'msg.msg_iov' field.
    msg.msg_iov = vec->iovs;
    msg.msg_iovlen = vec->count;

    // If the connection is a listening connection and has a wildcard address, set the source address to the 'msg.msg_control' field.
#if (NGX_HAVE_ADDRINFO_CMSG)
    if (c->listening && c->listening->wildcard && c->local_sockaddr) {

        // Assign a control message.
        msg.msg_control = msg_control;
        msg.msg_controllen = sizeof(msg_control);
        ngx_memzero(msg_control, sizeof(msg_control));

        // Get the first control message.
        cmsg = CMSG_FIRSTHDR(&msg);

        // Sets the type and length of the control message.
        msg.msg_controllen = ngx_set_srcaddr_cmsg(cmsg, c->local_sockaddr);
    }
#endif

    // Call the ngx_sendmsg() function to send the message.
    return ngx_sendmsg(c, &msg, 0);
}

[yudexiaozhou]

Yes, this is the code. Do you have any ideas on how I can help?

[normanmaurer]

I just created netty/netty#13495. This will need to be implemented via JNI for the epoll transport. Not sure yet when I will have time to do so atm, so if you have experience with JNI you might want to work on a PR against netty.

[yudexiaozhou]

I'm glad you'll be able to support such a feature in the future.
I don't have any JNI-related experience yet, but I guess I could give it a try, although it feels unlikely to be possible to do so