Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

libvma doesn't support the cascaded epoll usage model #219

Open
jiangdong-sun opened this issue Sep 22, 2016 · 5 comments
Open

libvma doesn't support the cascaded epoll usage model #219

jiangdong-sun opened this issue Sep 22, 2016 · 5 comments

Comments

@jiangdong-sun
Copy link

Tested on
VMA_VERSION: 8.0.4-0 Release built on May 31 2016 15:34:50
OFED Version: MLNX_OFED_LINUX-3.3-1.0.0.0

I guess all versions of libvma don't support cascaded epoll usage model.

BTW, libkqueue use epoll as cascade model for monitoring EPOLLOUT event, so libkqueue cannot work well with libvma.

It's easy to show the issue. But if you want the test code, I can provide then.

@jiangdong-sun jiangdong-sun changed the title libvma don't support the cascaded epoll usage model libvma doesn't support the cascaded epoll usage model Sep 22, 2016
@OphirMunk
Copy link
Contributor

@jiangdong-sun any test code that can demo your issue is appreciated. Please attach

@jiangdong-sun
Copy link
Author

jiangdong-sun commented Sep 23, 2016

The test code is attached here.
t_cascaded_epoll.c.gz

How to verify

  • compile the code

  • run it to connect any tcp server with/without libvma.so preloaded.

    The test process for your reference as below.
    without libvma.

$ unset LD_PRELOAD
$ ./t_cascaded_epoll host_ip host_port

The EPOLLIN event will be received after the connection succeeded.

But, with libvma.

$ export LD_PRELOAD=/usr/lib64/libvma.so
$ ./t_cascaded_epoll host_ip host_port

The EPOLLIN event will never be triggerred even the connection succeeded

@jiangdong-sun
Copy link
Author

@OphirMunk any feedback? Thanks!

@OphirMunk
Copy link
Contributor

@jiangdong-sun Thanks for sharing your code which enables us to recreate your case.
VMA is focused on low latency for RX/TX fast path. Is your cascaded epoll needed for low latency or not? Can the cascaded epoll be used on the "slow path" (for a specific TCP port).
If yes - I would try using libvma.conf file to indicate the specific TCP port that should be handled by the Operating System (OS) and not by VMA.
I would definitely would like to hear if you gave it a try and if it worked for you. You can find details on libvma.conf file here: http://www.mellanox.com/related-docs/prod_acceleration_software/VMA_8_0_4_User_Manual_DOC-00393.pdf (section 4.1)
In addition - I am forwarding your issue to a Mellanox field application engineer that may ask you more about the demand/justification for using cascaded epoll.

@jiangdong-sun
Copy link
Author

@OphirMunk Thanks for your reply.
Yes, cascaded epoll is needed for low latency in our environment. Actually, it's exist in libkqueue used by our application. So the slow path doesn't meet our requirement.
Hope you team can provide this feature, and I'd like to answer the questions from your field application engineer. Thanks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants