Jump to content


Has Anyone Messed With DPDK Along With Other Kernel Bypassing Libraries?

Recommended Posts

Hey everyone,


As some of you know, I've been getting into network programming recently with the C programming language. I'm hoping to create packet forwarding software that is capable of blocking and absorbing (D)DoS attacks via XDP on our Anycast network. I also plan to resolve some of our issues listed in this thread via network programming. For example, I plan to make software that'll attach to the IPIP tunnel endpoint and it'll capture all outbound packets. From here, it'll strip the outer IP header and forward the packet to the client directly. I believe this is the best approach since there's really no other way to achieve what I want (e.g. using another default route doesn't work because SRCDS expects replies on the wrong interface, etc).


So far, I've been using raw Linux sockets within the user space to achieve all of this. I even made an IPIP forwarding program that uses raw AF_PACKET sockets here. However, after reading further, the kernel copies the packet to the user space when using raw Linux sockets. Therefore, there isn't any way to drop or modify the original packet within the program itself and the original packet is sent regardless unless dropped specifically in something like IPTables. This also results in more overhead since the kernel is literally copying the packet to the user space. I want to ensure I'm making software that processes packets as quickly as possible.


Recently, I've been doing a lot of research and I found that there are a lot of network libraries that can do something called kernel bypass. Basically, this results in the packet being handled by the network library itself and doesn't go through the kernel. These libraries have support for the user space, so I can create C applications to utilize them. For more information on kernel bypass networking libraries, I'd suggest giving this and this a read. The thing that sucks about these network libraries is that you have to do everything the kernel would normally do. Therefore, this results in needing to understand a lot more along with writing more code to do what the kernel usually did. I think this will give me a lot more experience and knowledge of how networking works in Linux/the kernel along with more control.


After reading further, I discovered a library named DPDK. I remember this name from discussions that were going on in the Networking Discord server I'm a part of. It seems like one of the more popular kernel bypass libraries for networking. However, after reading the documentation here, I discovered this is VERY complex. I've even asked a couple experienced programmers and they were also saying this is very complicated. I feel this will take a long time to understand, but I do feel like the performance gain is worth it.


I was wondering if anybody else has ever messed with DPDK or another kernel bypass networking library. I've seen others out there from the CloudFlare post I linked earlier in the thread. However, I haven't looked into them yet since many suggested DPDK (which seems to be the best choice at this moment). I just want to know what my options are and if learning the ins and outs of DPDK is truly worth it long-term. I've read that a better kernel-bypass network library will more than likely come out in the near future.



Share this post

Link to post
Share on other sites

After reading a couple comments on the CloudFlare post I mentioned, I found out I can indeed use AF_PACKET sockets to achieve better performance. In this case, I'd have to use the PACKET_NMAP option. I found a Stack Overflow thread here that gives an explanation for this:



I’m writing a traffic generator in C using the PACKET_MMAP socket option to create a ring buffer to send data over a raw socket. The ring buffer is filled with Ethernet frames to send and sendto is called. The entire contents of the ring buffer is sent over the socket which should give higher performance than having a buffer in memory and calling sendto repeatedly for every frame in the buffer that needs sending.


When not using PACKET_MMAP, upon calling sendto a single frame is copied from the buffer in the user-land memory to an SK buf in kernel memory, then the kernel must copy the packet to memory accessed by the NIC for DMA and signal the NIC to DMA the frame into it's own hardware buffers and queue it for transmission. When using the PACKET_MMAP socket option mmapped memory is allocated by the application and linked to the raw socket. The application places packets into the mmapped buffer, calls sendto and instead of the Kernel having to copy the packets into an SK buf it can read them from the mmapped buffer directly. Also "blocks" of packets can be read from the ring buffer instead of individual packets/frames. So the performance increase is one sys-call to copy multiple frames and one less copy action for each frame to get it into the NIC hardware buffers.


I'm going to continue looking into this and see how this performs compared to DPDK. I still think DPDK is worth learning since it'd give me more experience on how the kernel works with networking, etc.



Share this post

Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

  • Create New...