Jump to content
Need help joining CS:GO servers now that CS2 has released? Click here for our guide on getting back into CS:GO. ×

Roy

Banned
  Milestones (ALPHA)
 View Profile  Activity

  • Posts

    2,853

  • Joined

  • Last visited

  • Days Won

    383

 Content Type 

Posts posted by Roy

    XDP or The DPDK For Bifrost?

    in GFL's Network

    Posted

    Hey everyone,

     

    I am making this thread for documentation purposes.

     

    Before continuing, I just want to state information in this thread is based off of what I've read online so far. This means information in this thread is subject to change over time. I'm going to be continuing to read more about the differences between XDP and the DPDK along with occasionally posting updates with new information I find, etc.

     

    With that said, if you see incorrect information in this thread, please feel free to reply with corrections. I am trying to understand and learn as much as I can regarding XDP and the DPDK because I want to use both for personal and GFL projects in the future.

     

    Quick Introduction

    @Dreae and I are going to be creating custom packet processing/filtering software that runs on POPs within our Anycast network. This software will be responsible for forwarding all legitimate game server traffic to the appropriate game server machines and dropping any other traffic (including malicious). Before continuing, I'd recommend reading this and this thread to learn more about our plans with Bifrost along with the Anycast infrastructure itself.

     

    In doing so, a very important part of the software is what we're going to be using to drop traffic that we don't see as legitimate/needed. Initially, our plan was to use a combination of XDP, TC, and NFTables. We'd use XDP to drop any traffic we don't need (or malicious), TC to handle outbound game server traffic (wrapping in FOU-IPIP) along with handshake processes with our game servers, and NFTables to forward legitimate traffic to the game server machines. However, I found that this setup might be a bit messy since we'd possibly have packets going through the XDP program down to the TC filters. With that said, we'd need to order all of our TC BPF programs under the same qdisc which will take some work.

     

    I also want to note that the DPDK and XDP are both great solutions for (D)DoS mitigation in my opinion. I'm just trying to find out what would be best for Bifrost. I will now make a summary of the DPDK vs XDP along diving into the specifics.

     

    XDP vs The DPDK (Summary)

    XDP (Express Data Path) is a low-level hook within the Linux kernel. The hook is implemented into the NIC driver assuming the NIC driver supports XDP-native. If that specific NIC driver doesn't support XDP-native, the program will need to use XDP-generic which is a hook later on in the networking path (after SKB allocation). While the XDP-generic hook is a lot slower than the XDP-native hook, I do believe it has similar performance to NFTables/IPTables. At the moment, we use Compressor as our packet processing software which was made by Dreae and also uses XDP. Unfortunately, loading Compressor with XDP-native breaks the AF_XDP program either due to outdated AF_XDP code or a bug within the virtio_net NIC driver. Either way, I figured we might as well work on Bifrost instead of rewriting the AF_XDP code (which is complicated and AF_XDP is just buggy based off of my experience).

     

    The DPDK (Data Plane Development Kit) is a collection of libraries and drivers which can be used for fast packet processing. This was made by Intel. The DPDK is considered a kernel-bypass framework because it creates a fast-path (zero-copy) from the NIC to the DPDK application running within the user space. With that said, the DPDK includes a PMD (Poll Mode Driver) that constantly polls the NIC for packets. What this means is instead of the NIC raising an interrupt to the CPU when a frame/packet is received, it constantly polls the NIC. Though, note this requires CPUs to be dedicated to the DPDK application and they will be utilized at 100% CPU at all times. This results in less latency when processing packets when using the DPDK.

     

    I will now write bullet points of what I've found for XDP vs the DPDK (the pros of each).

     

    XDP Pros

    • Less things to learn/keep track of.
    • Allows us to use kernel functionality.
    • Allows the option of using "busy" polling or interrupt driven networking.
    • Removes the need of dedicating specific cores to the program.
    • Removes the need of installing third-party dependencies which is needed for the DPDK.
    • Wouldn't need to rewrite the entire network stack including TCP/IP, ARP, etc. (assuming we don't use KNI with the DPDK or an already-made library on top of the DPDK, read below).
    • Dreae and I are already familiar with writing XDP programs.

     

    The DPDK Pros

    • Faster than XDP (I believe by around 5 - 10%). Here are benchmarks I found from 2018. Here's another PDF I was reading that includes information regarding performance with the DPDK vs XDP.
    • Since the DPDK application runs within the user space, we should be able to implement all the functionality we need into this single DPDK application instead of having it spread out between XDP, TC, and NFTables.
    • Wouldn't have to use eBPF maps to transfer data from the user space to the program processing packets.

     

    Note that the three pros for the DPDK would be very beneficial in our situation in my opinion. However, there are also a lot of drawbacks with the DPDK depending on how we approach it. I will go into detail below.

     

    XDP

    As noted above, XDP was our initial solution to dropping unneeded/malicious packets. I'm not entirely sure how we would go about the program itself. We could either have the XDP program running at all times and search the source IP on an eBPF map and if the source IP is found within this map, drop the packet (this map would act as a blacklist map). Otherwise, we could have something inspect packets later on in the networking stack and if malicious traffic is detected, spawn an XDP program that would drop said traffic.

     

    XDP is a great solution, but the DPDK is still technically faster when processing/dropping packets. With that said, our setup will be a bit more messy with XDP since our plan is to only have XDP drop the packets themselves (for now). Therefore, we'll need to setup TC BPF programs as well and if we were to take a whitelist approach, we'd need to figure out how to pass initial connections through the XDP program without having XDP drop them by default (a whitelist approach was our initial plan as well).

     

    Additionally, we'll need to have NICs where the driver supports XDP-native since using the XDP-native hook would be the best for performance. This shouldn't be an issue, but it's worth noting. Thankfully, I found a new presentation recently that goes over the process of implementing XDP-native support into a NIC driver here. This'll be useful if we ever need to do anything like that in the future (and it's something I'm interested in as well).

     

    The DPDK

    I still have a lot of exploring to do with the DPDK, but I will put down the information I know as of right now. The DPDK is a pretty big framework to learn. For example, here is the programmer's documentation that is 65 sections alone (some sections being VERY long such as sections 3 and 12). With that said, if we were to build our own DPDK application from scratch, we would need to implement our own network stack for handling TCP/IP connections along with ARP requests. Maintaining our own TCP/IP network stack for a firewall isn't ideal and that'd be very time-consuming to create and manage over time. That said, maintaining the performance of the TCP/IP stack is an entire different story. This is something I'm interested in doing in general, but not for a single firewall application and I don't believe I'd learn all of this any time soon.

     

    Now, there are two possible approaches we could take that'd result in us not having to implement our own network stack to my understanding.

     

    Firstly, we could use DPDK's KNI library (Kernel NIC Interface). I haven't read the specifics of this yet, but to my understanding it'd act as a TAP device within Linux. This would allow us to send packets we don't need to process down the default network stack on Linux. The only con I've heard of with doing this is performance degrades. For example, take a look at the replies from this and this Stack Overflow thread. People have suggested using the KNI library. However, also mentioned that it does come with a performance impact. Now, what's not made clear within these replies is whether this performance degrade occurs when sending packets down the default Linux network stack or if it includes packets that we want to process with the DPDK. If this impacts only packets sent down to the network stack (AKA packets we mostly don't care about such as traffic to the POP itself, BGP, and so on), this wouldn't be a big deal because having a slight performance degrade with them won't matter much. However, if the performance is impacted on packets we want to drop/process via the DPDK application while using the KNI library, there really isn't a point in using the DPDK to begin with since our main goal is to achieve the maximum performance possible.

     

    Secondly, we could use an already-written TCP/IP stack that is built on-top of the DPDK. I've seen some people suggest this, but they also mentioned it most likely comes with performance degrades depending on which library you go with. This is definitely something I'll continue looking into, though.

     

    I still need to do a lot more research regarding this. However, if we can use KNI or an existing TCP/IP stack (along with ARP) with no performance degrades on the packets we want to process/drop within the DPDK program, I still feel the DPDK is something we should definitely consider with Bifrost.

     

    Summary

    Overall, the DPDK and XDP are both great solutions to what we want to accomplish with the network (being able to mitigate large (D)DoS attacks). However, both have their own pros and cons. I still need to do a lot more research on the DPDK before making a decision and I will continue to post updates here as I find new information. I've been reading a lot of code from open-source DPDK applications on GitHub (e.g. searching for this topic) and from here as well. This has been helping greatly with understanding the DPDK :)

     

    Thank you for reading!

    Relaxing Instrumental Music

    in Off-Topic & Random Questions

    Posted

    Hey,

     

    I'm making this thread to share relaxing instrumental music and to see if anybody else is into this type of music. I haven't really been sharing the relaxing music I've listened to until recently. The people I've sent the music to so far seem to enjoy it which is why I'm making this thread.

     

    Feel free to share any relaxing instrumental music you like :) I'm honestly looking for more relaxing music!

     

    Here are the ones I really like. Most are available on Spotify as well, but I usually find them via YouTube.

     

     

     

     

     

    From The Walking Dead:

     

     

    Halo:

     

     

     

    I have others as well which I can post in the future.

    Backgrounds/Wallpapers

    in Off-Topic & Random Questions

    Posted

    Heya,

     

    I just wanted to make a thread where we can all share backgrounds/wallpapers. Personally, I really like space-themed backgrounds/wallpapers. I've been looking around for new ones as well and figured some people might like the ones I've found so far. That said, I'll probably like others people share here as well!

     

    Here are the backgrounds/wallpapers I personally like. I know some don't have the best quality, but I'm hoping to find them in better quality in the future!

     

    coolpic.jpg

     

    I3O1jZt3CK.jpg

     

    manipulation-1596242131789-8316.jpg

     

    phone-beaches-dream-moons-fi-waves-sci-planets-manip-ocean-sun-earth-sky-full-hd-jpg.jpg

     

    b43dfd485863e606287add7c959477eb.jpg

     

    Feel free to share more here :) I'll be posting in this thread as I find more on Google or w/e.

    Time Vote for COD Event

    in Closed Events

    Posted

    9 hours ago, RivalRevival said:

    Assuming this one won't be closed last second like the majority of EC plans that don't involve discord:

     

    @Roy me and you? Couple of dummy thicc sniper bois?

    Sounds good to me. Do we know which game mode we'll play?

    Anycast Update - New POP Hosting Provider, Upgrades, and More!

    in Updates

    Posted

    Hey everyone,

     

    I just wanted to announce some exciting news regarding our Anycast network!

     

    As some of you probably know, I have plans to upgrade our Anycast network. For more information regarding the expansion itself along with what we're planning to do to mitigate (D)DoS attacks, I'd recommend reading this and this thread. The goal of this expansion is to strengthen our (D)DoS protection and acquire better hosting providers which'll result in:

     

    • Better routing.
    • More network capacity to mitigate higher volume (D)DoS attacks.
    • Better support.
    • And more!

     

    I will now discuss the updates themselves individually.

     

    New Hosting Provider and Dallas POP (GSK)

    Recently, I've came into contact with the owner of Gameserverkings (GSK), Renual, and we've been talking a lot the past couple of weeks. At first, we were talking about Anycast and (D)DoS protection in general. However, Renual felt we'd benefit from having them as a part of our network and I agreed.

     

    After talking to Renual for the last two weeks, it was pretty obvious he knew what he was doing and he has given me A TON of useful information in regards to (D)DoS mitigation/attacks, Anycast, BGP, routing, ISPs, fiber optics, network equipment, and more. He also mentioned his plans for GSK and I'm honestly really excited for their future! When putting into account his mindset, talents, and goals for GSK, I would not be surprised if they became a very big hosting provider in the future. I'm honestly surprised I haven't heard of them before, but I don't think many gaming communities in the Source Engine world know of them. That'll probably be changing, though.

     

    Renual offered us a trial run for a machine with the following specs:

     

    • AMD Ryzen 5 3600 @ 3.6 GHz (4.2 GHz turbo).
    • 32 GBs of DDR4 RAM.
    • 500 GBs NVMe.
    • $75.00/m (we aren't paying for this yet until the trial run is over and if we are satisfied).

     

    These specs are probably more overkill with our current setup, but I want to give this a shot and if things run okay, I think I'm going to take a quality over quantity approach with the network and use GSK in our primary locations such as Dallas, TX and London, UK. They also have VPSs coming up, so if need to be, we can downgrade to a VPS which should still be powerful since it'll include dedicated cores IIRC.

     

    They also plan to expand into other locations in the future that includes NYC and Chicago. Once they have these locations available, we'll be likely purchasing machines or VPSs in those locations as well.

     

    GSK's DDoS protection is great and Renual has showed me proof of them being able to mitigate larger DDoS attacks (e.g. attacks that exceed 100 million PPS). In fact, a majority of the malicious traffic that goes through this new POP will more than likely be filtered by GSK's equipment. However, with BGP Flowspec support (which will be discussed below as a subject), any attacks BiFrost detects can be pushed to the upstreams. Therefore, GSK's upstreams and equipment could take care of those detected attacks.

     

    Another pro is if need to be, we can order special equipment (e.g. switches and so on) and get colocation space within GSK. I don't personally believe we'll need that for now, but it's something to keep in mind for the future. We may also request a replacement with any part in our server (e.g. if we wanted to, we could upgrade our server to a 10 or 40 Gb NIC and GSK has 10+ Gb links available for an extra ~$50/m).

     

    Finally, I feel having direct contact with Renual will be beneficial to us in the case an issue arises or just generally speaking.

     

    Anyways, the setup of the new POP hasn't been as smooth as I'd like, but it was to be expected. Since GSK's filtering is pretty strict (for protection reasons), we had some issues getting the POP up and working with Compressor and so on. Thankfully, all of that appears to be fixed now and Renual was very quick to resolve these filtering issues. I started announcing the POP this morning and everything appears to be working okay so far. I did have to make some changes to Compressor (that include the new filters) since the function to get the CPU count was actually returning 32 cores instead of 12, so when we were looping through CPUs on our handshake BPF map, after CPU #12, there was garbage data which was resulting in unexpected behavior. Probably something worth mentioning to the Linux kernel devs, haha.

     

    If you see any issues connecting to our game servers, please let me know!

     

    Also, here's a cool pic of our current Dallas POP server in the GSK rack :)

     

    Na1r0wcfBC.jpg

     

    P.S. These servers are liquid-cooled :)

     

    And I just wanted to give a big shout out to Renual for being so helpful recently!

     

    BGP Flowspec Support In Bifrost

    I just wanted to briefly go over our plans to implement BGP Flowspec support into Bifrost. What BGP Flowspec will allow us to do is push policies to our upstreams via BGP. What this means is if we detect an attack with BiFrost and push it to our upstreams, they will filter the attack instead of the POP server/Bifrost. This'll result in less load on the POP server and if an attack exceeded our POP's link, this would stop the attack from saturating our link since the upstreams would filter the malicious traffic instead.

     

    At the moment, this is only supported on GSK I believe, but given we'll be likely expanding with them in the future, this is something worth implementing in my opinion. Unfortunately, you can't push policies that include payload matching via BGP Flowspec. The policies operate more like an ACL. Therefore, if we're getting hit by a larger (D)DoS attack that gets through the GSK filters, but each malicious packet's source IP isn't spoofed randomly each time, we can push a policy to filter that source IP via BGP Flowspec and GSK's upstreams will accept that policy along with GSK's equipment.

     

    I believe we'll also be able to push policies including TTLs (time-to-live) and more as well which gives us a bit more control than a typical ACL. Just sadly payload matching isn't supported within the policies themselves.

     

    DPDK Over XDP

    Another subject I wanted to briefly go over is the possibility of us using DPDK to process/drop packets instead of XDP in Bifrost. While XDP is still great and has its own pros over DPDK (e.g. being able to use kernel functionality and just easier to learn/understand/work with at times since the network stack is already implemented), DPDK is still technically faster than XDP when dropping packets. If you want some benchmarks, feel free to look at this. Note that these are from 2018, but I do believe these are still relevant today (I've talked to a few experienced people about DPDK vs XDP recently and they all stated DPDK still has an edge over XDP). I do plan on making my own benchmarks in the future with XDP vs DPDK once I purchase my home servers. However, that'll take some time.

     

    The only cons with DPDK are that we'd need to implement the network stack ourselves and it is also a lot more complicated. For example, regarding the first con, we'd need to implement functionality ourselves to handle ARP packets in DPDK.

     

    With that said, I tried looking into DPDK in the past, but their documentation is VERY long and time consuming. Please take a look here if you're interested. The third section alone feels like a book to read! There may be easier ways to go about getting familiar with DPDK though, so that's something I'll be looking into.

     

    Overall, while DPDK is complicated and time-consuming, I do believe it's something we should try to do with Bifrost because it'd give us more control and it's also probably one of the fastest libraries to use when dropping (malicious) packets.

     

    HellsGamers And Our Anycast Network

    In the past two months or so, I offered HG a few Anycast IPs from GFL's IP block (92.119.148.0/24) to use for a trial run to see if they were interested in utilizing our Anycast network. I got told that they were indeed interested a couple weeks ago after the trial run and they decided to purchase their own /24 IPv4 block that is currently being pointed to our ASN. All the LOAs and BGP sessions are pretty much set up at this point and the only thing left to do is reconfigure BIRD (our BGP daemon) on each of our POP servers to announce their IPv4 block and add IP allocations to Compressor for them to use. I'm hoping to get this completed by early next week.

     

    HG will also be paying us monthly for this usage. I will not disclose the cost, but I believe this'll help with our Anycast expansion and allow us to spend more on it which'll result in better DDoS protection, routing, network capacity, and more!

     

    As of right now, HG will be the only community outside of GFL that'll utilize our network. I don't have any plans at the moment to accept other gaming communities. However, it's something I will consider in the future once my life starts improving and I'm less stressed since I feel like taking on the responsibility of more communities will add a lot more stress onto me (I'm already very stressed, to be honest). But then again, it's something I'm willing to do once the network is mostly expanded and my life situation starts improving (so I'll be able to put in work without feeling super stressed already).

     

    Overall, these changes are very exciting in my opinion! I know a lot of you will not understand what a lot of the above mean due to the technical aspect, but the things we're accomplishing with the network is very impressive. Especially for just a gaming community. We still have a lot more to do, but I'm confident we'll get there :)

     

    If you have any questions, please feel free to reply.

     

    Thank you!

    Anycast Network Issues

    in Updates

    Posted

    Another Update

    This issue was resolved! It appears our new POP hosting provider and Nexril use Hivelocity. Nexril and I are assuming there was some sort of internal conflict when announcing the new POP yesterday. Once Nexril made it so traffic back to the Anycast network routes through Cogent instead of Hivelocity, the POPs starting seeing IPIP traffic again from the game server machines.

     

    Honestly, this is one of the strangest issues I've seen so far. But I'm glad we were able to troubleshoot and resolve the issue. I've recompiled the IPIPDirect program on all machines and exclude the A2S_INFO for caching purposes. There doesn't appear to be any issues at the moment from this.

     

    Thanks!

    Anycast Network Issues

    in Updates

    Posted

    Update

    I've implemented IPIP support into my Packet Flooding tool here. I've concluded this is affecting IPIP traffic on all machines from Nexril (I thought initially it was only affecting 3/4 of our machines).

     

    I am sending 11 mbps consistently with the above tool so our hosting provider can track down these packets easier.

     

    I'm waiting for an update on their side.

     

    Thank you.

    Anycast Network Issues

    in Updates

    Posted

    Also, to add, I don't believe this issue is related to the new hosting provider directly. However, it seems perhaps the updating of routes when doing the BGP session may have messed up something with Nexril.

     

    Thanks.

    Anycast Network Issues

    in Updates

    Posted

    Hey everyone,

     

    Tonight, our Anycast network experienced a major issue for our servers located in Dallas, TX under Nexril. Around a couple hours ago, I started announcing a new POP server in Dallas, TX which is a part of the major Anycast expansion that I'll be making a thread about tomorrow or whenever this issue is fixed.

     

    What started occurring was all game servers under Nexril weren't responding to A2S_INFO queries. This resulted in the server being thrown off the server browser. At first, I thought this was due to the AF_XDP part of Compressor breaking on the new POP server (which is a common reason to why A2S_INFO wouldn't be working and we've ran into this in the past). However, after disabling the POP server and confirming the BGP session was inactive, the issue continued to occur. I confirmed these machines were routing to one of our Vultr's POPs and not the new one. Therefore, I was honestly stumped on what the issue could be.

     

    After troubleshooting, I discovered the servers completely went down for the Nexril machines when I disabled my IPIP Direct program. This meant all game server packets being sent back from the Nexril machines to the closest POP server (our Vultr Dallas POP) weren't making it back. This also would explain why A2S_INFO broke initially because my IPIP Direct program is configured to send A2S_INFO responses back through the POP only for caching purposes (by this line).

     

    So this had me confused because when running an MTR to the Anycast network from the Nexril machines, it was receiving ICMP replies fine. With that said, I tried disabling BGP on the Vultr Dallas POP so the machines routed to the Chicago POP. However, the same issue occurred. This is when I needed to dig even deeper and unfortunately, since XDP isn't debug-friendly (what Compressor, our packet processing software uses), I did have to stop Compressor on our Dallas POP multiple times after announcing it to the network again (so the machines routed to this POP again, etc).

     

    When I did this, I used my Packet Flooding tool here to continuously send UDP and TCP packets back to the Anycast network every second (the closest POP which is Dallas, TX) from the Nexril machine. When Compressor was disabled and I had a TCPDump running, it WAS able to see the standard UDP/TCP packets. However, when I disabled the IPIPDirect program on the Nexril machine, Compressor on the Dallas POP, and ran a TCPDump for any packet coming from the game server machine on the Dallas POP, it ONLY saw those standard TCP/UDP packets. It DID NOT see the IPIP packets being sent back. I also ran a TCPDump on the game server machine catching all the IPIP packets it was trying to send back to the POP and there were many of them. None of them made it back to the Dallas POP server via TCPDump.

     

    It appears one of the routers/hops from the Nexril machines to the Dallas POP are dropping/filtering our IPIP packets. I honestly haven't seen such a strange issue before on our Anycast network.

     

    I've opened up a ticket with Nexril and also recorded a video of all my troubleshooting showing there's clearly an issue with IPIP packets. I'm waiting for a response from them.

     

    In the meantime, what I did to get the servers up and running is run the IPIPDirect program and also include A2S_INFO responses so it'll send those out directly instead of through the POP. A2S_INFO caching will be disabled due to this, but once Nexril and us are able to figure out the issue, I'll be rebuilding the IPIPDirect program to send A2S_INFO responses back through the POP again for caching purposes.

     

    I do want to apologize for the inconvenience and downtime associated with our Dallas servers along with players routing through our Dallas POP since I needed to take that down at times for debugging purposes.

     

    Once I have an update, I will let you know.

     

    Thank you.

    need help for something

    in Programming

    Posted

    For glua (GMod lua to my understanding), I'd suggest starting here. If you want to get familiar with lua itself, I found this nice set of tutorials as well.

     

    In the first link, the "Developer Reference" section will be very useful for once you understand the lua syntax.

     

    With that said, I hope the game mode turns out well. If this is your first time messing with lua, I'd suggest starting off small such as making basic addons and scripts since making a game mode is a bigger task and may be overwhelming for beginners.

     

    I haven't messed with lua as much throughout the last few years. Therefore, other more experienced developers may have more pointers, etc. There used to be a lot of tutorials on FacePunch's forum. However, that was taken down some time ago without any official archives unfortunately.

     

    I hope the above helps :)

    need help for something

    in Programming

    Posted

    I've moved this to the programming section :)

     

    Notepad++ is a decent editor. However, you may also want to look into using Visual Studio Code. I switched from NotePad++ to VS Code a few months ago and I personally like it a lot more.

     

    In regards to lua, are you looking to learn lua to develop addons/scripts for Garry's Mod?

    The FoF servers could be cancelled, what's next?

    in General Discussion

    Posted

    I plan on looking more into this later today (currently busy with work). As @Liloz01 mentioned, a lot of the thoughts behind A2S_INFO caching were mentioned in my Anycast expansions thread as a post:

     

     

    I'd especially suggest reading my Google Doc as well here which goes over my thought process with it all. I'm not expecting people to agree with them and I understand why they wouldn't. Though, this was the decision I made after deploying the network in March of 2019.

     

    I don't blame the person who made that Steam thread as well. Their points are valid in terms of the inaccurate pings. However, caching the A2S_INFO packet does actually provide a benefit in terms of (D)DoS protection. The reason attackers are targeting this query to begin with is because it allows them to send A2S_INFO requests which are only 25 bytes in payload size. When the server responds (without caching the packet), it has to first gather all the information that the A2S_INFO response contains (e.g. mod directory, host name, player count, map name, etc) and send it back to the source which is usually always larger than the request size. The combination of getting all of the information when each request comes in and also sending back a response each time results in it being easier to peg the server's CPU resources. Given SRCDS is also mostly single-threaded, it really isn't that difficult to peg the server with these attacks from what I've seen (I've easily done this with my packet flooding tool I made here that supports specifying payloads when pen-testing internally). With that said, the attacker could technically use any of the other queries Valve provides to their advantage, but the reason people choose the A2S_INFO query specifically is if you don't have any caching mechanism, it puts the servers into a lose-lose situation. Valve DOES have commands to limit the in-game effects from the A2S_INFO attacks which are the following:

      

    sv_max_queries_sec
sv_max_queries_window
sv_max_queries_sec_global

     

    The problem is, typically, the global limit will be reached and this results in your server being thrown off of the server browser from the attack. Therefore, in situations where you're getting attacked via an A2S_INFO spam attack, your server will either become pegged and probably unplayable in-game or you set the above limits and your server doesn't come up on the server browser. This is why I say it's a lose-lose situation for the server owners/community.

     

    Now, you COULD also install extensions from let's say SourceMod/MetaMod (for most Source Engine games) or lua/C++ (for GMod typically) that caches the query on the game server level. Technically, this is a pretty good defense. However, firstly, it's hard to find an extension that actually works, especially for SourceMod/MetaMod (the three I've found I couldn't get to work on my test CS:S Linux server). If I spent more time trying to get it work, I probably could, but I'm assuming some things have to be rewritten and I don't have much experience with making C++ extensions for Source Engine games (using SourceMod or MetaMod as the addons). However, if you were to get this working, it'd still not be as good as caching the packet on each POP in terms of protection. This is because the game server would still be replying to each request with a cached response. Given a point for making the Anycast network was to allow us to find hosting providers with less (D)DoS protection for our game server machines (since we'd be relying on the Anycast network for protection which is the whole point), this would allow attackers to potentially still use the A2S_INFO vector to their advantage because they'd be able to send so many request that it'd over flood the game server machine itself (that doesn't have any real (D)DoS protection and just an one gigabit NIC/link). This is why caching it on each POP would be better since it'd distribute the load and each POP would respond to the queries instead of just the single game server machine.

     

    Anyways, given our FoF servers are less likely to see these attacks, we can disable caching for these servers specifically if Rebel wants us to do so or w/e. I also wouldn't mind if they reached out to me and we talked about this more in PM. I may reach out to them once time frees up.

     

    I hope the above helps some understand. Like I said, all my views are within that Google Doc. Some points I can definitely see people disagreeing with since it isn't all about protection and I understand them completely as mentioned above. However, given the combination of the points I listed along with the concerns, this is why I decided we'd be caching the A2S_INFO query on all POPs.

     

    Thank you.

    TC Ingress IPIP Blocker

    in Projects

    Posted

    TC Ingress IPIP Blocker

    Hey everyone,

     

    I just figured I'd share a small project I made last night. This project was made because I've heard of players bypassing the Source Engine IP blocking method on HG and GFL's JailBreak server. I know HG was trying to block these players through their firewall on the machine, but the problem is packets coming in from the Anycast network are in IPIP formation and unfortunately, most firewall tools like IPTables and NFTables don't have standard methods to block packets based off of the inner IP header's source address.

     

    You may be able to do this with IPTable's or NFTable's payload matching. But this would be a pain and you'd need to convert the IP to binary data along with network byte order. With that said, the TC ingress filter is definitely faster than both IPTables and NFTables.

     

    Description
    A simple TC BPF program that attaches to the ingress filter and blocks any IPs stored in the specified file (default is /etc/IPIPBlock/list.conf). This program checks the source IP of the inner IP header.

     

    Usage
    Usage is as follows:

      

    ./IPIPBlock --dev <interface> --list <file> --time <updatetime> [--help]

     

    Where <interface> is the interface incoming IPIP packets enter and <file> is the file that contains all the IPs to blacklist. The default interface is ens18 and the default file is /etc/IPIPBlock/list.conf. The <updatetime> value indicates how often to update the blacklist map from the local file.

     

    Note - Comments or characters after an IP in the blacklist file should be fine. I've tested this and there were no changes in behavior compared to nothing being added after an IP per line.

     

    For example, the following works:

      

    192.168.90.1
80.4.23.12 # Malicious host (not actually) and this will still block regardless of the comment.
garbage # This never gets processed from what I've seen and is just treated as a garbage value.

     

    Building
    You may use git and make to build this project. For example:

      

    git --recursive https://github.com/gamemann/TC-Ingress-IPIP-Blocker.git
cd TC-Ingress-IPIP-Blocker/
make && sudo make install

     

    Note - Clang and LLVM are required to build this project.

     

    Credits

     

    GitHub Repository

    Anycast Expansion Plans/Approaches

    in GFL's Network

    Posted

    27 minutes ago, LilyShiro said:

    Reading the google doc you listed was a good read, and I understand why you do what you do. Thanks for it.

     

    I still, of course as an outsider and someone who cares not about the popularity of a server or it's monetary status, still think it's not good to do. That being said, I see why you do it and it removes any suspicion I had that you do it for malicious reasons. I apologize if I seemed inflammatory or rude at all, in my previous post.

    No need to apologize! I understand all of your points and they're completely valid. Just an all-in-all shitty situation in my opinion.

     

    If you want to discuss it further in PM at any point, feel free to reach out on Discord (cdeacon#6401) if you have it installed or you can just PM me here.

×
×
  • Create New...