Jump to content
 Share

Roy

Anycast Update - New POP Hosting Provider, Upgrades, and More!

Recommended Posts

Hey everyone,

 

I just wanted to announce some exciting news regarding our Anycast network!

 

As some of you probably know, I have plans to upgrade our Anycast network. For more information regarding the expansion itself along with what we're planning to do to mitigate (D)DoS attacks, I'd recommend reading this and this thread. The goal of this expansion is to strengthen our (D)DoS protection and acquire better hosting providers which'll result in:

 

  • Better routing.
  • More network capacity to mitigate higher volume (D)DoS attacks.
  • Better support.
  • And more!

 

I will now discuss the updates themselves individually.

 

New Hosting Provider and Dallas POP (GSK)

Recently, I've came into contact with the owner of Gameserverkings (GSK), Renual, and we've been talking a lot the past couple of weeks. At first, we were talking about Anycast and (D)DoS protection in general. However, Renual felt we'd benefit from having them as a part of our network and I agreed.

 

After talking to Renual for the last two weeks, it was pretty obvious he knew what he was doing and he has given me A TON of useful information in regards to (D)DoS mitigation/attacks, Anycast, BGP, routing, ISPs, fiber optics, network equipment, and more. He also mentioned his plans for GSK and I'm honestly really excited for their future! When putting into account his mindset, talents, and goals for GSK, I would not be surprised if they became a very big hosting provider in the future. I'm honestly surprised I haven't heard of them before, but I don't think many gaming communities in the Source Engine world know of them. That'll probably be changing, though.

 

Renual offered us a trial run for a machine with the following specs:

 

  • AMD Ryzen 5 3600 @ 3.6 GHz (4.2 GHz turbo).
  • 32 GBs of DDR4 RAM.
  • 500 GBs NVMe.
  • $75.00/m (we aren't paying for this yet until the trial run is over and if we are satisfied).

 

These specs are probably more overkill with our current setup, but I want to give this a shot and if things run okay, I think I'm going to take a quality over quantity approach with the network and use GSK in our primary locations such as Dallas, TX and London, UK. They also have VPSs coming up, so if need to be, we can downgrade to a VPS which should still be powerful since it'll include dedicated cores IIRC.

 

They also plan to expand into other locations in the future that includes NYC and Chicago. Once they have these locations available, we'll be likely purchasing machines or VPSs in those locations as well.

 

GSK's DDoS protection is great and Renual has showed me proof of them being able to mitigate larger DDoS attacks (e.g. attacks that exceed 100 million PPS). In fact, a majority of the malicious traffic that goes through this new POP will more than likely be filtered by GSK's equipment. However, with BGP Flowspec support (which will be discussed below as a subject), any attacks BiFrost detects can be pushed to the upstreams. Therefore, GSK's upstreams and equipment could take care of those detected attacks.

 

Another pro is if need to be, we can order special equipment (e.g. switches and so on) and get colocation space within GSK. I don't personally believe we'll need that for now, but it's something to keep in mind for the future. We may also request a replacement with any part in our server (e.g. if we wanted to, we could upgrade our server to a 10 or 40 Gb NIC and GSK has 10+ Gb links available for an extra ~$50/m).

 

Finally, I feel having direct contact with Renual will be beneficial to us in the case an issue arises or just generally speaking.

 

Anyways, the setup of the new POP hasn't been as smooth as I'd like, but it was to be expected. Since GSK's filtering is pretty strict (for protection reasons), we had some issues getting the POP up and working with Compressor and so on. Thankfully, all of that appears to be fixed now and Renual was very quick to resolve these filtering issues. I started announcing the POP this morning and everything appears to be working okay so far. I did have to make some changes to Compressor (that include the new filters) since the function to get the CPU count was actually returning 32 cores instead of 12, so when we were looping through CPUs on our handshake BPF map, after CPU #12, there was garbage data which was resulting in unexpected behavior. Probably something worth mentioning to the Linux kernel devs, haha.

 

If you see any issues connecting to our game servers, please let me know!

 

Also, here's a cool pic of our current Dallas POP server in the GSK rack :)

 

Na1r0wcfBC.jpg

 

P.S. These servers are liquid-cooled :)

 

And I just wanted to give a big shout out to Renual for being so helpful recently!

 

BGP Flowspec Support In Bifrost

I just wanted to briefly go over our plans to implement BGP Flowspec support into Bifrost. What BGP Flowspec will allow us to do is push policies to our upstreams via BGP. What this means is if we detect an attack with BiFrost and push it to our upstreams, they will filter the attack instead of the POP server/Bifrost. This'll result in less load on the POP server and if an attack exceeded our POP's link, this would stop the attack from saturating our link since the upstreams would filter the malicious traffic instead.

 

At the moment, this is only supported on GSK I believe, but given we'll be likely expanding with them in the future, this is something worth implementing in my opinion. Unfortunately, you can't push policies that include payload matching via BGP Flowspec. The policies operate more like an ACL. Therefore, if we're getting hit by a larger (D)DoS attack that gets through the GSK filters, but each malicious packet's source IP isn't spoofed randomly each time, we can push a policy to filter that source IP via BGP Flowspec and GSK's upstreams will accept that policy along with GSK's equipment.

 

I believe we'll also be able to push policies including TTLs (time-to-live) and more as well which gives us a bit more control than a typical ACL. Just sadly payload matching isn't supported within the policies themselves.

 

DPDK Over XDP

Another subject I wanted to briefly go over is the possibility of us using DPDK to process/drop packets instead of XDP in Bifrost. While XDP is still great and has its own pros over DPDK (e.g. being able to use kernel functionality and just easier to learn/understand/work with at times since the network stack is already implemented), DPDK is still technically faster than XDP when dropping packets. If you want some benchmarks, feel free to look at this. Note that these are from 2018, but I do believe these are still relevant today (I've talked to a few experienced people about DPDK vs XDP recently and they all stated DPDK still has an edge over XDP). I do plan on making my own benchmarks in the future with XDP vs DPDK once I purchase my home servers. However, that'll take some time.

 

The only cons with DPDK are that we'd need to implement the network stack ourselves and it is also a lot more complicated. For example, regarding the first con, we'd need to implement functionality ourselves to handle ARP packets in DPDK.

 

With that said, I tried looking into DPDK in the past, but their documentation is VERY long and time consuming. Please take a look here if you're interested. The third section alone feels like a book to read! There may be easier ways to go about getting familiar with DPDK though, so that's something I'll be looking into.

 

Overall, while DPDK is complicated and time-consuming, I do believe it's something we should try to do with Bifrost because it'd give us more control and it's also probably one of the fastest libraries to use when dropping (malicious) packets.

 

HellsGamers And Our Anycast Network

In the past two months or so, I offered HG a few Anycast IPs from GFL's IP block (92.119.148.0/24) to use for a trial run to see if they were interested in utilizing our Anycast network. I got told that they were indeed interested a couple weeks ago after the trial run and they decided to purchase their own /24 IPv4 block that is currently being pointed to our ASN. All the LOAs and BGP sessions are pretty much set up at this point and the only thing left to do is reconfigure BIRD (our BGP daemon) on each of our POP servers to announce their IPv4 block and add IP allocations to Compressor for them to use. I'm hoping to get this completed by early next week.

 

HG will also be paying us monthly for this usage. I will not disclose the cost, but I believe this'll help with our Anycast expansion and allow us to spend more on it which'll result in better DDoS protection, routing, network capacity, and more!

 

As of right now, HG will be the only community outside of GFL that'll utilize our network. I don't have any plans at the moment to accept other gaming communities. However, it's something I will consider in the future once my life starts improving and I'm less stressed since I feel like taking on the responsibility of more communities will add a lot more stress onto me (I'm already very stressed, to be honest). But then again, it's something I'm willing to do once the network is mostly expanded and my life situation starts improving (so I'll be able to put in work without feeling super stressed already).

 

Overall, these changes are very exciting in my opinion! I know a lot of you will not understand what a lot of the above mean due to the technical aspect, but the things we're accomplishing with the network is very impressive. Especially for just a gaming community. We still have a lot more to do, but I'm confident we'll get there :)

 

If you have any questions, please feel free to reply.

 

Thank you!

Share this post


Link to post
Share on other sites


Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now


×
×
  • Create New...