Update On The New Filters With Compressor

[Roy 10,832 / 0]

Hey everyone,

 

I just wanted to provide a brief update on deploying filters to the rest of our POPs. For those that do not know, I was trying to deploy a version of Compressor I made that included filters that drop non-legitimate traffic. For more information, I'd suggest reading this post (under the "Temporary Solution"). With that said, this is all suppose to be temporary until Bifrost is made by Dreae and I.

 

Anyways, as of right now, we have the new filters deployed to all of our Asia and Europe POPs. It has been running stable on these POPs and I did try deploying the filters initially to our GSK POP as well. However, it ran into some sort of strange bug where Compressor would block my home IP until Compressor was restarted (this wasn't due to rate limiting either). This hasn't happened since I deployed my "V1" filters that doesn't include in-depth filtering (e.g. handshake validation, etc). Therefore, I'm assuming something specific is happening to my newest filters and the hardware on the machine since my newest filters are running stable on our Europe/Asia POPs. I was planning to debug this as well, but debugging in XDP is a pain and I just didn't have enough time at that moment to debug it.

 

With that said, I overlooked something when beginning to deploy the new filters to our NA POPs. I need to implement functionality that makes it so when a player's IP is validated via the handshake process to the game server, it sends the IP to a global list that each POP grabs from and whitelists as well. Without this, if a player is connected to our game server after the validation process to that specific POP, but their route changes while in-game to another POP server with the new filters, they will time out because they aren't added to the latest POP-specific handshake validated eBPF map. This makes it so they need to reconnect each time to get validated on that specific POP again. This is obviously a pain in some cases.

 

I was planning to implement this functionality into Compressor. However, this is starting to feel like I'm putting in A LOT of work for a temporary solution (the private GitHub repo that includes these filters has 150 commits from me alone on implementing functionality just for this specific branch). Therefore, I halted that plan for the time being until I decided if it was worth it or if we should just continue on with Bifrost instead. I've made the decision it'd be best to halt the temporary solution for now and continue planning out Bifrost.

 

With that said, I was notified by the GSK owner, Renual, recently on something really neat that he is able to do. When GSK's network detects an attack on our IP range, he is able to send policies to GTT (one of their carriers and our primary peer with Vultr other than NTT) that forces the single IP on the network to be offloaded via GTT. This makes it so all traffic goes through a scrubbing device with GTT and allows GSK's network to also mitigate the attack to my understanding. Since we use GTT primarily with our other hosting provider, this results in traffic going through GTT to our other POPs to go through this scrubbing device and GSK's network as well. This essentially allows GSK to mitigate malicious traffic on our network that would normally go through POPs that we don't have with GSK. This is pretty neat in my opinion.

 

All of our POPs in the US support GTT right now. So one thing I'd really like to do as a temporary solution is disable NTT entirely on our NA POPs. This way, all traffic to our NA POPs are going through GTT and this will allow GSK to mitigate the attacks and ingest all the malicious traffic being sent to our NA POPs via GTT offload.

 

Sadly, GTT isn't available for some POPs in Europe/Asia. However, since these are running my new filters, they shouldn't be forwarding the traffic to the game server machines anyways. So the worst that can happen is that specific POP gets overloaded, but it won't impact anybody else routing to different POPs assuming the attacker isn't smart enough to bypass the new filters I have deployed.

 

I will likely be disabling NTT as a peer for our NA POPs in the next couple of weeks. Otherwise, I can just pre-pend 1 - 3 hops to NTT as well which will make it so a lot less traffic goes through NTT. I will also replace our NA POPs with our V1 filters that'll include whitelisting Rust+ entirely. Therefore, players not being able to use Rust+ on our Rust servers at the moment (since they're routing through a POP with my "V1" strict filters currently) will be able to use Rust+

 

As mentioned, this is just a temporary solution since ideally, as an end-goal, I'd like each POP to be able to mitigate as much malicious traffic as possible (that's the entire point of using Anycast for (D)DoS protection, after all). Once we start expanding our Anycast infrastructure, I plan on introducing a blend of carriers including GTT, NTT, Telia, Comcast, and more. I'll definitely have to do BGP tuning for this, but this would result in better routing on the network if done correctly (players would see less latency to our network if their routes are optimized).

 

Also, as a side note, I made a script that notifies our Staff Discord when an attack is detected by GSK. Feel free to check the script out here. This script is temporary until I find out how to log these offloads with BIRD (sadly when logging everything right now, it doesn't include these offloads. I think this is due to the route not being imported correctly through BIRD). Once I find out how to log it with BIRD, I'll just make a Go program or something that watches the BIRD log file and detects entries for when GSK offloads one of our IPs to GTT and I'll make it send a message to the Discord Web Hook. We've been attacked twice already:

 

 

Thanks!