Just a quick update on something cool - we have recently taken delivery of a brand new dedicated machine from GSK:

• Ryzen 9 7900X CPU
• 128GB DDR5 RAM
• 2TB NVMe SSD

This machine will replace two of our existing public server machines in Dallas, GS15 and GS18, and will be labelled GS-US3 (following the new machine naming convention). This is quite the upgrade from GS15/GS18, both running an Intel i9-10900K CPU!

A selection of servers will be transferred to this new machine and will benefit greatly from the performance improvement:

• CS:GO Surf #1 Beginner, #2 Novice, and 24/7 Utopia
• CS:GO KZ #1 Easy & #2 Expert
• CS:GO Bhop #1 & #2
• GMod CWRP
• GMod TTT Anarchy
• GMod TTT Rotation
• GMod Prop Hunt
• TF2 2Fort Vanilla
• TF2 2Fort Improved #1 & #2
• TF2 Hightower and Hightower #2
• CS:S Zombie Escape

This performance upgrade is much needed for quite a few parts of GFL, for instance, this should allow CS:GO Surf to increase the total player capacity on some of the bigger servers.

We will be migrating these servers over time as we can, most migrations only result in a total downtime of 10-15 minutes, so these can be done pretty quickly! This new machine is spec'd with quite a lot of memory and disk, so we should be able to use it for future expansions and will have quite some space remaining once the initial migrations are completed. (We will have ~8 CPU threads unallocated once these migrations happen!)

The 25th was GFL's 12-year anniversary! It is almost surreal to think about how far GFL has come over the years, through its various ups and downs (of course, it wouldn't be the GFL we know and love without all the drama), and just how many generations of people have come and gone through all this time. GFL continuing to thrive after all these years wouldn't be possible without you all - thank you all so much for being here and being supportive!

Last year, we celebrated GFL's anniversary by having @annoying furry resign from her Director spot, fun! This time, I would like to do something a little less entertaining - reflect on some of our achievements over the past year, and talk about a few plans for the coming year!

-  2 0 2 2  -

Of course, I need to start this section with xQc's visit to GFL a few months back during one of his streams:

On that note, we've managed to achieve quite a few things this past year! This won't be an exhaustive reflection on the whole year by any means, but just a short list of things we did:

• We hosted our first ever major event, the GFL CS:GO Surf Tournament, in June of 2022! We had a prize pool of $2606.69 USD that was distributed to the Top 3 players in the event, making it the biggest tournament Surf has ever seen:
  Spoiler

   

• Speaking of events - our Christmas Santa event, where we gave out a wide variety of gifts, had over 200 participants! We gave out about 95 Steam keys, 31 months of VIP, 10 months of Supporter, and many more in-game rewards such as 160,000 GMod CWRP credits and 80+ CS:GO Surf player models!
  • We also hosted our first Minecraft event in a very long time, the Factions event!
• We sponsored two CS:GO Mapping Contests, the Mapeadores CS:GO Zombie Escape and the GameBanana CS:GO Jailbreak mapping contests!
• The forums have received a ton of much needed love from @Liloz01 throughout this year! Here are some statistics on the forums for 2022:

• Our infrastructure has evolved quite a bit over the past year, both in terms of cost saving and in capacity. We significantly upgraded our web infrastructure and have grown our Private Server infrastructure due to the program's growth! On the other hand, we have saved the community quite a bit on monthly expenses by consolidating Physion's infrastructure with GFL's core existing hardware.

-  2 0 2 3  -

2022 was a good year for us in terms of overall stability and growth! We were able to try a few things we had never done before, and we intend on pushing forward with the many things we've learned. Here are a few of our rough plans that we intend on pursuing over the course of the year:

• To keep up with the growth of the Surf Private Servers program, we're currently awaiting delivery of a dedicated machine in London! This won't be a traditional expansion as we're starting out only with Surf private servers, but this may change down the line.
• We aim to release GFLStore within the first half of this year, if all goes well! This new store will allow for players to donate for specific packages and premium perks on many of our servers.
• Physion is planning on major plugin upgrades as well, mostly pertaining to quality of life fixes as well as some new features for players of all ranks.
• The Events Team is hard at work planning and preparing for more large scale events, similar to the aforementioned Minecraft event!
• Last, but certainly not least, we will be working on our next iteration of the CS:GO Surf Tournament, with the goal of it being even bigger than the last one! Keep your eyes peeled, we will have more information later in the year. 😄

We'll be sure to post regular updates as we progress through our plans for the year! Once again, a huge thanks to everyone that has supported us all this time, we wouldn't be here without you all. 💖

Oh, also, we're running a small giveaway for 3-months of VIP and some Discord Nitro in our main Discord server to celebrate: https://discord.gg/njWJkR55Z3

Just wanted to share that we recently picked up a new dedicated machine to replace one of our existing web machines - the new machine is spec'd with a Ryzen 9 5950X (32 threads), 128 GB of DDR4 RAM, and 2 x 2TB NVMe SSDs! Compared to our previous machine from OVH, which was equipped with a Ryzen 5 5600X (12 threads) and 64GB DDR4 RAM, this is an upgrade several times more capable than the current machine.

WEB-4 is home to a lot of our key web services, such as our MariaDB database server dedicated to game servers and other vital services like the SurfAPI and our Discord bots. This machine upgrade should allow us to allocate far more resources to services than before, which means we should see significant performance improvements across the board along with being able to handle far more load.

We've acquired this machine in Los Angeles from Tempest Hosting, who were kind enough to introduce us to their network with a very good deal. Unfortunately, I'm unable to share the details on pricing of this machine, so all I can really say is that it is a very good price that should fit right in with our current monthly expenses on infrastructure. They've been very good to us throughout this and we're excited to see where this takes us!

Also, seeing 32 available CPU threads in htop never gets old:

Yesterday, we scheduled a maintenance period to begin migrating services over to the new machine. @annoying furry was able to get quite a few things across, however, we will need more time in the near future to move the remaining services - announcements will be made in Discord when this is scheduled to happen. Once we have everything migrated, we'll be canceling the old machine from OVH.

An Update on Migrating PHYSION-1

In a past update, we mentioned our plans to merge Physion's Unturned servers with GFL's main game server infrastructure. Our goal at the time was to have @Nick custom build a bunch of required plugins so we didn't need to depend on third-parties and any potential DRM issues when moving, however, after speaking to GSK (our main hosting provider), we have figured out a way to instead move the IP address from the Physion machine directly to one of our primary game server machines in Dallas. This means we no longer need to wait for every plugin to be remade and can instead proceed with migrating as soon as possible!

@Salad and I will soon be working together on planning out the migration. Once completed, we should see significant performance improvements on all the Physion Unturned servers along with cutting $150 USD from monthly expenses.

---

Short update compared to the previous one, but that's mostly because myself and many others have been super busy with university and other IRL obligations - now that we're heading into the holiday season, everyone should have far more time and hopefully we can get some cool things going! The stuff mentioned in this update is part of preparations to pursue the previously mentioned region expansion and a few other things, hopefully I'll have more exciting stuff to share down the line. 😄

As always, feel free to send me a message on Discord if you have any questions or ideas for things we should do!

I have a few cool updates I'd like to share with you all today. These updates involve two new machines we have acquired, our planned obsolescence of PHYSION-1 (Physion's dedicated machine for game servers), and progress on a potential region expansion!

It's been a while since one of these has been done. I will be working on changing that, so you should be hearing from me a lot more with giant (and probably boring) walls of text summarizing what GFL is up to! As always, if you have any questions, please don't hesitate to leave a comment or message me on Discord.

A New Machine Naming Convention

Before we get into the new dedicated machines, Aurora and I sat down and discussed revamping our current naming scheme for dedicated machines. We felt that it was misleading (with machine numbers starting from a weird index instead of reflecting the actual count of machines) and that we needed to have the convention be far more consistent across the board along with showing useful regional information. Thus, I'm covering a brief overview of the proposed changes as it'll give good context to why some references later in this thread are with different names than ones we have usually used.

With that, we have decided on a simple naming convention that addresses the aforementioned issues. I will spare you all the exact schema and instead show some of the potential machine name changes:

- GS15 -> GS-US1 (GS = Game server, US = United States, 1 = Machine count based on identifier.)

- WEB1 -> WS-CA1 (WS = Web server, CA = Canada, 1 = Machine count based on identifier.)

The machine counts will not be incremented when a machine is replaced, rather, machine replacements will count as replacements for the same count identifier. For example, a replacement of GS-US1 will have the identifier remain as GS-US1 instead of changing to GS-US2. This should resolve an inconsistency we have with our current naming scheme, such as there being a gap between GS16 and GS18 (fun fact: there is no machine in between!)

This naming schema is still a work-in-progress as we try to iron out some larger details to ensure it doesn't need to be worried about in the future. That said, it will take a while before we slowly ease our infrastructure to the new scheme and we're not too worried about rushing it - however, we will be using it immediately for any new machines we get.

Upgraded Machine for GS-US4

(In this context, GS-US4 is our fourth game server dedicated machine in the US, provided by NFO)

We have recently acquired a new dedicated machine to replace our current NFO machine, replacing our Xeon E-2288G CPU (5GHz turbo) with the newer Xeon E-2388G (5.1GHz turbo) and faster DDR4 RAM, up to 3200 MHz from 2666 MHz!

This should be a decent upgrade for our CS:GO Zombie Escape server, which is our primary resident on this machine. We're still in the process of setting it up and ensuring a smooth transition, with Vauff and Snowy hard at work preparing the move for the ZE server, but we should hopefully have everything ready to go soon! 

The new machine will cost us $279.99 USD/mo, which is an increase from our previous machine's rate of $249.99 USD/mo.

Our Very First Dedicated Machine for Private Servers!

For context, GFL began offering CS:GO Surf Private Servers on April 3rd this year after quite a lot of demand from the community. The private servers aim to provide players and their friends an opportunity to privately grind out world records on our collection of maps without needing to worry about other players, while simultaneously supporting GFL.

Since we started the project, we have been using third-party hosting providers to host our game servers in our supported locations. While this worked great for us in the short term as we could support more locations without needing a significant presence around the world, we very quickly ran into issues keeping up with the demand and automating required processes. It was clear that we needed to ramp up our efforts to ensure everyone gets a chance at this, so we decided to set some goals and try pushing onward so we could start stocking locations with our own hardware.

I'm happy to announce that we've finally hit those goals, and we have placed the order for our very first machine dedicated solely to private servers! We are getting this machine from GSK and it will be set up with the rest of our current infrastructure in the Dallas data center. It's a moderate machine designed to keep costs low while ensuring it can support a fair number of servers, with a 12-thread AMD Ryzen 5 3600X that should comfortably support 10-12 low-pop CS:GO servers without any issues. We will be paying very close attention to how we allocate CPU resources/threads to each server, just like we do for our public game servers, to ensure that we make the most of the CPU without going overboard.

As a neat bit of insight, here is a snippet of our spreadsheet that shows how we allocate resources on our GS machines at the moment, with GS-US4 as an example (it's a new machine, so it's very empty!):

Another great benefit of having our own hardware for this is that we can work on automating more of the process. This takes a lot of stress off of everyone responsible for it, our "ideal" situation would be one where the system basically maintains itself. As of right now, our automation consists of a lot of GitLab CI/CD pipelines that allow @Dini to make mass changes and updates to the servers with one single move, as well as a Discord bot to help with automatically setting up and deploying servers with all needed files:

GSK has been brilliant with ensuring that we can get any future machines with exact/similar specs activated in Dallas within short time-frames, in the case we ever need to deal with any future surges in demand. 😄

Planned Removal of the PHYSION-1 Machine

We mentioned this in a Council meeting previously, and I'm happy to say that we are getting much closer to our goals to eventually drop the PHYSION-1 machine from our line-up. Part of this includes migrating all of our Unturned servers to our existing game server machines, which will result in a significant performance upgrade and save us approximately $150 USD/mo in expense!

As of right now, @Nick is hard at work re-creating our premium plugins from the ground up to replace the ones we currently have. This will give us full control over the quality of experience we offer along with freeing us from DRM restrictions currently in place with a lot of these plugins. Once this is complete, we should be able to begin the process of slowly migrating servers over to the new machine.

This move is a win-win for us, as our Unturned servers receive a huge performance boost (moving from an AMD Ryzen 5 PRO 3600 @ 4.1 GHz to an Intel i9-10900K @ 5.3 GHz) along with saving GFL at least $150 USD/mo in expense. You can read more about this in a document we shared during a past Council meeting here: https://docs.google.com/document/d/1cg2GpqOpxK4nN367tTmZVU-_JDiMVeXfDbVEoip5CIk/edit?usp=sharing

Our Potential Region Expansion - Australia!

I am beyond excited to announce that we will officially be looking into expanding into Australia! We're working on planning out our approach over the long term, and we are looking to commit to the expansion for at least 6 months - 1 year (we'll confirm all of this as we plan stuff out over time).

For our initial approach, we're probably going to be entering the region with a line-up of CS:GO servers, as the game is very popular in the region. CS:GO Surf will likely be our flagship going into the region! That said, our intention here is to start small and grow as we build up in the region, so we likely will be starting with only a few specific servers that have higher chances of succeeding.

We're already working on setting up our foundations here, with work being done to set up new web infrastructure to host an extended wing of the Surf API network that currently powers our many CS:GO Surf servers around the world. We will likely be implementing Cloudflare's Geo-Steering load balancing features to ensure that the network performs to high standards and operates on a good level of redundancy (i.e, one region's web services going down will not halt operations as traffic will be automatically routed to the other regions).

To start, we have set up a CS:GO Surf invite-only server in Sydney, hosted on our Private Server infrastructure. We have been inviting players to come and try out our Surf experience and begin competing in our diverse community for about a month now, with over 35 people on the list at the moment with many more to come! This was done to help promote ourselves in the region as well as gauge interest from players - that said, things seem to be going rather well, as we already have a few private server orders lined up in Sydney while they wait for us to plan out our full expansion with public servers.

The promotional surf server is also a great way for us to benchmark performance with our accessory services and the SurfAPI network as we plan out our expansion.

We are still evaluating potential hosting providers for dedicated machines in the region. That said, our base of operations will likely be in Melbourne and/or Sydney - this is where our infrastructure is likely to be located. This should cover a good part of Oceania, with great coverage of both Australia and New Zealand.  

This is still a work-in-progress, and we're hoping to have many more updates to come! We would like to take it slow this time instead of rushing it, this way we'll have more time to spend on ensuring that everything goes well. Our next steps will be to continue evaluating what we will need to make this happen, along with what servers we should try our hand at in the region.

P.S: If you're from the region and want to help out/be a part of this, feel free to shoot me a message!

---

As mentioned near the start of this post and in the previous Council meeting, I will be doing more of these! I haven't been doing these at all when I really should have been, so please let me know if anyone has any ideas to make these better and/or more interesting for everyone.

One thing I would like to note is that these won't take away from any updates provided in the Council meetings, so we will continue to host updates on everything GFL at those meetings so everyone can ask questions live.

For a while now, people may have noticed servers being password-protected randomly, showing 255/255 player count, and the host names were cut off. We had this issue last year as well, but it had to do with our Sydney edge server and was corrected here.

I wasn't sure what was wrong this time, so I started a packet capture on our Redis server that handles distributing cached responses to all of our edge servers to send to our clients. I stopped the packet capture as soon as the issue was reported again and the packet capture file contained 2 million+ packets. I then filtered out all the packets besides the one containing the reported server's IP (in this case, was CS:GO Surf Timer #1, 92.119.148.18) and found this:

These were the packets being distributed from Redis to the other edge servers meaning one of our edge servers reported this response to our Redis server. It didn't take me long to figure out our Chicago edge server was the one (and only) edge server reporting these malformed responses. I then ran uname -r on all of our edge servers and every edge server besides the Chicago edge server reported Linux kernel 5.4.0+ whereas Chicago was still running on 4.18. The reason the server was running such an outdated kernel was due to this being our oldest edge server to date and still running Ubuntu 18.04. I decided to create a new edge server in Chicago, set it up, stop announcing/destroy the old server, and start announcing the new one. This one is running a similar kernel to the others and Ubuntu 20.04. Things appear to be running okay so far, but the malformed A2S_INFO responses issue should be fixed.

This also wouldn't have only impacted Surf Timer. This would have impacted all of our servers. Basically, when a server's A2S_INFO cache time was expired and the next A2S_INFO request went through our old Chicago edge server, there was a high chance it would have been malformed. It didn't do this for every single request, but I'd say a good chunk of them.

If you start seeing malformed responses again, please let me know!

13/05- Removed several emotes that weren't being used, added MonkaS. Changed where Dyno sends his welcoming message, added some custom commands, removed how Manager+ could get away with breaking the rules.

Around two - three hours ago, we started seeing many networking issues and slowly players were timing out along with services our servers connected to.

When I first started investigating this, it appeared completely random packet types were dropping somewhere. When I mean random, I mean completely random. For example, I was able to connect to our game servers fine, but my A2S_PLAYER challenge/requests were being dropped somewhere since that query never responded for me. However, this query worked on some of our other servers as well in Dallas, TX. With that said, I performed packet captures for our services our game servers connected to that were failing, we were seeing packets in both directions. This further proved it was random packet types being dropped somewhere.

I then started talking to our hosting provider in Dallas, TX because I wasn't seeing any issues with our NYC servers. From what I saw, certain packets weren't making it back from our hosting provider which are sent directly from our Anycast ranges on our machines (92.119.148.0/24 and 185.240.217.0/24). We were working together for a thirty minutes to an hour on this and then we discovered our BGP session on our GSK POP server wasn't active. This must be active in order to send traffic out as our Anycast ranges due to uRPF policies (basically we have a switch that our GSK POP server and game server machines plug into, since the POP server is announcing the IP ranges via BGP, our game server machines are able to send traffic out directly).

Now, an hour or so before all of this happened, I fixed an issue with our Squad server here and this was related to the Anycast network as well. At first, I suspected the changes I made to somehow be breaking stuff (even though the changes I made had nothing to do with the type of random packets being dropped). However, I enabled debugging within the XDP/BPF program which prints to the Linux kernel trace pipe when a packet is dropped (along with the unsigned 32-bit integer in network byte order, port, etc). The packets that weren't making it back weren't being filtered by the XDP/BPF program. Therefore, I thought things breaking soon after was just a coincidence. I also tried reverting back to the previous versions on the POP servers I was routing to and see if there were any changes. Unfortunately, I wasn't routing through the GSK POP server, so this didn't allow me to see the issue quicker.

Anyways, the packet processing/filtering software running on our POP servers is managed in a private GitHub repository. This is how I update everything, etc. The reason the BGP session broke is because I forgot to whitelist the BGP neighbor IP address from GSK. The whitelist code wasn't pushed to the GitHub repository which is why when I reset the GitHub repo and updated it with the Squad fix, that code was lost. The reason I didn't push this code to that repository is because the repository has 8 - 9 other server owners I've tried helping mitigate (D)DoS attacks for and gave them the filters I made for GFL. I felt pushing code for the neighbor address like that wouldn't be secure. We had to do the same thing on our Vultr POP servers as well, but they use a local link IP address (to save IP addresses I guess, I don't know), so pushing the Vultr neighbor whitelist code to the GitHub repository wasn't a big deal in my opinion. After I whitelisted the GSK neighbor address on the GSK POP server, the BGP session became active again.

When the BGP session goes down, all traffic is supposed to be dropped out of our IP ranges (I know this is also a single-point-of-failure, but I'm working on a solution for that as well where we'd have fallback BGP sessions). The reason not all traffic was dropped is because we had a lot of issues with sending traffic out as our Anycast ranges months ago because we weren't plugged into the same switch as our GSK POP server (some of you probably remember) and GSK tried manually accepting certain traffic from us. Therefore, some traffic was still allowed through even while the BGP session was down. Honestly, if all traffic got cut off, I would have probably found the issue quicker considering we could have easily narrowed it down. It was the fact that 10 - 20% of the traffic was coming through and random packets were being dropped on all connections.

Unfortunately, we don't have a dev environment for our Anycast network besides running a single forwarding server on my home server and testing out filters, etc. Something like this we wouldn't have been able to test. This is something I will be considering in the future though, but it'd cost us a bit of money which is why I haven't done so yet.

In the end, this was my major mistake and I do apologize (don't blame anybody else besides me for this). I hope the above went into detail on my mind process regarding it. I'm going to look into having fallback BGP sessions to eliminate this single-point-of-failure.

All servers have been up for around an hour or so now.

Thank you for your time and understanding.

I just wanted to provide two big updates.

Europe Update

I know many of you have seen updates on this subject for months now, but I have something finally exciting to announce. Please read the last update here if you haven't already.

The timeout issues we've been running into has been resolved with our hosting provider replacing a carrier in this location. I've confirmed this by staying on the server for 3+ hours and had others play for a while as well without any issues (it was normally timing me out for an hour after 5 - 10 minutes of playing time before). However, while this main issue is resolved, our hosting provider has stated they are making changes in the next two weeks. Therefore, I don't want to fully launch the Europe/UK division until this is sorted out. Things should be working regardless, though, and I'm still getting details on what changes they'll be making to see if it would impact us.

At the very least, we will be able to setup/test our Europe/UK game servers in the next two weeks and then once things are sorted completely, we can launch (this will be server/division-specific of course, as well).

Anyways, as of right now, ~4 - 6ms additional latency overhead is added to connections/packets to our UK servers because outgoing traffic is going through the closest POP to the UK game server machine instead of out directly. To resolve this issue, we'll need to fix the UK POP and the BGP session. Since our hosting provider has routers in the UK now, we will be able to do this easily and could possibly get this done by tonight (if not, definitely by the end of next week).

New Machine

Since we've been having issues with our GS18 machine in regards to CPU clock speeds and whatnot. I've decided the best option to avoid unnecessary downtime (from trying to fix it within the BIOS), we would just get a replacement machine. With this, we were offered a machine with different specs, but the same price. The processor is stronger in this new machine than the old machine and the RAM is also clocked faster.

The old machine (GS18)'s processor was the Intel i9-10900K with 10 cores and 20 threads. The new machine's processor is the AMD Ryzen 7-5800X with 8 cores and 16 threads. While we won't have as many cores to play with, the single-threaded performance on the AMD processor will be faster which matters most for most game servers. With that said, the old machine at 64 GBs of RAM clocked at 2133 MHz whereas the new machine has 32 GBs of RAM clocked at 3600 MHz. Initially, we were hoping the RAM could go clocked at 4000 MHz, but for some reason, even though the RAM states it's supported along with the manufacture confirming so, we've ran into stability issues and our hosting provider had reports of the same issue running 4000 MHz RAM before. Therefore, we've downclocked the RAM to 3600 MHz.

I am currently running a stress test that'll go on for 24+ hours to make sure we aren't running into stability issues. With that said, I've requested a KVM to be attached to the machine as well because I want to see if there's anything I can tune in the BIOS in regards to the CPU and memory performance before using the machine publicly. 

I setup a CS:S DM server with 32vs32 bots (close quarters) similar to what I did here with GS14 (which ran the Intel i9-9900K) and GS19 (the new machine) handled it beautifully at 70 - 80% CPU.

It never left from 66.(6)7 FPS!

To conclude, that's it for now. I'm pretty exciting for this and once I get done testing everything, I'll let everybody know.

Now it's time to start developing (D)DoS miitgation/prevention systems 🙂 

I am making this thread to announce our plans to expand into Europe along with a game called Squad and some other small things. I've talked this over with @FrenZy and they were in full support of this.

Before continuing, I just wanted to briefly go into why I'd like to overview these expansions. As some of you know, a year and a half ago I took a complete back-end role in the community due to the amount of energy required for maintaining the Anycast network. While maintaining the network has been great, I feel I haven't been able to have much fun in GFL since then because everything I do has been technical and most people don't understand 99% of the things I'm doing as well.

When I saw the opportunities to expand into Europe and Squad, I felt it was a chance to do something fun again. While this won't decrease the amount of stress I have (in fact, it could increase it a tad bit), I do think it'll be fun and I don't plan to play a hectic role. I simply just want to overview these expansions and ensure they're heading in the right direction. I will be doing my best to guide the people we assign to the servers and train them as well. I think this gives us a good opportunity to improve our internal training program within GFL as well.

Europe Expansion

The first expansion I want to talk about is Europe. For those that do not know, we used to have many Europe servers back years ago and they all did very well for the most part. In fact, many game modes we've tried in the past that didn't succeed in the US did succeed in Europe. These include Call of Duty 4, CS:GO Zombie Mod, Killing Floor 2, and more. We had to cancel our Europe machine at the time due to financial problems we were having in 2016. Here's the old thread I made regarding GFL's Europe expansion back in 2014.

Anyways, Europe arguably has better success rates for game servers than the US. This is because the US is oversaturated with game servers in my opinion. Therefore, I feel this Europe expansion could potentially double GFL's population if it succeeds. With that said, a lot of our successful staff (present and past) have came from Europe due to our Europe servers at the time and in general.

To start off, I'd like to setup servers in game modes that don't require many admins. This is because we likely will be short of staff at the start of these game servers. However, with the success rates I'm expecting, I don't think they will be shortly staffed for long. Obviously, we may go through a couple rough patches with admins at first since we'll likely be looking for admins ASAP (e.g. we may pick admins that aren't great quality), but this will balance itself out after a month with good management (these are risks you have to take when expanding and we've done this in the past).

Some servers I was thinking about expanding into that didn't require many admins were CS:GO Surf Timer, CS:GO Bunny Hop, GMod Prop Hunt, GMod Murder, and a few others. However, that list isn't certain since I haven't talked to our Division Leaders and Server Managers yet. If you have any game mode suggestions that won't require many admins, please feel free to reply with your suggestion(s)

In regards to the technical aspect, I ordered a new /24 IP block today for our Europe expansion (this will cost us $50.00/m and will give us another 256 IPs to play with). We'll be using this with the Anycast network and at the moment, this block's geo location on IP2Location (what the Steam Master Server uses) is set to California, US. I will be putting in a request to have this changed to London, UK since that's where our physical servers will be hosted at in Europe. This change can take up to 30 days since IP2Location updates these on the first of every month.

I still think we'll be able to get decent population with the IP block's geo location set to California, US. However, we'll see more population when it's properly set to London, UK.

The physical game server machine we'll be ordering to start from GSK in London, UK will include the AMD Ryzen 5 3600 @ 3.6 GHz (4.2 GHz turbo), 32 GBs of DDR4 RAM, and a 500 GBs NVMe drive. This machine will be temporary and will cost us $84.99/m. If the servers we place on the machine work out, we'll be looking to get a permanent machine that'll either include the Intel i9-10900K or the AMD 5 5600X when it releases (the fastest processor in regards to single-threaded performance).

The machines should be ready in the next one - two weeks. With that said, I'll need to setup BGP sessions and provide LOAs for the new IP block to GSK and Vultr (our POP hosting providers) which should take around a week to implement (probably sooner, lol).

As of right now, I'd say the timeline for when the Europe expansion will be ready to go will be in the next 3 - 4 weeks or sooner. I'll probably throw some unofficial game servers on the machine to start to see if they receive any population and release them officially if they succeed. With that said, @FrenZy and the Division Leaders + Server Managers will be working with us to see which game modes we can try out in Europe.

Squad Expansion

The next expansion I want to talk about is Squad. We actually tried expanding into Squad back in the Summer of 2018. While the server did get full a couple of times (80/80) which can be seen below, unfortunately the server didn't work out long-term.

I feel there wasn't enough time spent into populating the server though and we never ended up keeping the players we had around. This is something easily fixable if we have dedicated players/management which I'm expecting to have this time around.

Squad is a fairly easy game to manage since the servers are mostly vanilla and the game itself recently had a full release (it was in early-access beforehand). According to my community server population tracker website, Squad has the best player to server ratio over any other game tracked. Please take a look here for example. For every Squad server over 0 players, they have an average of 51 players this past month per server. This is a great amount of average players and compared to other games like CS:GO and TF2, is very high! I do believe we can offer perks as well such as reserved slots for Supporters/VIPs and more.

The first thing I need to check is whether our license from 2018 is still valid for game servers in Squad. Squad will require a license for all servers to use. If it is not valid, I will email their team regarding this request. The next thing I need to do is implement filters into Compressor so players routing to our Europe/Asia POPs will be able to connect to our Squad servers. I'm hoping this doesn't take long and they support a handshake sequence. If not, I can always make exceptions for Squad servers in our filters. I'm not expecting this to take long.

With that said, we will not be doing A2S_INFO caching with our Squad servers.

At the moment, I have an individual who I believe will be helping and heavily involved with this expansion, but since it isn't 100% confirmed yet (@FrenZy and this individual need to talk), I will not announce them until the next update.

The ETA on this expansion is probably within the next two - three weeks or sooner.

If you have any interest in helping with our Squad expansion, please let me know! We could use all the help we can get and considering we're a lot more popular now than mid-2018, I do believe we'll have a better chance at prepopulating our Squad servers.

London POP Replacement

Since we're getting a machine in London from GSK for our game servers, I figured this'll also be a good time to replace our Vultr London POP with a GSK machine. This was on the to-do list to begin with and will offer us better protection from (D)DoS attacks.

This process should be simple and straight forward, but when I'm implementing the change, I'll let everybody know more details.

Conclusion

That's really all for now. It's actually nice making a public-facing post again, I've missed it (I used to post threads like these weekly lol)

I just want to make clear I'll be overviewing these two expansions only and ensuring they're moving in the right direction. If you have any suggestions for other servers in GFL or the community as a whole, I'd recommend reaching out to our Server Managers, Division Leaders, and Directors.

If you have any questions, please feel free to reply.

Thank you for reading!

I just wanted to provide another update to the recent performance issues we've been suffering from. This is more of a follow-up to this update, but I wanted to make it a separate thread since we will be performing maintenance this upcoming Tuesday (January 19th).

Machines Overloaded + New Machine Issues

Our game server machines have been overloaded recently, especially GS16 which hosts our Rust servers specifically. We purchased another machine with the Intel i9-10900K and tried moving some servers to that machine for load-balancing last week. However, we ran into even more complicated issues regarding RPF policies (filters put in-place to prevent spoofed traffic originating from our hosting provider's network). We ran into these issues because with our current setup, technically speaking, we're "spoofing" as our Anycast network when sending traffic from our game server machines directly (we aren't announcing the Anycast IP ranges via BGP on our game server machines themselves and there's no real way to do that right now that doesn't conflict with our current Anycast setup). Resolving the issue is very tricky and it has become a huge pain to deal with without us purchasing a switch for GFL (read below). Unfortunately, we had to revert the move last week after finding it too complicated to get things working with the new machine.

At first, we were hoping we'd be able to setup a BGP session on our game server machines and announce our specific Anycast IPs allocated to the game server machine without traffic from the Internet also routing to them. This would allow us to send out as the Anycast IPs since they're being announced and RPF would succeed, but unfortunately, there's no way to prevent the Internet from routing to the game server machines in this case and this wouldn't work because we have an IPIP setup meaning we're expecting incoming packets to the game server machine to be in IPIP format. If we were to go with a solution like this, we'd literally need something to encapsulate each incoming packet into IPIP and even then, we'd also need to keep track of the internal IPs which would just be a mess (so more overhead since we'd need to encapsulate incoming packets, bigger packet sizes by 20 bytes each, and a solution to map internal IPs properly). This isn't ideal in my opinion.

A solution our hosting provider (GSK) offered was purchasing a switch for GFL and then we'd plug all of our game server machines into this switch along with our Dallas POP (which is with GSK). Since the Dallas POP is announcing our Anycast range already and is plugged into the same switch as our game server machines, this would allow us to send traffic out as our Anycast network directly and not have to worry about RPF policies dropping them since our Anycast IP ranges are being announced on the switch by our Dallas POP. I believe this makes most sense in my opinion and is the best option for us. This way, we wouldn't have to worry about not being able to spoof as our Anycast network from our game server machines.

I've decided to accept this solution. With that being said, 30 or so seconds of downtime will be required for each game server machine and our Dallas POP. The 30 seconds or so of downtime would be network-related and this is assuming the move is smooth. I'd like to move our least popular machine first to this switch and ensure everything is working properly. Afterwards, we will move the others. We're likely going to move the new game server machine and Dallas POP first to ensure things are running okay on them and we're able to send traffic out as our Anycast IPs that aren't associated with existing game servers (since they would fail anyways due to them being directed towards a different switch). For game server machines in production use, I'm planning for the maintenance to start after my current job finishes on Tuesday (January 19th) which is 4 PM CST. Therefore, the timeframe will be 4 PM - 7 PM CST on Tuesday most likely. There's a high chance we'll finish sooner, but I just wanted to provide a longer time period in case. We'll likely try getting the Dallas POP and new game server machine moved earlier if possible on Tuesday just so I can do testing on those while on my breaks or on lunch at my current job.

GS16 Heating Issues + Poor Performance

Last night, I was informed that many of our modified Rust servers (running additional addons and whatnot) were experiencing poor performance issues along with Vanilla from time to time. I started looking into this and immediately saw an issue. The processor was downclocking from 4.9 GHz to ~4.4 - 4.6 GHz. This was a huge issue and after trying to receive the CPU temperatures via the sensors command, the temperatures seemed okay mostly (~85 - 90C). However, our hosting provider confirmed this was due to thermal throttling. Therefore, those temperatures were most likely inaccurate.

I thought I had stress-tested this machine specifically before putting it into production use. However, after thinking about it, I believe I only stress-tested the GS14 and GS15 machines (GS15 having the same specs as GS16). The GS15 machine is performing fine and we ran additional stress testing (without impacting performance) to ensure it can receive equal amount of load to GS16 without thermal throttling occurring. I believe we were trying to get GS16 up as fast as possible because of performance issues we were having on our old machines, but I should have still stress-tested this machine before production use and I apologize for this.

Our hosting provider believes having the thermal paste reapplied to the CPU will resolve this issue and I agree. Therefore, at some point (unknown when we'll do this), we will most likely be taking out the machine and reapplying the thermal paste. This would result in 30 minutes or so downtime assuming everything is smooth. I'm not sure if I'd want to do stress-testing after the machine comes back up to ensure it can handle high loads without thermal throttling and this would require more downtime unfortunately unless if we want server performance being heavily impacted during this time. I still have no ETA, but since the switch maintenance on Tuesday should allow us to move servers to the new machine without issues, I don't believe this is urgent. Moving servers to a new machine will lower the load on GS16 which'll result in cooler CPU temperatures and the CPU being clocked at 4.9 GHz on all cores.

Once I have an update on the GS16 maintenance specifically, I'll let everybody know.

In the meantime, I believe we had our Rust 10x server's tick-rate set to 20 instead of 30 to see if that helps with the performance issues at higher player counts for now. A lower tick-rate should result in less CPU consumption from the server itself. Therefore, it should have improved performance at higher player counts even while the CPU is being throttled to 4.4 - 4.6 GHz. The drawback to this is the server itself won't be as smooth as before (since it's calculating less ticks per second), but hopefully it isn't really noticeable. This is all a temporary solution anyways unless if nobody sees a difference running at 20 instead of 30 ticks per second.

Conclusion

I understand the frustration recently in regards to poor performance. We saw a huge spike in players recently as stated in the other update post and we did try moving servers to a new machine last week, but ran into complicated issues. The truth is, we have a very unique setup since we own the network itself and while this comes with many pros, it also makes things a lot more complicated (e.g. having to find ways to obey RPF policies and whatnot). Thankfully, we'll be doing things a lot differently with the new packet processing/forwarding/filtering software we'll be making for the Anycast network, but that's still in development and quite complicated as you can see in this post. The new software will make it so we don't have to obey RPF policies since we won't be sending out as our Anycast network directly, thankfully.

Overall, we'll be having maintenance on our game server machines and Dallas POP on Tuesday, January 19th (2021). This will result in ~30 seconds network downtime for each game server machine and the Dallas POP assuming all goes well. We're going to try moving our Dallas POP and new machine to our new switch at first to ensure things are working. Once things are confirmed working with the new machine and POP, we will be moving the rest of our game server machines (GS14, GS15, and GS16) to the switch from 4 - 7 PM CST (when my current job ends for the day). Afterwards, @Aurora will be able to offload servers to the new machine without running into issues assuming everything works correctly. This alone should resolve the GS16's thermal throttling issues mentioned above since it'll lower the CPU load/temps and therefore, the CPU will be clocked at 4.9 GHz on all cores again.

We'll also want to schedule a time at some point to take out our GS16 machine and reapply thermal paste. This would likely result in at least 30 minutes downtime, but isn't urgent right now since we're going to be moving servers after the Tuesday maintenance which'll result in the machine load going down and CPU being clocked at 4.9 GHz like normal. More updates on that will be announced at a later time.

Server performance should improve after the machines are load-balanced. If servers are still experiencing performance issues after the machines are load-balanced and it is confirmed that both the CPU is clocked at 4.9 GHz on all cores and that it isn't a network-related issue, the server itself is most likely unoptimized. At that point, managers of the server need to look through its addons to see if there's anything consuming too many CPU cycles, change the server's tick-rate, adjust the player count, or more.

I also want to state this is in no way our hosting provider's fault (GSK). It's really good that they have RPF policies (not enough hosting providers do this in my opinion), it's just unfortunately our setup is unique and currently relies on spoofing traffic out as our Anycast network. I really appreciate GSK's help in regards to this as well since a lot of hosting providers wouldn't allow you to spoof traffic at all.

If you have any questions, please let me know and once again, I apologize for the inconvenience recently. I hope this post clears things up and allows people to see what issues we're running into while trying to resolve everything. I'm trying to be as transparent as possible in regards to these issues and our solutions to them.

Thank you for your time.

This year represents GFL's 10 year anniversary (January 25th, 2011 is when it was founded) and we're already off to a great start!

I just wanted to provide a small update to everybody regarding two topics.

Recent Surge Of Players + New Machine

Since New Years, we've seen a big surge in players on GFL's servers. In fact, today, we beat our all-time record of players online on all of GFL's servers at the same time (thank you to everybody who made this possible!).

Our all-time record was 1299 players on at the same time back in mid-2015 and this included a TeamSpeak 3 server at 100+ users! More accomplishments can be found here for those interested!

We've seen a 200 - 300+ peak player increase just in the past two days (mostly from our Rust servers). With this happening, we've noticed our machines becoming overloaded. While our servers run with great hardware (which are detailed here), we're starting to run into performance issues due to the amount of CPU all of the game servers are consuming in combination.

@Aurora asked if we could purchase another US machine and I agreed. Therefore, I am going to be talking to GSK and requesting a new US machine (located in Dallas, TX like the rest of our US machines).

I believe I'm going to request another machine with the Intel i9-10900K processor just like our GS15 and GS16 machines unless if the newer AMD series is available with GSK which have better single-threaded performance according to the benchmarks I've seen (single-threaded performance matters the most for game servers since most are mostly single-threaded).

Once I have an update on this, I will let you know.

I just wanted to also apologize for the poor performance on the affected servers. I was, at least, not expecting this big of a player surge so suddenly, but regardless, I'm going to try getting this new machine ASAP so we can mitigate the issues. In the meantime, we may try load-balancing servers between machines to help with performance/load.

Small Update On Europe Expansion

We're going to be expanding into Europe and the plan can be found here! 

This process has taken longer than I was expecting and I apologize for the delay. Towards the bright side, IP2Location has set our EU IP's block geo-location (185.240.217.0/24) to London, UK which was a huge step in this process (so the Valve Master Server will show the EU servers quicker for EU players and whatnot due to it using geo-location). All we're waiting for now is the EU POP to be setup with GSK. The EU machine itself is setup and has been ready for some time. However, we're unable to send traffic out as our EU IP block (185.240.217.0/24) and this is because the GSK network blocks these packets due to them not being a part of the RPF policies. We must announce the EU IP block on the GSK POP in order to resolve this issue. I am waiting for GSK to correct the BGP session issue we're running into at the moment and GSK said it shouldn't take much longer.

I'm hoping we can get this going this weekend or next week at some point!

If you have any questions about the above, feel free to reply!

Thank you!

Around two hours or so, a majority of our servers experienced a full outage. Specifically any servers on our GSK machines.

This was due to stricter RPF filtering GSK started enforcing earlier tonight. This basically made it so traffic originating as our Anycast IP range from our GSK machines were being dropped. With that said, our GSK POP wasn't able to send IPIP traffic to our game server machines and vice versa which broke A2S_INFO responses and players routing through our GSK POP wouldn't be able to send packets to any of our servers.

 This change was brought to my attention a couple of weeks ago or so, but preparations were more complex since it involved additional BGP sessions to my understanding. I wasn't aware they were making the change today. As a temporary solution, they've lifted restrictions for our specific Anycast IPs on our GSK machines and also allow IPIP traffic to certain destinations from the GSK POP. The permanent solution will be setting up the needed BGP sessions to allow us to obey the RPF filters without issues. However, this will likely time more time and will be more complex.

I'm going to talk to GSK and ensure communication for things like this is established better in the future since we weren't aware of the specific date and the only communication we had was it probably happening in a couple of weeks without BGP session details, etc.

I do apologize for the inconvenience.

If you have any questions, please feel free to reply.

How will the new team work?

In the beginning, we only had a few Discord servers, and managing only a few discords with our prior system worked very well for a while, but now we have many different Official GFL Discord servers, the majority of them pertaining to a certain Division (E.g.. CS:GO, GMOD, Rust, TF2). 

While the structure of our Discords have drastically changed, our Moderation team did not. The team remained mostly the same, with very few changes to its own structure. Each of our Official Discords will now have a dedicated team of 2+ moderators to tend to their server, and only their server. 

We will also have Senior Moderators, their responsibility is to ensure that Moderators are behaving appropriately, making sure that Moderators are following our Moderation Etiquette, and they will also be moderating globally in both of our Forums and Discords. This is expanded on below. 

Localization

One of the main concerns of the previous team was that they “didn’t understand” the culture of all of the Discords that they moderated, and that our team was "spread too thin". For this reason, we have decided to implement moderators that know the communities they will be moderating. We’ve seen great success with the CWRP Moderators, and this is our effort to replicate that on a much larger scale. This also reduces the barrier of activity in all of our Discords, thus allowing the community to have more mods where we need them. Instead of everyone being required to moderate all of our Official Discords, they will only be tasked with one.

Filling in the Gaps

A team as large as this will need to be overseen by not only it’s Team Leader, but also a few Senior Moderators. These Senior Moderators will be experienced and assist the Moderation Team Leader in training and maintaining the team. Senior Moderators will be available to consult on situations that other Moderators are unsure of.

What else has happened?

• The former Moderation Team was relieved of their duties on 11/14/20 in favor of the new team we’re building. https://gflclan.com/forums/topic/65809-announcement-regarding-the-moderation-team/
• We understand the way that this was handled was brash, and we have since issued an apology to all former Moderators for not notifying them beforehand of their team being dissolved. In the future, we will take consideration to inform staff members of situations such as this before announcing it publicly.
• The Global Discord ruleset and Discord Punishment guidelines have been updated and expanded upon.
• We released these modified rules/guidelines so that everyone can provide some feedback before implementation, it will be up for two days. The thread can be found here: https://gflclan.com/forums/topic/66399-feedback-moderation-rules-and-punishment-guidelines/
• The Moderator Etiquette guidelines for all of our Moderators have been updated, adding a list of punishments for all team members and guides on how to handle criticism.
• On 11/17/20, we began the process of selecting members for the new team. To do this we are working with Division staff to find people with interest, as well as looking over Moderation applications that already exist.
• On 11/26/20, with assistance from Division Leaders, I chose the divisional-specific Discord Moderators. The new Moderators in each division can be found at the bottom of this post.
• We are still working to appoint a new Team Leader.

New Division-Specific Discord Moderators:

CS:GO

QueenKill1o1

Quad

ColaCanFan

Infra

SQUIRRELY

GMod

Jarm

Plum

Aurora

Duck

Rust

Hatehim222

MadTurkey

Bue

Ace7

TF2

Lachaine

Cuasi

Scott

Some of the moderators will also be moderating Main Discord. 

Additionally, applications will finally be dealt with - within the next few days, be prepared to see some new Forum Moderators, some former Discord Moderators returning.

Around 1:50 PM CST today, our GS14 machine went down who is hosted with GSK. I was at a GP appointment at this time and @Aurora was looking into this issue. Unfortunately, we didn't have KVM access. Therefore, we asked GSK to reboot the machine which resulted in it coming back online. However, once we tried launching a game server, it went down again.

When I got home, GSK was in the process of getting a KVM attached to the machine. At 4:51 PM CST, the KVM was successfully attached and I was able to connect. However, I received a call and had to go for another ten - twenty minutes. So investigation with the KVM didn't start until 5 - 5:10 PM CST.

Last night, we updated our control panel which included a daemon update that was applied to all of our game server machines. For some reason, the new configuration file on GS14 was setting our control panel daemon to use the entire host machine as its network instead of the separate bridge it creates internally (172.18.0.0/16). Basically what this does is exposes the interfaces on the host machine within the game server's Docker containers and network namespaces. Now, without our Anycast setup, this probably would have worked fine. However, exposing each container to all the interfaces on the machine defeats the purpose of isolating each container's network and would be less secure.

The reason the machine was going down is because we use Docker Gen to deploy an IPIP tunnel for our Anycast network to use (since Compressor forwards traffic from our POPs to our game servers via IPIP formatted packets, we need to setup an IPIP tunnel/endpoint in each container with the remote host and internal IP). When running our control panel in "host" mode, Docker Gen would execute these commands on the main host interface instead of inside each game server's network namespace. Two commands this ran were deleting the old default gateway and replacing it with the new gateway which pointed towards the IPIP tunnel. Since this was being ran on the host, it removed the main machine's gateway (which went to GSK's router of course from the NIC) and replaced it with the IPIP tunnel (which wasn't even configured at this time). This resulted in the machine losing network connectivity and could not be restored until you either restart the machine or attach a KVM, log in that way, remove the old default gateway, and add the correct default gateway (which is what we did).

To resolve this issue, firstly, we modified the control panel's daemon config to set up its own bridge which are linked to the veth pairs our control panel adds when spinning up a game server within Docker containers. Afterwards, restart the daemon and if the machine is still offline, you can check the default route via the ip route command. If the default route is set to the IPIP tunnel Docker Gen was trying to set up, you will need to delete the default route via ip route delete default and add the original default route via ip route add default dev <Main Interface> via <Gateway IP>.

After this, we were able to spin up servers, but they weren't getting network connectivity. This was due to my IPIPDirect program here and when it first started up (on machine boot), it's supposed to get the default gateway's MAC address here and here. I believe it was starting up when the default gateway was set to the IPIP tunnel (so the MAC address was probably 00:00:00:00:00:00 or something virtualized/fake). This resulted in outbound game server traffic not being routed out properly. Simple restarting the application via systemctl restart IPIPDirect resolved this issue since it was able to save the correct destination MAC address (the default gateway's) and all game servers were online again.

What's not yet clear is why the machine didn't go down after the panel update last night. My suspicion is we updated the panel and it started using the internal network initially. However, the config file was still specified to use the host network. Therefore, at some point, the panel restarted today and that's when everything went down. @Aurora and I will most likely be digging through the logs to see if we can find anything that would have caused this.

With that said, we're going to be looking into purchasing our own KVMs for GS14 along with our future game server machines with GSK. This is just so we have our own dedicated KVMs we can access at any time and don't have to rely on GSK manually attaching a KVM (which can be used by other clients which is why it took longer than usual this time).

I understand this post is more technical, but figured I'd give this information for those interested and who knows, maybe if I'm not here, this'll help others like @Aurora with what to look at and so on in the case this happens again.

I also wanted to say thank you to @Aurora for looking into this and contacting GSK while I was at my GP appointment  This would have taken longer if she hadn't started looking into it.

I apologize for inconvenience and thank you for your patience regarding this. If you have any questions, please feel free to reply!

This thread is being made for documentation purposes.

At 10:16 AM CST today, our Dallas POP with GSK lost network connectivity. Since the POP at the time was still announcing our IP range, this resulted in players being briefly disconnected until our IP range stopped being announced. For the next thirty or so minutes, the POP was announcing the IP range on and off which caused connectivity issues for users routing through our Dallas POP. This was because the machine was still online and BIRD was running which resulted in our range still being announced (it only needs to communicate with the neighbor address), but external traffic into the POP failed.

Players routing to a POP not directly affected by this incident may have not been able to see our game servers in the server browser as well during this time, but should have been able to connect to them directly. This is because our game servers in Texas route through the Dallas POP and it would use that POP server to cache A2S_INFO responses. Since this POP server went down, the A2S_INFO responses never made it back to the Redis server causing the players to never receive A2S_INFO responses (the server wouldn't show up in the server browser in this case). With Bifrost, we'll have a better method of handling A2S_INFO responses/caching which should resolve this issue from occurring again if the main POP goes down.

The Dallas POP's issue was corrected 45 minutes or so after the initial incident. However, our GS14 machine (with GSK as well) went down soon afterwards. This resulted in downtime for game servers on this machine. It turns out it ran into similar issues regarding the network as the POP server.

GSK (Gameserverkings) released an official statement here stating what caused the outage.

I apologize for the inconvenience and thank you for understanding. I don't believe this'll happen again and I do appreciate the transparency from GSK on this situation.

If you have any questions or concerns, please feel free to reply.

Thank you.

I just wanted to announce some exciting news regarding our Anycast network!

As some of you probably know, I have plans to upgrade our Anycast network. For more information regarding the expansion itself along with what we're planning to do to mitigate (D)DoS attacks, I'd recommend reading this and this thread. The goal of this expansion is to strengthen our (D)DoS protection and acquire better hosting providers which'll result in:

• Better routing.
• More network capacity to mitigate higher volume (D)DoS attacks.
• Better support.
• And more!

I will now discuss the updates themselves individually.

New Hosting Provider and Dallas POP (GSK)

Recently, I've came into contact with the owner of Gameserverkings (GSK), Renual, and we've been talking a lot the past couple of weeks. At first, we were talking about Anycast and (D)DoS protection in general. However, Renual felt we'd benefit from having them as a part of our network and I agreed.

After talking to Renual for the last two weeks, it was pretty obvious he knew what he was doing and he has given me A TON of useful information in regards to (D)DoS mitigation/attacks, Anycast, BGP, routing, ISPs, fiber optics, network equipment, and more. He also mentioned his plans for GSK and I'm honestly really excited for their future! When putting into account his mindset, talents, and goals for GSK, I would not be surprised if they became a very big hosting provider in the future. I'm honestly surprised I haven't heard of them before, but I don't think many gaming communities in the Source Engine world know of them. That'll probably be changing, though.

Renual offered us a trial run for a machine with the following specs:

• AMD Ryzen 5 3600 @ 3.6 GHz (4.2 GHz turbo).
• 32 GBs of DDR4 RAM.
• 500 GBs NVMe.
• $75.00/m (we aren't paying for this yet until the trial run is over and if we are satisfied).

These specs are probably more overkill with our current setup, but I want to give this a shot and if things run okay, I think I'm going to take a quality over quantity approach with the network and use GSK in our primary locations such as Dallas, TX and London, UK. They also have VPSs coming up, so if need to be, we can downgrade to a VPS which should still be powerful since it'll include dedicated cores IIRC.

They also plan to expand into other locations in the future that includes NYC and Chicago. Once they have these locations available, we'll be likely purchasing machines or VPSs in those locations as well.

GSK's DDoS protection is great and Renual has showed me proof of them being able to mitigate larger DDoS attacks (e.g. attacks that exceed 100 million PPS). In fact, a majority of the malicious traffic that goes through this new POP will more than likely be filtered by GSK's equipment. However, with BGP Flowspec support (which will be discussed below as a subject), any attacks BiFrost detects can be pushed to the upstreams. Therefore, GSK's upstreams and equipment could take care of those detected attacks.

Another pro is if need to be, we can order special equipment (e.g. switches and so on) and get colocation space within GSK. I don't personally believe we'll need that for now, but it's something to keep in mind for the future. We may also request a replacement with any part in our server (e.g. if we wanted to, we could upgrade our server to a 10 or 40 Gb NIC and GSK has 10+ Gb links available for an extra ~$50/m).

Finally, I feel having direct contact with Renual will be beneficial to us in the case an issue arises or just generally speaking.

Anyways, the setup of the new POP hasn't been as smooth as I'd like, but it was to be expected. Since GSK's filtering is pretty strict (for protection reasons), we had some issues getting the POP up and working with Compressor and so on. Thankfully, all of that appears to be fixed now and Renual was very quick to resolve these filtering issues. I started announcing the POP this morning and everything appears to be working okay so far. I did have to make some changes to Compressor (that include the new filters) since the function to get the CPU count was actually returning 32 cores instead of 12, so when we were looping through CPUs on our handshake BPF map, after CPU #12, there was garbage data which was resulting in unexpected behavior. Probably something worth mentioning to the Linux kernel devs, haha.

If you see any issues connecting to our game servers, please let me know!

Also, here's a cool pic of our current Dallas POP server in the GSK rack

P.S. These servers are liquid-cooled

And I just wanted to give a big shout out to Renual for being so helpful recently!

BGP Flowspec Support In Bifrost

I just wanted to briefly go over our plans to implement BGP Flowspec support into Bifrost. What BGP Flowspec will allow us to do is push policies to our upstreams via BGP. What this means is if we detect an attack with BiFrost and push it to our upstreams, they will filter the attack instead of the POP server/Bifrost. This'll result in less load on the POP server and if an attack exceeded our POP's link, this would stop the attack from saturating our link since the upstreams would filter the malicious traffic instead.

At the moment, this is only supported on GSK I believe, but given we'll be likely expanding with them in the future, this is something worth implementing in my opinion. Unfortunately, you can't push policies that include payload matching via BGP Flowspec. The policies operate more like an ACL. Therefore, if we're getting hit by a larger (D)DoS attack that gets through the GSK filters, but each malicious packet's source IP isn't spoofed randomly each time, we can push a policy to filter that source IP via BGP Flowspec and GSK's upstreams will accept that policy along with GSK's equipment.

I believe we'll also be able to push policies including TTLs (time-to-live) and more as well which gives us a bit more control than a typical ACL. Just sadly payload matching isn't supported within the policies themselves.

DPDK Over XDP

Another subject I wanted to briefly go over is the possibility of us using DPDK to process/drop packets instead of XDP in Bifrost. While XDP is still great and has its own pros over DPDK (e.g. being able to use kernel functionality and just easier to learn/understand/work with at times since the network stack is already implemented), DPDK is still technically faster than XDP when dropping packets. If you want some benchmarks, feel free to look at this. Note that these are from 2018, but I do believe these are still relevant today (I've talked to a few experienced people about DPDK vs XDP recently and they all stated DPDK still has an edge over XDP). I do plan on making my own benchmarks in the future with XDP vs DPDK once I purchase my home servers. However, that'll take some time.

The only cons with DPDK are that we'd need to implement the network stack ourselves and it is also a lot more complicated. For example, regarding the first con, we'd need to implement functionality ourselves to handle ARP packets in DPDK.

With that said, I tried looking into DPDK in the past, but their documentation is VERY long and time consuming. Please take a look here if you're interested. The third section alone feels like a book to read! There may be easier ways to go about getting familiar with DPDK though, so that's something I'll be looking into.

Overall, while DPDK is complicated and time-consuming, I do believe it's something we should try to do with Bifrost because it'd give us more control and it's also probably one of the fastest libraries to use when dropping (malicious) packets.

HellsGamers And Our Anycast Network

In the past two months or so, I offered HG a few Anycast IPs from GFL's IP block (92.119.148.0/24) to use for a trial run to see if they were interested in utilizing our Anycast network. I got told that they were indeed interested a couple weeks ago after the trial run and they decided to purchase their own /24 IPv4 block that is currently being pointed to our ASN. All the LOAs and BGP sessions are pretty much set up at this point and the only thing left to do is reconfigure BIRD (our BGP daemon) on each of our POP servers to announce their IPv4 block and add IP allocations to Compressor for them to use. I'm hoping to get this completed by early next week.

HG will also be paying us monthly for this usage. I will not disclose the cost, but I believe this'll help with our Anycast expansion and allow us to spend more on it which'll result in better DDoS protection, routing, network capacity, and more!

As of right now, HG will be the only community outside of GFL that'll utilize our network. I don't have any plans at the moment to accept other gaming communities. However, it's something I will consider in the future once my life starts improving and I'm less stressed since I feel like taking on the responsibility of more communities will add a lot more stress onto me (I'm already very stressed, to be honest). But then again, it's something I'm willing to do once the network is mostly expanded and my life situation starts improving (so I'll be able to put in work without feeling super stressed already).

Overall, these changes are very exciting in my opinion! I know a lot of you will not understand what a lot of the above mean due to the technical aspect, but the things we're accomplishing with the network is very impressive. Especially for just a gaming community. We still have a lot more to do, but I'm confident we'll get there

If you have any questions, please feel free to reply.

Thank you!

Tonight, our Anycast network experienced a major issue for our servers located in Dallas, TX under Nexril. Around a couple hours ago, I started announcing a new POP server in Dallas, TX which is a part of the major Anycast expansion that I'll be making a thread about tomorrow or whenever this issue is fixed.

What started occurring was all game servers under Nexril weren't responding to A2S_INFO queries. This resulted in the server being thrown off the server browser. At first, I thought this was due to the AF_XDP part of Compressor breaking on the new POP server (which is a common reason to why A2S_INFO wouldn't be working and we've ran into this in the past). However, after disabling the POP server and confirming the BGP session was inactive, the issue continued to occur. I confirmed these machines were routing to one of our Vultr's POPs and not the new one. Therefore, I was honestly stumped on what the issue could be.

After troubleshooting, I discovered the servers completely went down for the Nexril machines when I disabled my IPIP Direct program. This meant all game server packets being sent back from the Nexril machines to the closest POP server (our Vultr Dallas POP) weren't making it back. This also would explain why A2S_INFO broke initially because my IPIP Direct program is configured to send A2S_INFO responses back through the POP only for caching purposes (by this line).

So this had me confused because when running an MTR to the Anycast network from the Nexril machines, it was receiving ICMP replies fine. With that said, I tried disabling BGP on the Vultr Dallas POP so the machines routed to the Chicago POP. However, the same issue occurred. This is when I needed to dig even deeper and unfortunately, since XDP isn't debug-friendly (what Compressor, our packet processing software uses), I did have to stop Compressor on our Dallas POP multiple times after announcing it to the network again (so the machines routed to this POP again, etc).

When I did this, I used my Packet Flooding tool here to continuously send UDP and TCP packets back to the Anycast network every second (the closest POP which is Dallas, TX) from the Nexril machine. When Compressor was disabled and I had a TCPDump running, it WAS able to see the standard UDP/TCP packets. However, when I disabled the IPIPDirect program on the Nexril machine, Compressor on the Dallas POP, and ran a TCPDump for any packet coming from the game server machine on the Dallas POP, it ONLY saw those standard TCP/UDP packets. It DID NOT see the IPIP packets being sent back. I also ran a TCPDump on the game server machine catching all the IPIP packets it was trying to send back to the POP and there were many of them. None of them made it back to the Dallas POP server via TCPDump.

It appears one of the routers/hops from the Nexril machines to the Dallas POP are dropping/filtering our IPIP packets. I honestly haven't seen such a strange issue before on our Anycast network.

I've opened up a ticket with Nexril and also recorded a video of all my troubleshooting showing there's clearly an issue with IPIP packets. I'm waiting for a response from them.

In the meantime, what I did to get the servers up and running is run the IPIPDirect program and also include A2S_INFO responses so it'll send those out directly instead of through the POP. A2S_INFO caching will be disabled due to this, but once Nexril and us are able to figure out the issue, I'll be rebuilding the IPIPDirect program to send A2S_INFO responses back through the POP again for caching purposes.

I do want to apologize for the inconvenience and downtime associated with our Dallas servers along with players routing through our Dallas POP since I needed to take that down at times for debugging purposes.

Once I have an update, I will let you know.

Thank you.

Council
Council is a new decision mechanism for high-level decisions. This will resemble a board of GFL that sets the direction and makes the final calls for large decisions. The amount of members of Council will vary, but it will be preferred to have an odd number of members since that makes voting easier.

The Council in its entirety will not be responsible for the daily operations. This will be handled by the Board of Directors who will always have spots in Council as well.The chairman of the board will be called an Executive Director who will also serve as the chairman of Council. This board will be elaborated in the next section.

Each well-established game in GFL will have a representative on the board. Whether a game is well-established or not will be decided by the Council. These will be elaborated in an upcoming section.

One or more non-staff members of the community will also take part in Council as Community Representatives.

Each Council member has equal say and a vote for each decision. If a vote is tied, the Executive Director makes the final call. The quorum of the Council will be half of the members.

Executive Director
The Executive Director is the chairman of the Council and the Board of Directors; this might sound very powerful at first, but it is important to remember that everyone has an equal say in the Council. The Executive Director just has more responsibilities to the Council and the Board of Directors. To list some of these:

• Ensure that the Council is fulfilling their duties and decisions are being responsibly made at an acceptable pace.
• Ensure that Directors are fulfilling their duties during daily operations.
• During daily operations, the Executive Director will handle the events that “slip through the cracks”.

Board of Directors
The Board of Directors are responsible for the daily operations of GFL under the direction of the Council. They are allowed to make decisions within their area without going to the entire Council, but they should never go against the direction set by the Council. Major decisions will be discussed with the Council beforehand.

A Director (excluding the Executive Director) is responsible for a specific area in GFL. The following areas exist at the moment:

• Communication
  The Director of Communication is responsible for the daily operations concerned with communication. Primarily this is the Discord servers and the forums.
• Tech
  The Director of Tech is the leader and spokesperson for the TAs. Furthermore, the Director will be responsible for the daily operations in the backend systems such as game servers, the network, web services, and so on.
• Divisions
  The Director of Divisions is the leader and spokesperson for the DLs. This entails ensuring the DLs are fulfilling their duties. Only the Council in its entirety can promote and demote DLs.
• Teams
  The Director of Teams is the leader and spokesperson for the Team Leaders. This entails ensuring that TLs are fulfilling their duties. The Director will be the interim leader of leaderless teams. The Council in its entirety must approve new teams or the removal of existing teams.

Areas will be added and removed as needed.

Division Representatives
In order to ensure that our divisions get proper representation in the Council, each well-established division (WED) has a spot in the Council as a Division Representative (DR). Each WED appoints a DL to be the DR in the Council. The Director of Divisions will act as their division’s representative if applicable.

The following divisions are considered well-established at the moment:

• CS:GO
• GMOD
• Rust

Community Representative
One or more non-staff members of the community will serve as Community Representatives in the Council. These are ideally people who often play on GFL’s servers or similar. These representatives will be elected by the community and serve as a member of Council for six months.

You must be recommended by at least one Council member before being able to run for Community Representative.

I just wanted to briefly address recent performance issues with our GS12 machine. Unfortunately, the machine's processor down-clocked a lot more than I expected at higher load. As of right now, it clocks to 4.3 GHz on all cores at ~50 - 60% load. This is because the processor is getting too hot running all cores at > 4.3 GHz and it needs to down clock. At 20 - 30% load, we see 4.6 - 4.9 GHz on all cores which is what I was hoping we'd get at 50 - 60%+ load. This is resulting in bad performance with servers on GS12 that include all of our Rust servers.

@Xy and I have been talking for the last few hours in voice and we've been talking to our current hosting provider in Texas. We believe we've found a solid solution and will be looking to order a new machine in the next day or so depending on the benchmarks we get back from the hosting provider (which are likely going to suit our needs). Our hosting provider also has better solutions coming at the end of the month that'll result in better cooling. Therefore, we'll be able to get higher clock speeds with the same processor. We will be using the same processor as the GS12 machine (the Intel i9-9900K). However, we'll have higher clock speeds at the same load GS12 is running at (likely > 4.6 GHz).

Once I have another update, I will let you know. I apologize for the poor performance as well. I wasn't expecting the machine to down-clock this much at 50 - 60% load.

Thank you for understanding.