Roy 10,832 / 0 Report Post Posted October 30, 2020 Hey everyone, I am making this thread to announce our plans to expand into Europe along with a game called Squad and some other small things. I've talked this over with @FrenZy and they were in full support of this. Before continuing, I just wanted to briefly go into why I'd like to overview these expansions. As some of you know, a year and a half ago I took a complete back-end role in the community due to the amount of energy required for maintaining the Anycast network. While maintaining the network has been great, I feel I haven't been able to have much fun in GFL since then because everything I do has been technical and most people don't understand 99% of the things I'm doing as well. When I saw the opportunities to expand into Europe and Squad, I felt it was a chance to do something fun again. While this won't decrease the amount of stress I have (in fact, it could increase it a tad bit), I do think it'll be fun and I don't plan to play a hectic role. I simply just want to overview these expansions and ensure they're heading in the right direction. I will be doing my best to guide the people we assign to the servers and train them as well. I think this gives us a good opportunity to improve our internal training program within GFL as well. Europe Expansion The first expansion I want to talk about is Europe. For those that do not know, we used to have many Europe servers back years ago and they all did very well for the most part. In fact, many game modes we've tried in the past that didn't succeed in the US did succeed in Europe. These include Call of Duty 4, CS:GO Zombie Mod, Killing Floor 2, and more. We had to cancel our Europe machine at the time due to financial problems we were having in 2016. Here's the old thread I made regarding GFL's Europe expansion back in 2014. Anyways, Europe arguably has better success rates for game servers than the US. This is because the US is oversaturated with game servers in my opinion. Therefore, I feel this Europe expansion could potentially double GFL's population if it succeeds. With that said, a lot of our successful staff (present and past) have came from Europe due to our Europe servers at the time and in general. To start off, I'd like to setup servers in game modes that don't require many admins. This is because we likely will be short of staff at the start of these game servers. However, with the success rates I'm expecting, I don't think they will be shortly staffed for long. Obviously, we may go through a couple rough patches with admins at first since we'll likely be looking for admins ASAP (e.g. we may pick admins that aren't great quality), but this will balance itself out after a month with good management (these are risks you have to take when expanding and we've done this in the past). Some servers I was thinking about expanding into that didn't require many admins were CS:GO Surf Timer, CS:GO Bunny Hop, GMod Prop Hunt, GMod Murder, and a few others. However, that list isn't certain since I haven't talked to our Division Leaders and Server Managers yet. If you have any game mode suggestions that won't require many admins, please feel free to reply with your suggestion(s) In regards to the technical aspect, I ordered a new /24 IP block today for our Europe expansion (this will cost us $50.00/m and will give us another 256 IPs to play with). We'll be using this with the Anycast network and at the moment, this block's geo location on IP2Location (what the Steam Master Server uses) is set to California, US. I will be putting in a request to have this changed to London, UK since that's where our physical servers will be hosted at in Europe. This change can take up to 30 days since IP2Location updates these on the first of every month. I still think we'll be able to get decent population with the IP block's geo location set to California, US. However, we'll see more population when it's properly set to London, UK. The physical game server machine we'll be ordering to start from GSK in London, UK will include the AMD Ryzen 5 3600 @ 3.6 GHz (4.2 GHz turbo), 32 GBs of DDR4 RAM, and a 500 GBs NVMe drive. This machine will be temporary and will cost us $84.99/m. If the servers we place on the machine work out, we'll be looking to get a permanent machine that'll either include the Intel i9-10900K or the AMD 5 5600X when it releases (the fastest processor in regards to single-threaded performance). The machines should be ready in the next one - two weeks. With that said, I'll need to setup BGP sessions and provide LOAs for the new IP block to GSK and Vultr (our POP hosting providers) which should take around a week to implement (probably sooner, lol). As of right now, I'd say the timeline for when the Europe expansion will be ready to go will be in the next 3 - 4 weeks or sooner. I'll probably throw some unofficial game servers on the machine to start to see if they receive any population and release them officially if they succeed. With that said, @FrenZy and the Division Leaders + Server Managers will be working with us to see which game modes we can try out in Europe. Squad Expansion The next expansion I want to talk about is Squad. We actually tried expanding into Squad back in the Summer of 2018. While the server did get full a couple of times (80/80) which can be seen below, unfortunately the server didn't work out long-term. I feel there wasn't enough time spent into populating the server though and we never ended up keeping the players we had around. This is something easily fixable if we have dedicated players/management which I'm expecting to have this time around. Squad is a fairly easy game to manage since the servers are mostly vanilla and the game itself recently had a full release (it was in early-access beforehand). According to my community server population tracker website, Squad has the best player to server ratio over any other game tracked. Please take a look here for example. For every Squad server over 0 players, they have an average of 51 players this past month per server. This is a great amount of average players and compared to other games like CS:GO and TF2, is very high! I do believe we can offer perks as well such as reserved slots for Supporters/VIPs and more. The first thing I need to check is whether our license from 2018 is still valid for game servers in Squad. Squad will require a license for all servers to use. If it is not valid, I will email their team regarding this request. The next thing I need to do is implement filters into Compressor so players routing to our Europe/Asia POPs will be able to connect to our Squad servers. I'm hoping this doesn't take long and they support a handshake sequence. If not, I can always make exceptions for Squad servers in our filters. I'm not expecting this to take long. With that said, we will not be doing A2S_INFO caching with our Squad servers. At the moment, I have an individual who I believe will be helping and heavily involved with this expansion, but since it isn't 100% confirmed yet (@FrenZy and this individual need to talk), I will not announce them until the next update. The ETA on this expansion is probably within the next two - three weeks or sooner. If you have any interest in helping with our Squad expansion, please let me know! We could use all the help we can get and considering we're a lot more popular now than mid-2018, I do believe we'll have a better chance at prepopulating our Squad servers. London POP Replacement Since we're getting a machine in London from GSK for our game servers, I figured this'll also be a good time to replace our Vultr London POP with a GSK machine. This was on the to-do list to begin with and will offer us better protection from (D)DoS attacks. This process should be simple and straight forward, but when I'm implementing the change, I'll let everybody know more details. Conclusion That's really all for now. It's actually nice making a public-facing post again, I've missed it (I used to post threads like these weekly lol) I just want to make clear I'll be overviewing these two expansions only and ensuring they're moving in the right direction. If you have any suggestions for other servers in GFL or the community as a whole, I'd recommend reaching out to our Server Managers, Division Leaders, and Directors. If you have any questions, please feel free to reply. Thank you for reading! Share this post Link to post Share on other sites More sharing options...
Serenity 546 / 7,380 Report Post Posted October 30, 2020 · Hidden Hidden gmod deathrun : ) Share this post Link to post Achievements
Guest Saizy Report Post Posted October 30, 2020 · Hidden Hidden big pog thanks roy Share this post Link to post
Foe 29 / 1,339 Report Post Posted October 30, 2020 · Hidden Hidden Yes! I can’t wait for squad! I’ve been playing since the earliest days of their early access so it’s safe to say I’m pretty keen. why are you even reading this bub Share this post Link to post
Alexis 578 / 11,997 Report Post Posted October 30, 2020 · Hidden Hidden Awesome! @Roy Where is my personal PING?!?! Joking, love you though. Share this post Link to post
Jinzu 67 / 2,013 Report Post Posted October 30, 2020 · Hidden Hidden TTT EU. Lots of EU people in MC like playing but are held back by ping @Pachimo Share this post Link to post Achievements
Roy 10,832 / 0 Report Post Posted October 30, 2020 Also, here are some success images from our Europe expansion in 2014 - 2016 Share this post Link to post Share on other sites More sharing options...
Joshy 4,363 / 44,847 Report Post Posted October 30, 2020 Edited October 30, 2020 by Joshy · Hidden Hidden Gosh I think I remember writing about recycle admin applications to get admins on low pop servers, and people were very vocal. Glad to see some type of acceptance in some modest form probably 4 ish years later I think EU servers are a good move. Something I didn't like about those is that the admins and server managers were often totally separated. I think something that helps is just if it's the same game mode / same rules it would help if they had their permissions on both (US and EU) servers even though odds are they'll primarily play on one; also a good way to train new server managers if they have a successful counterpart. I don't know anything about Squad. Doesn't look bad. Edited October 30, 2020 by Joshy Share this post Link to post
Roy 10,832 / 0 Report Post Posted October 30, 2020 3 minutes ago, Joshy said: think EU servers are a good move. Something I didn't like about those is that the admins and server managers were often totally separated. I think something that helps is just if it's the same game mode / same rules it would help if they had their permissions on both (US and EU) servers even though odds are they'll primarily play on one; also a good way to train new server managers if they have a successful counterpart. I definitely agree with this. If we have two servers in different locations running the same game mode/configuration, I believe the admin teams should be shared. I've always felt that way but don't remember why some people had an issue with that. Share this post Link to post Share on other sites More sharing options...
Infra 714 / 12,386 Report Post Posted October 30, 2020 · Hidden Hidden Exciting! Discord: Infra#0001 | Steam: /id/infra- | GitHub: /1zc Executive Director & CS:GO Division Leader Server Manager of CS:GO KZ Share this post Link to post
positive 210 / 4,553 Report Post Posted October 30, 2020 · Hidden Hidden dope Share this post Link to post
Roy 10,832 / 0 Report Post Posted October 30, 2020 I've received the following reply from IP2Location regarding getting the new IPv4 block geo-located to London, UK Quote HI Christian, Thank you. We will review and update the range in the future release. Regards, Kim Loong IP2Location Customer Success Team Share this post Link to post Share on other sites More sharing options...
TheJitFace 699 / 18,157 Report Post Posted October 30, 2020 · Hidden Hidden nice jitticus Share this post Link to post
Bue. 664 / 0 Report Post Posted October 30, 2020 · Hidden Hidden Rust Division can't wait to get more involved! Share this post Link to post
L0st 106 / 2,791 Report Post Posted October 30, 2020 Edited October 30, 2020 by Daddy Issues · Hidden Hidden Yooo squad server?? Lets goooo cant wait for it. Always wondered why we had a squad sub section. Didnt know we actually had a squad server in 2018 Edited October 30, 2020 by Daddy Issues Share this post Link to post
Roy 10,832 / 0 Report Post Posted October 31, 2020 I've sent the two LOAs I compiled for our new IPv4 block to Vultr and GSK. Once the BGP session is completed, I'll need to reconfigure all of our POP servers to announce the new IP block. With that said, I've implemented functionality into Compressor with filters that allows Squad servers to run without any issues and I've been testing this most of tonight. This was more difficult than I expected because of how Squad/Unreal Engine does handshakes between the client and server. I may be able to get the Squad server up by tomorrow to at least begin testing. I've also sent an email to the Squad Developer team regarding a server license since it appears the old one we have is invalid. I'm hoping to get a response on that in the next few days. Thank you! Share this post Link to post Share on other sites More sharing options...
Roy 10,832 / 0 Report Post Posted October 31, 2020 We're looking for Squad admins! If you're interested or know somebody that would be, please fill out this form or have the individual do so. Thank you! Share this post Link to post Share on other sites More sharing options...
Angie 145 / 2,647 Report Post Posted October 31, 2020 · Hidden Hidden On 10/30/2020 at 8:24 AM, Bue. said: Rust Division can't wait to get more involved! Yes! Share this post Link to post
Roy 10,832 / 0 Report Post Posted February 25, 2021 European Expansion Update I know it has been months since I've made this thread and we haven't seen any progress on our European expansion. I do apologize for the delay. The truth is the biggest roadblock we were facing until a few weeks ago was unfortunately something outside of our control and our hosting provider has been very busy. We haven't been getting charged for any of the machines we have in Europe as well which is good at least. I'm happy to say that I have an update and we've made really great progress. We aren't all the way there yet, but we're close to having something that can be released. Our biggest roadblock for a long time was the BGP session on our new UK POP not coming up successfully and this was an issue with our hosting provider. Since the BGP session wasn't activated, we weren't able to announce our IP blocks. This is due to RPF policies that prevent spoofing of packets. If we weren't announcing the IP blocks via BGP on the same switch the UK game server machine was connected to, the outbound filtering would classify our traffic as spoofed and drop the packets (RPF policies are very good to have for hosting providers as well). This simply didn't allow us to send traffic back from the game server machine directly. While the above was unfortunate, there is a temporary solution to this that didn't become an option until a few weeks ago. We could simply send the outbound traffic through the nearest POP to the game server machine like our setup used to be before I made the IPIPDirect program here. Up until a few weeks ago, our hosting provider enforced strict policies on packets using the IPIP protocol which is what we needed. Therefore, this prevented us from using this solution until then. Now this temporary solution does come with a situational pro/con and a few other cons which can be found below. [Situational Pro/Con] Since outbound traffic is going back through the nearest POP server, it would stay within the Anycast network we have and not go back through the game server machine's network. This has the potential to either improve or worsen the latency depending on the route it takes. Staying within the network is always better since it's consistent and is the plan for the future. However, this specific implementation is not recommended due to the cons listed below. [Con] The latency from the game server machine to the closest POP is ~6.7ms meaning this will add an additional 6.7ms to every packet/client. [Con] This creates a single-point-of-failure meaning if the POP the game server machine is routing to goes down, no traffic will make it back to the clients. [Con] This puts more stress onto the POP server since all outbound traffic from the game server machine is going through it. I've put this temporary solution in-place until the BGP session issue is resolved which should be hopefully soon. After testing the solution, I am able to connect to the game servers for a few minutes or so, but run into another issue. My packets start dropping and I timeout for up to an hour. After doing a bunch of debugging and testing with IPIP traffic using my Packet Generator tools, our hosting provider and I believe this is due to filtering within the upstreams. Our hosting provider has put in a ticket to the upstreams to have them take a look at this issue. IPIP traffic itself isn't filtered, but it seems like there's something that scans the inner IP header's source and destination IP addresses and makes filters based off of those fields which is interesting. This is currently the biggest roadblock to implementing our temporary solution since it impacts the ability to get the servers up and running smoothly. I'm hoping we can get this issue resolved soon and once we do, this should allow our server divisions to setup the new Europe servers for expansion! Other than that, I also gained back access to the UK POP and updated Compressor's config so once the BGP session is active, Compressor will be forwarding traffic to the right game server machines (I was an idiot and locked myself out last time). This was another step in getting the BGP session resolved on the UK POP. Small Change To IP/ASN Listing Program I just wanted to mention some changes I made last night to my IP/ASN listing Go program here. I ran into issues last night when performing initial Europe server testing when on server startup, it would not connect to Steam and web traffic was also not flowing properly (e.g. requests to a REST API). It turns out the IP prefix lists I have Compressor pull from were blank and this was due to my IP/ASN listing program. The API I use for this called BGPView was rate-limiting a majority of our API requests. Therefore, I ended up implementing functionality to do the following. When scanning the lists/ directory, I've added an option to randomize the lists to update. Added a max lists option which is a cap on how many lists that can be updated per interval. Added an option that doesn't write to the list when any ASN lookup fails for that specific list. This has been working great so far and this is just one less thing to worry about for the future. Anyways, that's all I have for now. Once things are up and running, I will be making a global announcement. From there, it will be up to the divisions to announce updates for the servers they plan to expand in within Europe. Thank you for your patience regarding the Europe expansion and I hope you enjoyed the update! Share this post Link to post Share on other sites More sharing options...