[Announcement] [MAJOR] Our New Network, Physical Dedicated Machines + More!

[Roy 10,832 / 0]

Hello everyone,

 

I would like to announce the release of a major project that we’ve been working for the last three weeks or so. GFL now owns its network that will be running in-front of our future game servers! This is by far my most favorite project I’ve worked on, ever. While the last three weeks have been very hectic (exchanging many emails/tickets/paperwork, actively trying to learn BGP routing + BIRD configuration, and so on), I do believe the end result will definitely be worth it, especially for GFL! I can say personally that I am learning a lot from this project and it further advances my knowledge in networking as a whole.

 

Our New Anycast Network
@Dreae and I decided to build an Anycast network to put in-front of GFL’s game servers. For those that do not know what an Anycast network is, this article by CloudFlare explains it well in my opinion. 

 

Anycast itself will benefit us because we’ll be able to absorb and filter (D)DoS attacks at the PoP server level. This results in overall better (D)DoS protection. With that said, the more PoP servers we have, the more network capacity we’ll contain.

 

Our Current Setup
As of right now, we have nine PoP (Point Of Presence) servers scattered around the world and plan to add more as time goes on. These locations include:

 

• Dallas, TX (US South).
• New York City, NY (US East).
• Chicago, IL (US Central).
• Miami, FL (US South).
• Los Angeles, CA (US West).
• Seattle, WA (US West).
• Paris, FR (Europe).
• London, UK (Europe).
• Frankfurt, DE (Europe).

 

Here’s our Anycast layout. As of right now, we’re using the Vultr setup and don’t have any PoP servers in Tokyo and Singapore. Though, we plan to get PoP servers in these locations in the future. As of right now, the above PoP servers combined cost us $135.00/m.

 

Our PoP servers run SRCDS packet filtering software made by @dreae which is open-source and can be found here. @Dreae has done a great job with this!

 

Ongoing Maintenance
Due to the nature of routing and networking, this new Anycast network will more than likely require active monitoring for sub-optimal routes due to things always changing. This is why we have made the following sub-forum:

 

https://gflclan.com/forums/forum/959-gfls-network/

 

In here, we will make a form where users can submit sub-optimal routes they find to our PoP servers via traceroute/MTR. With that said, we will be posting maintenance and outage events in here as well.

 

@Dreae and I are still exploring the world of BGP routing and BIRD configuration. However, we are making progress by using BGP communities and so on to influence routing to our PoP servers. I hope we can eventually find something better than BGP communities, but it’ll do for now.

 

F.A.Q.
Q - Do you plan on making a public guide going over how you’ve done all of this?

 

A - Yes, @Dreae and I plan to be as transparent as possible. We will be making a public guide in the future explaining everything we had to go through in order to create this network. We are still pretty new to all of this ourselves, but as time goes on and we learn, we will be making our techniques/advice public!

 

Q - Would you be up to host non-GFL servers via the Anycast network?

 

A - As of right now, this will be used specifically for GFL only. However, I am considering this in the far future (I’d LOVE to become a hosting provider some day, but A LOT goes into that and I need to gain more knowledge in networking itself before that’s a possibility).

 

Q - Do you plan to use other providers to host your PoP servers?

 

A - Yes, this is the plan and overall, this would make our network better since we wouldn’t be relying on one hosting provider to host our PoP servers. However, in order to do this, we will need our own ASN for our IPv4 block. This is a longer process since it requires verification, contracts with at least two transit providers, etc. However, in the next week I will be looking to apply for one through RIPE (our IPv4 blocker provider can sponsor us and we have two transit providers as well).

 

Our New IPv4 Block
We have leased our very own /24 IPv4 block. This comes with 256 IPs as Anycast only supports a minimum of /24 IPv4 blocks. Thankfully, we were able to get this entire IPv4 block for only $56.00/m, which in my opinion, is a very great deal!

The IPv4 block is 92.119.148.0/24 (usable IPs go from 92.119.148.2 to 92.119.148.254).

 

While being able to use this with our Anycast network is a benefit of its own. There are also other benefits of owning an IPv4 block. For example, if we were to want to switch physical hosting providers, we can do so without any IP changes as long as it’s a BGP-supported hosting provider. However, since our Anycast network itself forwards traffic from the PoP servers to the physical hosting provider, we won’t need a BGP-supported physical hosting provider. Thankfully, our new physical hosting provider has BGP-support either way. Therefore, if we were to remove our Anycast network for whatever reason, we can still setup a BGP session with them which will result in our IPs still not changing.

 

Geo Location
Since this IPv4 block was purchased using RIPE, the geo-location of the block was either non-existent or pointing towards somewhere in Europe. We want this to be set to Dallas, TX, US to start. I’ve emailed three of the biggest geo-location database providers (MaxMind, IP2Location, and Neustar). MaxMind has already made this correction last week. I emailed IP2Location last night and they replied wanting further verification which I then provided. Therefore, I am just waiting to hear back from them. Neustar has yet to reply.

 

I believe the Valve Master Server may use IP2Location (I initially thought they used MaxMind, but I believe I’m wrong). Therefore, our game servers under this IPv4 block may show up late or incorrectly in the Valve server browser based off of the client’s geo location. Once the correction is made through IP2Location, we’ll just have to wait for the Master Server to update their database. As soon as that’s done, we should be good to go!

 

Acquiring Our Own ASN
The next big process is acquiring our own ASN. As of right now, we have to use a hosting provider’s public ASN (in this case, Vultr’s) since we don’t have our own. Once this is done, we’ll be able to use multiple hosting providers for our PoP servers. This will make our network more stable/better and less costly since we’ll be able to use cheaper and more efficient providers with lesser locations.

 

Our IPv4 block provider will be able to sponsor us. Therefore, the only other thing I will need is contracts with two transit providers (I believe I have two I can use) and my driver’s license. After I apply for our own ASN, the process itself will take around 2 - 3 weeks.

 

New Physical Hosting Provider In US
In addition to the above, we have found a new physical hosting provider that will be ran under the new network. We have purchased two machines from this new hosting provider which are the following:

 

First

• Intel Xeon E3-1271v3 @ 3.6 GHz.
• 32 GBs of DDR4 RAM.
• 500 GBs SSD.
• 15 TBs bandwidth.
• $80.00/m.
• Dallas, TX, US.
• Status - Active.

 

Second

• Intel i7-7700K @ 4.2 GHz.
• 64 GBs of DDR4 RAM.
• 500 GBs SSD.
• 15 TBs bandwidth.
• $120.00/m.
• Dallas, TX, US.
• Status - Inactive (still being built).

 

I was aware that our GS06 machine was very overloaded. Therefore, I decided to purchase two machines from the start to be on the safe side. This hosting provider will be able to setup machines within a reasonable time-frame as well and appears to always have dedicated machines available on the spot.

 

The first machine will host medium-ranged servers (e.g. <40 slot servers). The second machine will be used to host our bigger servers.

 

We found that this hosting provider also has pretty decent routing and good support. I believe they’ll work for us in the future! In the end, if they don’t, we’ll be able to switch hosting providers without changing IPs  

 

What Servers Will We Be Running?
With the new physical machines, we will have a lot of room to play with. We will be moving servers over to this new network that aren’t so popular. We will primarily move servers in Garry’s Mod and Rust. Unfortunately, any moved servers will require an IP change. But I’m very hopeful that’ll be the last time we switch IPs  

 

We also plan to setup new servers as well! As soon as our IPv4 block’s geo location is corrected, I believe we’ll be good to go.

 

As of right now, I don’t have any list of servers we’ll be moving. This will be up to the Division Leaders + Directors. I will say I would like to see our Garry’s Mod servers from GS06 moved along with Rust.

 

Test Servers
I’ve setup some silly test servers on our first new physical machine. You can find all servers that run under our new IPv4 block on GameTracker here.

 

 

 

Feel free to connect to these servers and see how your connection is from the PoP servers to the new physical hosting provider! Remember, you will be able to submit sub-optimal routes if you have one under the new network forum found here.

 

Special Thanks To @Dreae!
I just wanted to thank @Dreae again! He has been a big help with this project. He has made the packet filtering software that runs on all of our PoPs, given advice on the network setup itself, altered some needed routing via BGP and BIRD, and more! This project definitely wouldn’t be possible without him.

 

Conclusion
All in all, this is a BIG addition to GFL. There are going to be required tweaks in the future and there is still a lot for @Dreae and I to learn. But I am confident we’ll do great!

 

I have yet to see another gaming community that hosts game servers have their own network that runs in-front of their game servers. I believe this will be something special to GFL. I’m hoping the public guides @Dreae and I post publicly will influence other gaming communities to do the same since in my opinion, this is fairly cheap to setup (the entire setup itself is costing us <$200.00/m and we will be able to get that cost down once we acquire our own ASN). It might be hectic at first, but if you’re interested in networking, you’ll definitely enjoy and learn a lot from it You’re basically becoming your own mini-ISP!

 

If you have any questions, please reply to the thread or post in the GFL Network sub-forum!

 

Google Doc

 

Thank you for reading!

View full article

[Guest]

Good job Roy and Dreae! Thanks for making GFL better!

[Skittlez 721 / 11,279]

Can’t wait to see where this takes the community! Awesome work guys! 

[SS PigeonShit 231 / 0]

Thank you Roy, very cool!

[JGuary55 707 / 14,712]

The new Dust 24/7 is SOO fun!!!! you already know i had to drop 300 kills in it

 

[Bae 2,489 / 41,520]

Ohh I did t read a single word but that explain why you didn't sleep. You and xy being bad on sandbox.

[Angry 104 / 2,761]

You guys are awesome! Make GFL Great Again!!! 

[Roy 10,832 / 0]

While we'll still be able to roll out servers onto this new network, we're currently facing a limitation for games that use the Valve Master Server. The Valve Master Server tries to choose servers based off of the player's geo-location. It is unknown what database they use for this. I've tried emailing the Valve developers and making an AlliedMods thread here. I still haven't received a response from Valve and I don't expect to.

 

Anyways, since our new IPv4 block appears to not have been used in the past, it didn't have a geo-location set on any databases I've done lookups on. This results in nobody being able to see our servers in the in-game server browser unless if you specify tag filters (even then, we're still getting some players at least ).

 

Initially I thought the Valve Master Server used MaxMind. However, our IPv4 block was updated in MaxMind a couple weeks ago after I put in the request and it appears the Master Server still doesn't detect a geo-location (for example, our IPs come up at the top of the Master Server list when I query it via the API). Unless if the Master Server hasn't pulled updates from MaxMind's database yet, I don't believe they use them. I believe they are using IP2Location. I've been only running into issues when trying to get IP2Location to update the geo location (apparently they don't understand that we're running an Anycast network and forwarding traffic to a physical machine). Anyways, once we give them solid proof our physical machines from Dallas, TX and London, UK will be announcing these IPs, they will queue the change. The problem is, they won't be able to do this until May because they're already 'validating' their upcoming April release. It's stupid as hell, but unfortunately the case:

 

Quote

Hi Christian,

 

It will be available in the May 2019 release at the soonest. The April 2019 release is currently under validation and no changes can be done.

 

I guess we won't reach the full potential of our game servers until this change is successfully implemented.

 

To sum it up, MaxMind > IP2Location any day.

 

Thanks.

[Roy 10,832 / 0]

An Update

Although IP2Location told us our IPv4 block wouldn't be updated until May, we got updated this month anyways:

 

Dats a sick ISP

 

With that said, the Master Server doesn't show our IPs at the beginning of the query anymore and I can see our test servers immediately when refreshing the server browser!

 

(first server)

 

I would say it's confirmed the Valve Master Server does indeed use IP2Location

 

Thanks!