Update On GFL's Network

[Roy 10,832 / 0]

Hi all,

 

I just wanted to give an update on GFL's network. In this update, I will be talking about our current issues along with the steps we will take to correct them.

 

Routing Issues From Physical Hosting Provider => Dallas PoP Server

We're using Nexril as a physical hosting provider at the moment. We have ordered two machines with them. One in which is already setup and in-use at the moment.

 

The issue we're having is with the route from our physical machine to the Dallas PoP. I've made this thread addressing the issue.

 

In short, the route is going from Dallas, all the way to New York City, and then back to Dallas. The route should be staying within Dallas since both the physical machine and PoP server are located there. However, since it routes all the way out to NYC, an additional ~30-40ms latency overhead is added to the network (all clients have an additional ~30-40ms latency). Here is an MTR from the physical machine to the Dallas PoP:

 

 1. x.x.x.x                                                 0.0%   235    0.6   0.7   0.5   1.0   0.0
 2. be-20.cr01.dal.nexril.net                                                         0.0%   235    0.9   0.8   0.5   1.2   0.0
 3. 66-206-18-73.static.hvvc.us                                                       0.0%   235   17.6  17.0   1.9  68.0   7.1
 4. ???
 5. ???
 6. ???
 7. ???
 8. ???
 9. nyk-b5-link.telia.net                                                             0.0%   234   45.8  47.8  45.6  79.7   4.7
10. dls-b22-link.telia.net                                                            0.0%   234   32.1  32.3  32.0  46.1   1.1
11. vultr-ic-318447-dls-b22.c.telia.net                                               0.0%   234   32.8  33.1  32.4  66.4   2.8
12. ???

 

The first hop is censored since that includes our machine's IP. The provider causing the route to go out to NYC is named Hivelocity. In the MTR above, since Hivelocity disables ICMP packets, we cannot see them in the route itself (the '???' hops). However, to confirm this, you can take a look at Nexril's peers here. Cogent does not disable ICMP and with that said, Nexril tried changing the route to use Cogent. Unfortunately, it resulted in the route going to our NYC PoP instead of Dallas, lol. With that said, Nexril's Cogent line has been down for the last week as seen here. Plus, Nexril themselves said this was Hivelocity causing this.

 

This same issue occurred last week and I sent a ticket to Nexril. Nexril contacted Hivelocity to get the issue fixed (which it was, until later on). However, it came back again 5 or so days ago. While I submitted a ticket with Nexril again to have this issue corrected, I also contacted Hivelocity directly via Live Chat. In the first live chat session, they created a ticket for me and gave me the number. This was over the weekend and they said the networking team would update me within that same day. However, I never received a response.

 

For the last four days, I've been kindly contacting them via Live Chat asking if I could get an update. I learned that the networking team only works on the weekdays for non-urgent issues. However, it is now mid-week and I am still being told "Our networking team will email you with an update shortly", and nothing ever comes out of it. With that said, the route is still the same...

 

Nexril has tried contacting them as well, but Nexril has no updates on the issue. At this point, I've pretty much given up. I cannot get the issue escalated since it isn't critical (i.e. an outage) and we aren't a direct customer with Hivelocity.

 

As far as I'm aware, Nexril cannot afford better peers at the moment. Therefore, they're stuck with Hivelocity and their Cogent line is still down (this wouldn't matter regardless since when we use Cogent, it routes us to the NYC PoP either way).

 

Other Issues With Nexril

I believe Nexril has a lot of potential. However, due to the company only being ran by two people (one in which doesn't have much access to do stuff), there are major delays. For example, it has been two weeks since our order for the i7-7700K was supposed to be completed and we have only received one update regarding it (the last update was a week and a half ago stating the server was delivered but is failing to boot possibly indicating a hardware failure somewhere). Once they hire staff that can do more, I think they'll do great.

 

With that said, the peers issue stated above is a limitation as well.

 

Our Own ASN

As stated in the past, we are in the process to acquire our own ASN so we can use multiple hosting providers with our PoP servers. We were trying to get our own ASN through RIPE which requires a lot of verification and back-and-forth. It can take up to a month to get your own ASN through RIPE.

 

However, I've learned that you can get an ASN through ARIN within two days without much back-and-forth, etc. Though, the one thing I need to know is if we can get it when using a RIPE IPv4 block. I've emailed ARIN about this and they should be getting back to me soon.

 

I wanted to thank @dagreek for pointing this out (I believe he told @Dreae and Dreae relayed that information to me).

 

New Physical Hosting Provider

I am currently looking into finding a new physical hosting provider since we're having issues with Nexril. @Dreae and I found Wholesale Internet. From the looks of it, they have great peering along with good dedicated machines for a cheap price. They're located in Kansas City and here is the route from our Dallas PoP to an IP I found from Wholesale Internet (208.110.64.2):

 

 Host                                                                                Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. ???
 2. vl199-ds1-j3-650.01.06.dal4.choopa.net                                            0.0%     4    6.0   6.1   5.1   6.8   0.7
 3. ???
 4. dls-b22-link.telia.net                                                            0.0%     3    0.6   1.1   0.6   1.5   0.4
 5. kanc-b1-link.telia.net                                                            0.0%     3   11.1  11.0  11.0  11.1   0.0
 6. wholesale-ic-342389-kanc-b1.c.telia.net                                           0.0%     3   12.0  11.3  11.0  12.0   0.6
 7. 208-110-64-2.web1.mci.us.wholesaleinternet.net                                    0.0%     3   11.0  11.0  11.0  11.1   0.1

 

Only ~11ms from Dallas, TX to Kansas City is pretty good in my opinion!

 

In order to use Wholesale Internet, we will need to get our own ASN so we can deploy a PoP server there as well (Vultr doesn't have any locations near Kansas City). Initially, I was thinking about buying a $20 machine from them to use as the PoP server and the Intel i9-9900K machine to use as our game server machine. However, after emailing them the following, I've learned that they only offer BGP sessions on $60/m or over machines:

 

Quote

Hello,

My name is Christian and I am interested in some of the dedicated machines you own. I have my own Anycast network along with my own IPv4 block (/24) and looking for a hosting provider that can host a PoP server (Point of Presence) along with another physical machine for our services.

My questions are the following:

1. Do you support BGP sessions and announcing our IPv4 block? If so, are there any fees?

2. We are interested in the i9-9900K or i7-9700K machine. However, we want to know if you support overclocking or not. The services we host highly depend on the single-threaded performance of the processor. Therefore, the higher the clock speed is, the better performance we'll have.

3. What does your (D)DoS protection look like? How much capacity does your network have?

Please let me know!

Thank you.
 

 

Their Reply

Quote

Attempting to overclocking servers is against TOS.

 

We could announce your IP block on a i9 or i7 9xxx gen server.  

 

We do have a robust network however our DDos protection is currently in beta, but we provide 2gbit of DDos mitigation at no charge.  We will be offering higher levels of protection at a later date, but do not have an exact ETA or pricing yet available.

 

We would be glad to setup a new server for you.  For a full list of available servers, please visit: 

  https://www.wholesaleinternet.net/dedicated/

 

If you'd like to complete your order online, your preconfigured server can be deployed and online within minutes of your approved order.

Jon
 

 

My Reply

Quote

Hey Jon,

Thank you for your response!

Would we be able to get a BGP session or IPv4 block announcement on other machines such as your "Intel I3-540 Preconfigured" server? 

 

Their Reply

Quote

We offer BGP sessions on a minimum of $60/mo+ servers.

Jon

 

And finally, my latest reply

Quote

Hey Jon,

If we were to buy an i9-9900K machine along with another smaller one for <$60.00/m, would it be possible to get a BGP session on the smaller machine?

Thanks. 

 

So yeah, it'd suck if we couldn't get a BGP session for our smaller machine. If this was the case, we would either have to pay for a $60/m machine for our PoP server or find another hosting provider in Kansas City that has great peering with Wholesale Internet to use as our PoP server. Still unsure what we're going to do and waiting for a reply back from the above.

 

With that said, it's unfortunate overclocking isn't allowed. Being able to overclock the i7-9700K ($129.99/m) or i9-9900K ($149.99/m) would be awesome and would result in greater performance for our game servers. Though, those processors at a normal clock would more than likely perform very well regardless.

 

Keep in mind that they're located in Kansas City instead of Dallas, TX. I still think this is a good location, though. If we do go with them, I will have to update our IPv4 block's geo-location as well. I'd like to get this done before May so we can get the IP2Location change pushed forward (setup for Dallas, TX right now).

 

Conclusion

As you can see, there are still a lot of things we need to do. However, we are making a lot of progress. It's great being able to move physical hosting providers without any worry since our game server IPs won't be changing at all

 

Once I receive more information, I will make another update!

 

If you have any questions or concerns, please let me know!

 

Thank you for reading.

[Pedro 249 / 6,080]

[Skittlez 721 / 11,279]

Glad to see progress is (sorta) being made! 

14 minutes ago, Roy said:

At this point, I've pretty much given up.

Don't give up though! Hopefully, they'll get back to you ASAP and things will begin to start moving again!

 

14 minutes ago, Roy said:

Being able to overclock the i7-9700K ($129.99/m) or i9-9900K ($149.99/m) would be awesome and would result in greater performance for our game servers. Though, those processors at a normal clock would more than likely perform very well regardless.

This seems pretty impressive for a gameserver, even without overclocking. Why can you not overclock though?

Edited April 3, 2019 by Skittlez

[Roy 10,832 / 0]

2 minutes ago, Skittlez said:

Glad to see progress is (sorta) being made! 

Don't give up though! Hopefully, they'll get back to you ASAP and things will begin to start moving again!

 

This seems pretty impressive for a gameserver, even without overclocking. Why can you not overclock though?

This is the second week of having our Anycast network nearly complete and this issue has occurred twice already with Hivelocity. Although their standard support is responsive via Live Chat, their networking team is not. I am not going to risk our players having +40ms latency due to a ridiculous routing issue that shouldn't be happening in the first place. Once we get our own ASN, we'll be in good shape either way since we'll be able to get a PoP server with the same hosting provider as our physical machine.

 

Overclocking is against Wholesale Internet's TOS as stated in the email replies I provided.

 

Also, exciting news! We can get our own ASN with ARIN while having a RIPE IPv4 block (they don't really care about the IPv4 blocks themselves). Will be filling out an application soon!

 

Thank you.

[Skittlez 721 / 11,279]

7 minutes ago, Roy said:

I am not going to risk our players having +40ms latency due to a ridiculous routing issue that shouldn't be happening in the first place.

That's not what I meant 

9 minutes ago, Roy said:

Once we get our own ASN, we'll be in good shape either way

Hopefully this is soon!

9 minutes ago, Roy said:

Overclocking is against Wholesale Internet's TOS as stated in the email replies I provided.

I was just wondering why though. I'm assuming they're concerned for the stability of the machines(?). Regardless, the machine's specs are impressive.

11 minutes ago, Roy said:

Also, exciting news! We can get our own ASN with ARIN while having a RIPE IPv4 block (they don't really care about the IPv4 blocks themselves). Will be filling out an application soon!

Exciting 😁