Jump to content

Roy

Banned
  • Posts

    2,853
  • Joined

  • Last visited

  • Days Won

    383

Posts posted by Roy

  1. Servers appear to be stable now. It was more than likely an attack. Here is the response we received from NFO's support:

     

    Quote

    Thanks for contacting us today. It appears CPU usage for the machine spikes to 100% and stayed there. I've triggered a reboot for the machine to bring the servers back online. This should complete shortly.

     

    Based on the graphs, this may have been triggered by an attack against a server, likely causing a server or servers to lock up.

     

    Thank you.

  2. Just now, Xy_ said:

    i already had them power cycle

    Probably an attack then. Could be a process stuck on the OS itself as well (less likely).

     

    I'll get the CEO to look into it.

     

    Also, RIP like 1+ year(s) up-time for the machine haha.


    Thanks.

  3. Hello,

     

    I just wanted to notify everyone our NFO machine is currently experiencing issues at the moment. It appears our machine is pegged at 99% CPU (this may be caused by an attack). This is resulting in our game servers going offline on this machine.

     

    I have submitted a ticket with NFO and look forward to hearing from them soon regarding this incident.

     

    I will reply back here once I have more information.

     

    Thank you and I apologize for the inconvenience.

  4. Hello,

     

    This morning our GS05 machine was down for the last 7 - 8 hours. When I woke up, I notified our hosting provider. It turns out it was an issue with the hosting provider's IPv4 block which is now corrected.

     

    I understand GS05 has been going down quite often recently. Therefore, when I'm not busy with life/work, I will be talking to our hosting provider about this.

     

    Thank you and I apologize for the inconvenience.

  5. 3 minutes ago, DEV Lars said:

    Hoping to see more of this. Good read. 

     

    Will there be a tutorial for this PoP Setup?

    Yes, @Dreae and I plan to make a public guide going over everything we've done + advice that will be released on the website once the network is completely setup (running our own ASN, etc.).

     

    Thank you.

  6. Hey everyone,

     

    I just wanted to give another update on GFL's Anycast network! We've been making a lot of progress as of lately, but there is still more work to do.

     

    ASN Update

    We are still in the process of acquiring our own ASN. In a post I submitted here, I said we could get our own ASN through ARIN within two days (compared to RIPE that will take a few weeks). However, what I did not see is that we would have to pay $200 up front and there's an annual fee afterwards (we do not have any sponsors with ARIN). Therefore, we are going to try to get our ASN through RIPE instead which was our initial plan anyways (I've confirmed that we can get it through our sponsor for completely free). This will take a few more weeks but we're in the process of doing so.

     

    Physical Hosting Provider + Main PoP Server (Point of Presence)

    As stated in the last update thread, we are looking to move our physical hosting over to Wholesale Internet (if we get the main PoP server and physical server with the same hosting provider in the same location, the route will be more optimized and there shouldn't be many hops). What we wanted to do initially was to buy a cheaper machine (~$20-$25/m) to use as our PoP server and then get our game server machine (either the i7-9700K for $129.00/m or i9-9900K for $149.00/m). However, after contacting Wholesale Internet, we found we cannot do BGP sessions with any machine under $60.00/m (regardless if we purchase another higher-end machine to come along with it). Personally, I don't want to pay for a $60.00/m machine for just our main PoP server. We are currently only paying ~$20.00/m for it with Vultr.

     

    We decided to go hunting for other data centers in Kansas City (where Wholesale Internet is). These data centers will need the following:

     

    • Support for BGP sessions/IPv4 block announcements on cheaper machines.
    • Good peering with Wholesale Internet to ensure we have the most optimized route from the PoP server to the physical machine possible.

     

    We found one hosting provider (Dedispec) already that have powerful cheap machines along with a lot of bandwidth (~100 TBs) and @Dreae asked them if they supported BGP sessions on these machines along with a test IPv4 address if possible. They came back stating they do support BGP sessions/IPv4 block announcements and they gave us a test IPv4 address. It turns out they actually use Wholesale Internet as a hosting provider either way (they probably colocate with them). This means the PoP server and physical machine will technically be in the same data center, but with different providers. This will still mean the route will be more than likely fully optimized.

     

    We are also in contact with another hosting provider for the PoP server and awaiting their reply. It's good knowing our options :) 

     

    If we are able to get our physical machine with Wholesale Internet and get our PoP server with another hosting provider for cheap that ALSO uses Wholesale Internet themselves (along with BGP support), it's a win-win situation for us, lol.

     

    Aulerion

    I just wanted to state we won't be able to use Aulerion once we get our own ASN. It turns out they shut down (thankfully we didn't start with them, haha). Looks like I might have to find another transit provider so that I can show further proof when acquiring our own ASN (need two transit providers as a part of the validation process). Here's the message on their website:

     

    Quote

    Dear customers and visitors.

     

    Aulerion is currently facing difficulties in terms of financials, forcing a temporal closure of the service.

    The reason behind this is that we tried remaining competitive in pricing, which worked for a long time as we
    were able to cover all expenses with revenue. Aulerion to date has not generated a single cent profit for
    it's owners. So far, the project has costed various persons which invested in the project over $30000:

     

    Florian B. $20000
    Robert R. $5000
    Wim B. $5000

     

    As a result of recent massive DDoS attacks targeting smaller and larger parts of our infrastructure, which
    caused tremendous bandwidth bills especially in our Asian regions, we are no longer able to cover all our
    costs and ran out of our last reserves in the company under which Aulerion operated.

     

    To sumarize, the whole project has costed the owners over $30000 in costs to run, whilst not turning a
    single cent profit, aside of dozen hours spent on maintaing the project. We are at this time unable to
    continue operating Aulerion. The project has turned loss in every month of operation, up to the point
    where continuing both makes no more sense and is simply not possible.

     

    Servers are still online at the moment, please make sure to backup anything of relevance as soon as possible.
    We are actively talking to suppliers to postpone termination of our agreements to allow our customers to backup,
    but we do not anticipate a lot more room.

     

    Finally, we wish to apologize for the huge inconvenience caused and we hope to be able to provide services to you
    again in a smaller fashion in the near future, where you can potentially spend your potentially lost credit.

     

    Conclusion

    That's basically it. @Dreae has done an incredible job improving Compressor on our PoP servers recently and reaching out to some hosting providers! I just wanted to thank him again for everything he has done.

     

    Since IP2Location decided to update our IPv4 block this month (thank god) instead of next month like they stated initially, our servers are showing up properly in the server browser (Valve's Master Server). We are starting to move servers over to this new machine along with setting up new servers. You can see the list of public servers running on our new network here. Our Fistful of Frags server we've setup is doing okay so far:

     

    b_560_95_1.png

     

    We will be changing physical hosting providers more than likely once we get our own ASN. However, keep in mind our game servers won't change public IPs under this network since we own the IPv4 block ;)

     

    If you have any questions, please let me know!

     

    Thank you.

  7. An Update

    Although IP2Location told us our IPv4 block wouldn't be updated until May, we got updated this month anyways:

     

    310-04-05-2019-XJAE6mHO.png

    Dats a sick ISP ;)

     

    With that said, the Master Server doesn't show our IPs at the beginning of the query anymore and I can see our test servers immediately when refreshing the server browser!

     

    311-04-05-2019-r1Ijf7Pj.png

    (first server)

     

    I would say it's confirmed the Valve Master Server does indeed use IP2Location :)

     

    Thanks!

  8. The issue appears to have been fixed.

     

    New route (lower latency than before):

     

     1. x.x.x.x.static.nexril.net                                                0.0%  3985    0.6   0.5   0.4  63.9   1.1
     2. be-20.cr01.dal.nexril.net                                                        0.0%  3985    0.9   0.5   0.4   2.1   0.0
     3. 66-206-18-73.static.hvvc.us                                                      0.1%  3985   21.9  18.7   1.2 1261.  38.0
     4. ???
     5. ???
     6. ???
     7. ???
     8. ae-34.a02.dllstx09.us.bb.gin.ntt.net                                             0.1%  3985    0.4   1.1   0.4  36.3   4.2
     9. ae-4.r10.dllstx09.us.bb.gin.ntt.net                                              0.1%  3985    0.7   0.6   0.6   1.3   0.0
    10. ae-0.a01.dllstx09.us.bb.gin.ntt.net                                             33.8%  3985    1.5   2.0   0.4  52.9   5.2
    11. ae-0.choopa.dllstx09.us.bb.gin.ntt.net                                          26.5%  3985    1.0   0.6   0.6  81.8   2.8
    12. ???
    13. ???
    14. server15.us.gflclan.com                                                          0.0%  3984    0.6   0.6   0.5   1.1   0.0
    

     

    Still yet to hear anything from Hivelocity. I'll wait a bit to see if I receive an update stating it's fixed from them.

     

    Thank you.

  9. 2 minutes ago, Skittlez said:

    Glad to see progress is (sorta) being made! 

    Don't give up though! Hopefully, they'll get back to you ASAP and things will begin to start moving again!

     

    This seems pretty impressive for a gameserver, even without overclocking. Why can you not overclock though?

    This is the second week of having our Anycast network nearly complete and this issue has occurred twice already with Hivelocity. Although their standard support is responsive via Live Chat, their networking team is not. I am not going to risk our players having +40ms latency due to a ridiculous routing issue that shouldn't be happening in the first place. Once we get our own ASN, we'll be in good shape either way since we'll be able to get a PoP server with the same hosting provider as our physical machine.

     

    Overclocking is against Wholesale Internet's TOS as stated in the email replies I provided.

     

    Also, exciting news! We can get our own ASN with ARIN while having a RIPE IPv4 block (they don't really care about the IPv4 blocks themselves). Will be filling out an application soon!

     

    Thank you.

  10. Contacted their peer (Hivelocity) directly via live chat. I got a ticket opened with the networking team.

     

    Chat transcript is below:

     

    Quote

    Hello Christian Deacon,

     

    Please find your requested chat transcript:

     

    Chat started on Mar 30, 2019 05:06 PM
    Christian Deacon      12:06 PM
    Hi Support,

     

    My name is Christian Deacon. I have an Anycast network setup which forwards traffic to a physical hosting provider in Dallas, TX. We use Vultr (AS20473) to host our PoP servers (Point Of Presence) and use Nexril (AS13830) as our physical hosting provider. We are having a routing issue from Nexril to our Dallas PoP server (they are both in the same location). Nexril peers with Hivelocity as well and goes through them. Hivelocity is routing all the way out to NYC. We had the same issue last week and Nexril contacted you and had this issue corrected. However, it is back and it adds ~60ms latency overhead to our network which is not acceptable for the service we host right now. I would rather contact you directly about this issue so I am not waiting for anybody else (e.g. Nexril).

     

    Here is the current MTR from Nexril to our Anycast PoP server in Dallas:

     

    
    1. x.x.x.x                                                 0.0%     2    0.7   4.1   0.7   7.5   4.7
     2. x.x.x.x                                                         0.0%     2    0.8   0.8   0.7   0.8   0.0
     3. 66-206-18-73.static.hvvc.us                                                       0.0%     2   14.7  17.3  14.7  19.9   3.6
     4. ???
     5. ???
     6. ???
     7. ???
     8. ???
     9. nyk-b5-link.telia.net                                                             0.0%     2   45.8  45.7  45.7  45.8   0.0
    10. dls-b22-link.telia.net                                                            0.0%     2   32.3  32.3  32.3  32.3   0.0
    11. vultr-ic-318447-dls-b22.c.telia.net                                               0.0%     2   32.8  32.8  32.8  32.8   0.0
    12. ???

     

    It is going from Dallas all the way to NYC, back to Dallas. When this is corrected, it stays in Dallas and results in only ~1ms latency overhead.

     

    This is getting frustrating since this is the second time it has occurred and we've only had this Anycast network setup for a week now.

     

    Can we please get this fixed?

     

    Thank you.

     

    Branden Tomski      12:06 PM
    Hello, How may I assist you today?

     

    Christian Deacon      12:06 PM
    Hey Branden, I've submitted a long message above with our current issue.

     

    Branden Tomski      12:09 PM
    Could I have the ip of your server please?

     

    Christian Deacon      12:10 PM
    The physical hosting provider or our Anycast setup?

     

    Christian Deacon      12:10 PM
    Physical hosting provider - xxx.xxx.xxx.xxx

     

    Christian Deacon      12:10 PM
    Anycast IP - 92.119.148.5 (92.119.148.0/24)

     

    Branden Tomski      12:11 PM
    Thank you for that. I will need to open a ticket for you regarding this and get it over to networking so they may correct it for you

     

    Christian Deacon      12:12 PM
    Alright, thank you!

     

    Christian Deacon      12:12 PM
    Do you know if the networking team works during the weekends?

     

    Branden Tomski      12:12 PM
    okay I have created Ticket #xxxxxx

     

    Branden Tomski      12:12 PM
    and got it over to networking for you

     

    Christian Deacon      12:13 PM
    Alright, thank you again!

     

    Branden Tomski      12:13 PM
    Not a problem have a great day Christian

     

    Chat ended on on Mar 30, 2019 05:13 PM
    Visitor Details
    Christian Deacon     Christian Deacon
    [email protected]
    Department: Support

     

    Hivelocity's support was quick and very helpful :) 

     

    Now to wait for the networking team.

     

    Thanks!

  11. Hello,

     

    This morning our GS05 machine went down. When I woke up, I notified our hosting provider and within an hour or so, they were able to get it back online.

     

    I will be requesting a RFO soon.

     

    Thank you.

  12. Earlier, the physical server started routing to the NYC PoP for whatever reason. The route was using Cogent[AS174]. We already prepended to Cogent 1x hops. So we tried raising it to 2x hops via:
     

    bgp_community.add ((64602,174));

     

    In the BIRD configuration export filter on the NYC PoP. This changed the route back to an optimized route of the Dallas PoP!

     

     1. x.x.x.x                                                 0.0%    10    0.7   0.7   0.7   0.8   0.0
     2. x.x.x.x                                                         0.0%    10    0.8   0.8   0.6   0.9   0.0
     3. 66-206-18-73.static.hvvc.us                                                       0.0%    10   17.4  16.3  11.4  21.0   2.7
     4. ???
     5. ???
     6. ???
     7. ???
     8. 66-206-30-6.static.hvvc.us                                                        0.0%     9    0.5   0.5   0.5   0.6   0.0
     9. ae4-703.cr6-dal3.ip4.gtt.net                                                      0.0%     9    0.7   4.4   0.5  30.6   9.9
    10. xe-7-2-0.cr0-dal2.ip4.gtt.net                                                     0.0%     9    0.9   0.8   0.7   0.9   0.0
    11. ip4.gtt.net                                                                       0.0%     9    3.8   4.6   1.3  14.6   5.4
    12. ???
    13. ???
    14. 92.119.148.15                                                                     0.0%     9    0.9   0.9   0.8   0.9   0.0

     

    Let's hope it stays this way :P

     

    This problem should be completely gone when we get our own ASN and get a PoP server using the same hosting provider as our physical machine.

     

    Thanks.

  13. I'd suggest trying to idle on the server. That's what I did on the server when I managed it and it was very successful. PVKII is a small game and only two - three servers are populated at a time if I remember correctly. Therefore, idling helps a lot.

     

    I will look to see if we have room on our GS05 machine.

     

    Thanks.

  14. MTR example from physical hosting provider to Dallas PoP:

     

     Host                                                                                Loss%   Snt   Last   Avg  Best  Wrst StDev
     1. x.x.x.x                                                 0.0%     8    0.8   0.8   0.6   0.9   0.0
     2. x.x.x.x                                                         0.0%     8    0.8   0.8   0.6   1.0   0.0
     3. 66-206-18-73.static.hvvc.us                                                       0.0%     8   12.4  14.8  11.7  22.2   3.4
     4. ???
     5. ???
     6. ???
     7. ???
     8. ???
     9. nyk-b5-link.telia.net                                                             0.0%     7   46.1  46.9  45.6  53.7   2.9
    10. dls-b22-link.telia.net                                                            0.0%     7   63.7  63.7  63.6  63.8   0.0
    11. vultr-ic-318447-dls-b22.c.telia.net                                              14.3%     7   65.3  64.7  64.1  65.8   0.6
    12. ???

    First two hops were removed.

     

    As you can see, it is being routed from Dallas to NYC and all the way back to Dallas adding ~65ms latency overhead to the network which is terrible. Our physical hosting provider has confirmed that this is due to Hivelocity[AS29802] (as @Dreae expected) and contacted them regarding this issue.

     

    We're hoping this is fixed soon!

     

    Thanks.

  15. While we'll still be able to roll out servers onto this new network, we're currently facing a limitation for games that use the Valve Master Server. The Valve Master Server tries to choose servers based off of the player's geo-location. It is unknown what database they use for this. I've tried emailing the Valve developers and making an AlliedMods thread here. I still haven't received a response from Valve and I don't expect to.

     

    Anyways, since our new IPv4 block appears to not have been used in the past, it didn't have a geo-location set on any databases I've done lookups on. This results in nobody being able to see our servers in the in-game server browser unless if you specify tag filters (even then, we're still getting some players at least ;)).

     

    Initially I thought the Valve Master Server used MaxMind. However, our IPv4 block was updated in MaxMind a couple weeks ago after I put in the request and it appears the Master Server still doesn't detect a geo-location (for example, our IPs come up at the top of the Master Server list when I query it via the API). Unless if the Master Server hasn't pulled updates from MaxMind's database yet, I don't believe they use them. I believe they are using IP2Location. I've been only running into issues when trying to get IP2Location to update the geo location (apparently they don't understand that we're running an Anycast network and forwarding traffic to a physical machine). Anyways, once we give them solid proof our physical machines from Dallas, TX and London, UK will be announcing these IPs, they will queue the change. The problem is, they won't be able to do this until May because they're already 'validating' their upcoming April release. It's stupid as hell, but unfortunately the case:

     

    Quote

    Hi Christian,

     

    It will be available in the May 2019 release at the soonest. The April 2019 release is currently under validation and no changes can be done.

     

    I guess we won't reach the full potential of our game servers until this change is successfully implemented.

     

    To sum it up, MaxMind > IP2Location any day.

     

    Thanks.

  16. Hi everyone,

     

    I just wanted to let you know that earlier today the routing from our physical machine to our Dallas PoP started having issues. For whatever reason, it started routing from Dallas to NYC to all the way back in Dallas. This is adding ~40ms latency overhead to the network.

     

    We have contacted our physical hosting provider and they're contacting the peer they are using who should be able to fix this issue.

     

    Once it is fixed, I will reply to this thread.

     

    Thanks.

×
×
  • Create New...