Jump to content
 Share

Roy

Outages On Dallas POP and GS14 [September 23rd, 2020]

Recommended Posts

Hey everyone,

 

This thread is being made for documentation purposes.

 

At 10:16 AM CST today, our Dallas POP with GSK lost network connectivity. Since the POP at the time was still announcing our IP range, this resulted in players being briefly disconnected until our IP range stopped being announced. For the next thirty or so minutes, the POP was announcing the IP range on and off which caused connectivity issues for users routing through our Dallas POP. This was because the machine was still online and BIRD was running which resulted in our range still being announced (it only needs to communicate with the neighbor address), but external traffic into the POP failed.

 

Players routing to a POP not directly affected by this incident may have not been able to see our game servers in the server browser as well during this time, but should have been able to connect to them directly. This is because our game servers in Texas route through the Dallas POP and it would use that POP server to cache A2S_INFO responses. Since this POP server went down, the A2S_INFO responses never made it back to the Redis server causing the players to never receive A2S_INFO responses (the server wouldn't show up in the server browser in this case). With Bifrost, we'll have a better method of handling A2S_INFO responses/caching which should resolve this issue from occurring again if the main POP goes down.

 

The Dallas POP's issue was corrected 45 minutes or so after the initial incident. However, our GS14 machine (with GSK as well) went down soon afterwards. This resulted in downtime for game servers on this machine. It turns out it ran into similar issues regarding the network as the POP server.

 

GSK (Gameserverkings) released an official statement here stating what caused the outage.

 

I apologize for the inconvenience and thank you for understanding. I don't believe this'll happen again and I do appreciate the transparency from GSK on this situation.

 

If you have any questions or concerns, please feel free to reply.

 

Thank you.

Share this post


Link to post
Share on other sites


Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now


×
×
  • Create New...