GS05 Downtime + Explanation

[Roy 10,832 / 0]

Hi everyone,

 

I just wanted to give an update on the situation we had with our GS05 machine. This machine hosts the following servers so far:

 

• CS:S Zombie Escape.
• Garry's Mod Murder
• Call of Duty 4 HC FFA.

 

At some point yesterday morning we started experiencing issues with our Docker containers. Basically the required network interfaces weren't showing up in the Docker container. I didn't get to inspect this but this is what I was told. We also started running into DNS issues where queries would take a long time to respond or it would just not resolve host names properly. This resulted in downtime for our game servers.

 

I believe the first issue was due to a configuration error with our control panel where it would isolate the Docker container in its own network which isn't how we want it setup for now. I don't know HOW this was changed. However, it may have been done manually when trying to fix the DNS issue. Anyways, I've changed it back to what everything should be and that issue is resolved.

 

The second issue regarding DNS not properly resolving was the main issue I believe. I've spent a while this morning ensuring everything was properly setup, however, nothing was working. We were using Google's DNS (8.8.8.8 and 8.8.4.4). I ended up contacting our hosting provider to see if there were any changes done to their hardware firewall and it turns out due to some sort of attack, they blocked Google's DNS on their firewall level. Therefore, obviously it wouldn't work. They setup our host machine with the Cloudflare DNS servers (1.1.1.1 and 1.0.0.1) which is why they didn't contact us about this. However, the Docker containers still used Google's DNS. So it wasn't resolving any host names within the Docker containers which is what our game servers are hosted in.

 

Since I'm currently on my laptop and don't have any source files to the custom Docker images I've built for our game servers, I cannot rebuild the Docker images to use Cloudflare's DNS instead of Google's.  These source files are stored on my desktop which is currently turned off right now to use less electric within my apartment. Once I get home, I will look into rebuilding our Docker images to use Cloudflare's DNS instead of Google's.

 

Anyways, I asked our hosting provider to unblock Google's DNS for now and they said that was fine. So everything is resolving correctly within the Docker containers now and our game servers are functioning correctly.

 

The Delay

The last thing I want to talk about is the delay. I want to apologize for it. Yesterday I was out from the early morning until later at night due to events happening at my job. I'm currently on a business trip and we had a Happy Hour yesterday where I met many people including the CEO. Unfortunately, the Happy Hour lasted long at the older office and by the time we got done, the newer office was locked and nobody had access to get in. My laptop and stuff was there... So I didn't have any access to my laptop pretty much all day/night yesterday until I came back into the office this morning. I didn't have an SSH terminal on my phone, so I was pretty much useless until I got my laptop back this morning. Unfortunately a very bad time for us to suffer from an issue such as this.

 

I just wanted to thank @Xy for trying to look into this issue and fix it at the time. Unfortunately, this issue took a bit to figure out and required to contact the hosting provider. As of right now, Xy doesn't have their contact information. However, in the future I'm going to ask our hosting provider if Xy can contact them if there's a critical issue with our GS05 machine.

 

I am also going to be talking to our hosting provider to ensure these things are communicated properly in the future.

 

I apologize for this inconvenience and thank you for understanding.