Jump to content
 Share

Roy

[11-6-20] DNS Outage

Recommended Posts

Hey everyone,

 

I am creating this thread for documentation purposes.

 

Last night, we started seeing a lot of servers on our GS15 machine experiencing DNS-related issues. This resulted in many services such as connecting to Steam (and showing on the server browser) breaking since hostnames weren't resolving.

 

This included GMod Prop Hunt and GMod TTT Minecraft. It's still unknown whether this impacted all game servers on this machine (some servers appeared to be fine while using the same DNS servers).

 

It turns out most requests to CloudFlare's DNS servers weren't making it back since that's what we primarily use for DNS. I noticed I could cURL 1.1.1.1 (CF's IP) inside the Docker container (since we don't have a privileged user inside the container to use MTR or trace route, I had to use cURL instead, I could have used ip netns exec <netnsid> mtr 1.1.1.1> on the host though which I plan to do in the future).

 

After discovering this, @Aurora and I were trying to switch the DNS servers to Google (8.8.8.8). This was more difficult than I had thought (I tried creating a /etc/netns/<netnsid/resolv.conf file containing Google's DNS servers for specific network namespaces and /etc/docker/daemon.json including Google's DNS servers, but it still used CloudFlare for some reason). Aurora discovered our control panel actually sets DNS servers and we had to set it in that config instead. After we started using Google's DNS servers, everything started resolving again. A few minutes afterwards, I also noticed CF's DNS queries were making it back again.

 

I asked Renual (owner of GSK) about this and he stated they specifically whitelist CloudFlare's servers. However, they did start peering with them recently. Therefore, there's a possibility that had something to do with it. But it still didn't make sense that I could cURL the CF IP (if it was a routing issue, I wouldn't have been able to do that).

 

Anyways, we started using Google's DNS primarily and Renual also suggested using their DNS servers which basically uses both Google and CF's servers as forwarders, but has a larger cache which should technically be faster for us.

 

Thank you.

Share this post


Link to post
Share on other sites




×
×
  • Create New...