Jump to content
 Share

Roy

Paris POP Fixed & New IP Blocks Announced

Recommended Posts

Hey everyone,

 

I just wanted to post an update regarding our Anycast network.

 

Paris POP Fixed

Our Paris POP server hasn't been announcing any of our IP blocks for a while now. When modifying our BIRD configuration to announce the two new IPv4 blocks tonight, I found out why our Paris POP wasn't announcing any blocks beforehand. I was missing this part of the BIRD config which is needed for scanning:

 

protocol device
{
        scan time 5;
}

 

One thing to note here is because our Paris POP has my filters deployed within Compressor, anybody who was rerouted to the Paris POP while on our game servers would have lost connection until they reconnected (since the filters are based off of the handshake sequence between the client and game servers). I didn't expect my fix to work since I thought I attempted this months ago. However, it did and apologize for the inconvenience for those players impacted.

 

3514-11-21-2020-oW6O0K3j.png

 

It is now receiving a good amount of traffic :)

 

New /24 IPv4 Blocks Announced

I modified our BIRD configuration tonight on all of our Anycast POP servers and started announcing HG's /24 IPv4 block (185.141.204.0/24) and our new /24 IPv4 block (185.240.217.0/24) being used for our Europe expansion. This was smooth because I used birdc configure to reload the BIRD configuration. This would check for syntax errors and any other issues before applying the new config and also wouldn't restart the BGP session itself which could cause BGP session flaps (this could happen if we restarted via the systemd service itself).

 

I've confirmed both IPv4 blocks are being announced on all our POP servers/hosting providers by executing birdc show route on the POPs themselves and also using this website to perform MTRs to IPs on these blocks from many locations around the world.

 

3507-11-21-2020-VOwOu51H.png

 

3506-11-21-2020-8DjagU2W.png

 

There does seem to be some sort of strange routing loop going on for these new IP blocks on some providers:

 

3510-11-21-2020-WDfC8ZT3.png

 

3512-11-21-2020-X2qeiw0K.png

 

I'm still investigating this, but there's a good chance this will correct itself. I've also asked the GSK owner, Renual, how they'd go about troubleshooting this issue. If it continues, I'll be using popular Looking Glass websites to perform MTRs from major ISPs and see if I can narrow it down to the specific POP or upstream causing this issue.

 

Note - Some locations timeout because ICMP replies are disabled on those specific POPs. This is because beforehand, we were forwarding ICMP requests to our game server machines and attackers could use this vector to over-saturate the 1 gbps NICs our game server machines have (a single point of failure). The locations responding to ICMP replies have the POP server itself respond to these requests, so it isn't as big of a deal (though, it could still consume more resources on the POP doing it this way rather than dropping the request entirely and not sending back a response. But it allows us to check for loss packets on the POP itself which is very helpful for debugging performance issues with the network).

 

If you have any questions, please let me know!

 

Thank you for your time.

Share this post


Link to post
Share on other sites




×
×
  • Create New...