Sydney POP Issue

[Roy 10,832 / 0]

Hi everyone,

 

I just wanted to let everyone know about an issue that came up today regarding players routing through our Sydney POP. This POP had a forceful reboot from our hosting provider earlier today at around 4:00 PM CST (I suppose there was an issue with the node the server resided in).

 

This would have caused a complete timeout for at least 30 seconds or more (depending on whether the POP went completely down before the hosting provider rebooted the POP).

 

Additionally, users were experiencing issues after the POP came back online. One main symptom would be not seeing our servers in the server browser. This is because the POP rebooted on a kernel (5.6) that breaks Compressor's AF_XDP program. This is a known issue to me and we've seen this times before. Therefore, I rebooted the POP On 5.4 (the stock Ubuntu 20.04 kernel that works without any issues) which resolved the issue.

 

Compressor has outdated AF_XDP code that breaks in newer kernels and reprogramming the AF_XDP program would become a huge pain/not worth it in my opinion (AF_XDP is also very complex, but I understand it a lot more than what I did months ago). I was seeing about doing this to get XDP DRV mode working (since for some reason, it doesn't work with the current AF_XDP program), but sadly the virtio_net driver has a bug with AF_XDP sockets right now (it states the 'device or resource is busy' after initial load and you won't be able to rebind the XSK socket after the first load) and I've tested it using the most up-to-date AF_XDP code with a program I made here with the same issue. I'm still inspecting the virtio_net driver code within the kernel to see if I can narrow down why AF_XDP sockets do not clean properly, but this is a more complex issue as you'd imagine.

 

Anyways, I'm going to be removing the 5.6 kernel from this POP so it doesn't reboot on it next time.

 

I apologize for the inconvenience and thank you.