Machine Performance Issues That Started On June 24th

[Roy 10,832 / 0]

Hello,

 

I just wanted to inform everyone about an ongoing event that started occurring on June 24th. I've been posting updates on the public GFL Discord. However, I never got around to posting them on the website. I apologize for this.

 

Early Sunday morning, one of our NFO machines started experiencing high CPU usage and poor performance. We noticed our stock servers (no custom addons, etc.) consuming more CPU as well. Therefore, we opened a ticket with NFO thinking this is either an issue with the machine or some sort of attack (we've had attacks that aimed for the OS and affected all servers on the machine in the past).

 

In the afternoon, we received the following update from NFO:

 

 

This was the cause to the higher CPU usage and poorer performance we are experiencing.

 

After I found this out, I requested to move two popular servers (CS:GO Zombie Escape and GMod TTT 1) to a temporary machine. Unfortunately, the temporary machine wasn't set up until early this morning. I also asked what the ETA was for the new PSU and they told me they won't install it until probably tomorrow.

 

Due to the ETA being one day from now and the issue already going on for over twenty-four hours, I decided to start moving our CS:GO Zombie Escape and GMod TTT 1 servers to the temporary machine. I try to stay away from moving big servers like this because there are a lot of files and we've had corrupted moves in the past (which are a hassle to resolve). However, the servers are almost unplayable at this point and I don't want to keep them this way for any longer.

 

I have already triggered the move of CS:GO Zombie Escape and will be triggering the move of GMod TTT 1 soon afterwards. It takes time for these moves to complete since the machine has to move everything over. I will continue to give updates on these moves as progress is made.

 

I apologize for this inconvenience. When I played GMod TTT 1 yesterday and saw the server FPS at a steady <5 FPS, I was disgusted.

 

I am hoping these server moves will go smooth without any major issues and downtime. I will reply to this thread as I make progress and receive updates.

 

Thank you for understanding.

[rapperdan 2,323 / 0]

@Roy I am slightly concerned and would like to know if this involves an IP change.

[annoying furry 6,300 / 63,950]

6 minutes ago, rapperdan said:

@Roy I am slightly concerned and would like to know if this involves an IP change.

It should not.

[Roy 10,832 / 0]

1 hour ago, rapperdan said:

@Roy I am slightly concerned and would like to know if this involves an IP change.

Moves in the same datacenter do not require IP changes.

 

The IPs will be staying the same.

 

Thanks.

[Hello. 1,452 / 24,189]

The heads up is much appreciated.

 

Once the issue at NFO is resolved, I'm assuming the server files will be moved back to their original machine? Is it accurate to state that given the amount of files for said servers, this should all be resolved by this coming Sunday(pending everything goes according to plan)?

[Roy 10,832 / 0]

6 minutes ago, PB-n-J said:

The heads up is much appreciated.

 

Once the issue at NFO is resolved, I'm assuming the server files will be moved back to their original machine? Is it accurate to state that given the amount of files for said servers, this should all be resolved by this coming Sunday(pending everything goes according to plan)?

I believe that is what will happen. And yes, I am hoping it is resolved by this upcoming Sunday. I expect to have it moved back by Wednesday to be honest (if they install the new PSU tomorrow).

 

Thanks.

[Roy 10,832 / 0]

Just an update, I have triggered the move for CS:GO ZE this morning. However, it still hasn't completed. I have open a ticket with NFO just to check on the status of the move (we've had moves get "stuck" in the past).

 

The machine has to compress all the server files (addons, maps, etc.) and I would assume this is above 10 GBs. Unfortunately with the machine performing poorly already, I'm sure it will take a while for it to compress everything.

 

Thanks.

[im not inactive you are 888 / 16,340]

@Roy its all your fault for ddosing our servers

[Roy 10,832 / 0]

Another update,

 

CS:GO Zombie Escape was moved to the new machine with one minor issue caused by the move process.

 

I am waiting to move GMod TTT 1.

 

I have contacted the CEO of NFO. What I am thinking of doing is purchasing a new machine that will be an upgrade from the old one. Currently, the old machine has the Intel Xeon 1270v5 @ 3.6 GHz. The machine I want to purchase in replacement to the old machine has an Intel Xeon 1275v6 @ 3.8 GHz in it.

 

With that said, there will be an one-time installation fee of $99. But due to the installation fee, it will actually be $10.00/m less than what we pay for the old machine ($164.98/m is what we will pay, we currently pay $174.98/m for the current machine). In my opinion, this will benefit us more. Especially considering we plan on keeping this machine for over a year at least.

 

I am waiting to see how long it will take to get the machine setup. If it can be set up by tonight, I will be moving all servers over to it and cancelling the old machine.

 

I will reply to this thread once I receive an update.

 

Thanks!

[Violator 2,242 / 20,845]

He forgot to update, we should get the machine this weekend.

[Roy 10,832 / 0]

The machine will be ordered either today or tomorrow (whenever it becomes available in Chicago). We will have it set up by the weekend according to the CEO.

 

Thanks.

[Roy 10,832 / 0]

One last update to this thread,

 

The old machine is back to performing normally. I have placed an order in for the E3-1275v6 machine yesterday. The CEO of NFO will have it set up by the weekend.

 

I will be making a separate announcement when I move all the servers to the new machine.

 

Thanks!