Jump to content
 Share

Roy

GS2990wx Hard Drive Failure + Ordered New Machine

Recommended Posts

Hey everyone,

 

I just wanted to let everyone know our GS2990wx machine had a sudden hard drive failure. @Xy and I are working to get the server files off of the machine.

 

I also ordered a new machine that includes the Intel i9-9900K, 64 GBs of RAM, and 1 TB NVMe for $139.99/m. This machine will be setup soon and we will be moving our servers from GS2990wx to this machine. The DC hasn't setup the machine for us yet, but they told us it should be setup soon. I also believe the recent performance issues should be resolved when moving to this new machine. The new machine will be located in Chicago, IL.

 

Servers that were on this machine included Rust Modded #1 and #2, CS:S 24/7 Dust2, GMod nZombies, and a few TF2 servers.

 

Once I have an update, I will let you all know.

 

I apologize for the inconvenience and thank you for understanding.

Share this post


Link to post
Share on other sites


10 minutes ago, GrandmaDebra said:

Hello Roy! 😄 How are you 

Hola! I've been busy lol. Been coding all day (made this) and doing pen-testing against our machine's NICs as well. But then the above happened (GS2990wx hard drive failure) and now I'm all over the place lol.

 

I also hope you've been doing well :) 

 

Thanks!

Share this post


Link to post
Share on other sites


The new machine is setup and we're working to move servers over.

 

CS:S 24/7 Dust2 is already moved. @Xy will be moving the rest of the servers tonight/tomorrow morning. The network is prepared for the move.

 

One thing I want to mention is this new hosting provider has not removed ACL rules to allow us to spoof out as our Anycast network. Therefore, I cannot load the TC BPF program I made onto it. However, they're working with the DC to remove the ACLs for our network. I upgraded the server's kernel and installed the TC BPF program anyways so once they do remove it, I can just load the program quickly.

 

In the case they can't remove the ACLs, we will have to find a new provider most likely. There are plenty of options out there, though. We're going to see how this goes first. I don't think the Anycast network was the issue with the performance on the NYC machines anyways. I believe this was due to the NIC or hops at the data centers.

 

In the meantime, I've upgraded our Chicago POP to two cores so it can handle the traffic from the game server machine(s).

 

Thank you.

Share this post


Link to post
Share on other sites


Update

The new machine went down last night after an hour or two of running. I sent a ticket to our hosting provider at the time and requested IPMI/KVM access. Afterwards, I went to bed since I was exhausted.

 

This morning, they provided me credentials to the IPMI/KVM. After inspecting some logs within the IPMI portal itself, I discovered this was most likely a motherboard issue with the new machine according to one of the events triggered at the same time the machine went down. I told our hosting provider about this and they stated while the motherboard they had in the new machine is supposed to handle the load from the Intel i9-9900K, they had other clients have the same issue that we had along with the same event triggering in the IPMI console. The hosting provider had other machines available with the Intel i9-9900K, but I needed to install the OS myself through the KVM along with setup another machine again. For the last few hours, I've been doing this and got everything setup properly. This new (new) machine comes with a motherboard that is proven stable with the Intel i9-9900K.

 

I've moved the Rust Modded and CS:S Dust2 files over to the new machine, updated the Anycast network config, and the servers are running fine right now. I will continue to monitor the server. Thankfully I have full IPMI/KVM access which is nice :)

 

We have also been refunded for the lost time and granted an additional day of service for the above from our hosting provider.

 

@Xy will finish moving the other servers when he has time.

 

I just wanted to apologize for the inconvenience and thank you for your understanding with all of this. I'm trying my best to get us up and running as quick as possible.

Share this post


Link to post
Share on other sites


Hidden

Nice! Sounds like a headache to deal with. I wish you further luck. :)


╔                               ╗

Scott

                               

Share this post


Link to post

Thank you for the appreciation everyone :) It means a lot!

 

3 minutes ago, yuptodat said:

Nice! Sounds like a headache to deal with. I wish you further luck. :)

It was indeed a huge headache, but things appear to be stable for now!

 

The next thing I'm hoping that happens is the hosting provider's DC is able to remove the ACL rules preventing us from spoofing traffic out as the Anycast network. I need these ACL rules removed so we can use the TC BPF program I made that'll send outgoing IPIP packets back to the clients directly instead of back through the Anycast POP servers/IPIP tunnel. This would result in less load on the POP servers, less overage fees, eliminates a single-point-of-failure, and more!

 

Thank you!

Share this post


Link to post
Share on other sites


Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now


×
×
  • Create New...