[Go] Pterodactyl Game Server Watch

[Roy 10,832 / 0]

Hey everyone,

 

I just wanted to share my newest project! I decided to switch up for one day and instead of programming in C, program in Go/GoLang. This Go tool sends A2S_INFO requests to specified servers inside of a config file. If the server fails to respond to x amount of A2S_INFO requests, it will attempt to kill the server and start it again via the Pterodactyl API.

 

This is helpful for servers that are hung meaning they technically didn't crash, but something is taking up so many resources that the server is unplayable or offline (most likely a poorly coded addon with an infinite loop).

 

I'd like to mention I'm still fairly new to Golang. There are probably improvements I can make but the program ultimately works and I've tested it!

 

Here's the information!

 

Description
A tool programmed in Go to automatically restart 'hung' game servers/containers via a Pterodactyl API (version 0.7). This only supports game servers that respond to the A2S_INFO query (a Valve Master Server query).

 

Config File
The config file's default path is /etc/pterowatch/pterowatch.conf. This should be a JSON array including the API URL, token, and an array of servers to check against. The main options are the following:

• apiURL => The Pterodactyl API URL.
• token => The bearer token to use when sending HTTP POST requests to the Pterodactyl API.
• servers => An array of servers to check against (read below).

 

The servers array should contain the following members:

• enable => If true, this server will be scanned.
• ip => The IP to send A2S_INFO requests to.
• port => The port to send A2S_INFO requests to.
• uid => The server's Pterodactyl UID.
• scantime => How often to scan a game server/container in seconds.
• maxfails => The maximum amount of A2S_INFO response failures before attempting to restart the game server/container.
• maxrestarts => The maximum amount of times we attempt to restart the server until A2S_INFO responses start coming back successfully.
• restartint => When a game server/container is restarted, the program won't start scanning the server until x seconds later.

 

Configuration Example
Here's an configuration example in JSON:

 

{
        "apiURL": "https://panel.mydomain.com",
        "token": "12345",

        "servers": [
                {
                        "enable": true,
                        "ip": "172.20.0.10",
                        "port": 27015,
                        "uid": "testingUID",
                        "scantime": 5,
                        "maxfails": 5,
                        "maxrestarts": 1,
                        "restartint": 120
                },
                {
                        "enable": true,
                        "ip": "172.20.0.11",
                        "port": 27015,
                        "uid": "testingUID2",
                        "scantime": 5,
                        "maxfails": 10,
                        "maxrestarts": 2,
                        "restartint": 120
                }
        ]
}

Building
You may use git and go build to build this project and produce a binary. Example:

 

git clone https://github.com/gamemann/Pterodactyl-Game-Server-Watch.git
cd Pterodactyl-Game-Server-Watch/src
go build

Credits

• @Roy - Creator.

 

GitHub Link

 

Thanks!

[Roy 10,832 / 0]

I'd like to mention this can be used for GFL's servers as well. I am going to ask one of our SMs if they would like it setup as a test. Once it's proven stable, etc. we can start adding it to more servers. I've noticed sometimes our servers get hung and aren't restarted in hours. Thankfully this should resolve that

 

We'd set the IPs to our server's internal IPs and have the program running on our machines hosting the game servers themselves. Although the server's IPIP tunnels are inside a Docker container, you can send A2S_INFO requests to them from the host still. Since we cache the A2S_INFO response on our POP servers, an attack targeting the A2S_INFO query specifically won't be able to make this program have false-positives.

 

Thanks!

[annoying furry 6,300 / 63,950]

HEY @Nick IT'S WRITTEN IN GO!!!!!!

[Roy 10,832 / 0]

3 minutes ago, Xy said:

HEY @Nick IT'S WRITTEN IN GO!!!!!!

@Nick is going to despise me for this, but #YOLO! I just wanted to get into some higher-level languages for a temporary change lmao. Plus, this project benefits GFL  

[_Rocket_ 656 / 8,277]

The moment I saw Go in the title, @Xy popped in my head immediately. I could just hear and see him bust down a wall while wearing a sign saying "GOLANG"

Edited May 10, 2020 by _Rocket_

[Roy 10,832 / 0]

I made a few big changes to this project today. Changes may be found below.

 

• Updated the code to support the latest Pterodactyl API (1.4.2).
• Updated the structure.
• Added an option to retrieve all servers from the Pterodactyl API automatically.
• Added egg variable overrides for server-specific configs with servers pulled from the API.
• Organized code.

 

The egg variable overrides may be found here and includes the server IP/port. There's a high chance GFL will use something like this in the future to automatically restart 'hung' servers (servers where the process is still up, but the server is down due to it being pegged at 100% CPU from an addon for example). The egg variable overrides will allow server managers+ to disable the watcher for the specific server or tune its values if things are too or not enough sensitive. Since GFL's servers have an internal/LAN IP due to our Anycast configuration, the PTEROWATCH_IP egg variable override will be important in our setup.

 

I am still working to see if I can find something other than the A2S_INFO query to detect whether a server is down or not. This is because technically without caching the A2S_INFO query, an attack could exceed the Source Engine limits (which is configurable, but default limits are low, especially for attacks with high amounts of packets per second) and cause the query to timeout for a period of time. In this case, it could easily spark false-positives which would cause the server to restart with no reason.

 

My goal is to find a way to detect if an application is hung to the point where we can do it for any application and not game servers specifically. I'll be studying the Pterodactyl API in the next few days to see if this is possible or not.

[Roy 10,832 / 0]

I've made more big changes to the project.

 

• Implemented Slack support for web hooks inside of misc options.
• Added a reload timer that'll reload the configuration + servers.
• Added settings to set the default configurable for servers added via the Pterodactyl API.
• Added debug levels (1-4).
• Added a "report only" server option that makes it so only debugging/misc options are executed when a server is down instead of the server being restarted as well (great for first time use and testing for a few days to ensure there are no false-positive and whatnot).

 

The report only option will be useful in our case so we can enable it on all servers and then have it only report for a period of time. This will help narrow down any false-positives (and no actions will actually occur) and I'm going to look into implementing this on GFL.