Jump to content

Roy

Banned
  • Posts

    2,853
  • Joined

  • Last visited

  • Days Won

    383

Posts posted by Roy

  1. Hello all,

     

    I just wanted to talk about the possibility of selling services on our Anycast network. This includes any type of service including game servers as long as Compressor (developed by @Dreae) supports it. As of right now, there are no pricing estimates, but we're talking to a couple communities about this already. There is a possibility we'll make a deal with communities we are already talking to. However, this service won't be completely public for a while. Once/if it does go public, we'll have an entire website section dedicated to the service including the pricing, contacts, details, FAQs, etc.

     

    As many of you know, the Anycast network has been a VERY big project for myself and @Dreae. I can easily say it is the biggest and most exciting project I've worked on in GFL thus far.

     

    There are a few reasons we're considering this. The biggest reason is because GFL doesn't make enough money to support a great Anycast setup. Our budget is too low and due to this, we cannot purchase high-performance options from our hosting provider(s). If we start selling these services to other communities, we can potentially start putting in an extra hundreds of dollars (to thousands depending on how successful it is) a month into the network. My end-goal is to eventually have it so we can have either a very powerful VPS or a dedicated machine in every location. If it came down to it or was necessary, maybe even having a hardware firewall in-front of each PoP. This will allow us to extend Compressor's features along with being able to block larger (D)DoS attacks.

     

    Please keep in mind this is the end-goal. With that said, I want this having a community-feel to it. This means I would encourage the other clients to give advice or help develop the network and we would be able to work together on this (e.g. learning how to block attacks or improving Compressor). The reasoning for this is because all the money that our 'clients' spend on this will go into network and GFL only. I along with any others will NOT be making any profit off of this service. I'm into this for the technical experience it gives me and having it advance my career in networking and programming (that has been my drive this entire time, anyways).

     

    @Dreae is working on a control panel for Compressor. Keep in mind, this is a long time away, but it'll come in handy once this entire service does go public. It will allow clients to make very specific packet filtering rules for their IPv4 addresses only from the Anycast network.

     

    We may also be able to sell dedicated machines to clients as well (we'd be a re-seller at this point). I might go as far as colocating dedicated machines (building them, shipping them to the data center, and having them ran by the data center). However, that is WAY down the line and I don't know if I'd want to do all of this without making any personal profit from it (I still have to work a full-time job outside of GFL, lol). However, as for buying machines from hosting providers and offering them to clients, I am fine with that. Might even be able to get our clients an Intel i9-9900K for $149.00/m, lol.

     

    In the end, this is a WAYYYY in the future thing. Perhaps in the next six to twelve months depending on how things go. I'm sure there are a lot of other things I would need to change for this (e.g. our business license). I am going to be researching a lot on that and talking to some colleagues who are good at the business-side of things. 

     

    I made a poll for any current service owners that may be interested in this.

     

    If you have any questions or concerns, please let me know!

     

    More updates about this will be posted in the future.

     

    Thanks!

  2. @BlaX Are you still banned? I checked the server's ban list via listid and listip. You didn't show up in that list. The one thing I noticed is the SBan module on the server had an outdated config file in the server's data/ directory (I updated this). However, the lua file was up-to-date.

     

    What's strange is it banned you and put it into the correct database. Perhaps updating the data config file resolved this issue (will probably take effect on map change).

     

    Thanks.

  3. Hello all,

     

    I am aware users haven't been able to receive their Supporter, VIP, and Member perks in-game since the website move detailed here. I've identified the issue with our back-end system and fixed it. It is confirmed working now.

     

    With that said, I've added two weeks of free Supporter or VIP to the users who have purchased them after the web move (chances are, they didn't receive any perks in-game due to this issue). I've manually inputted all the Members that haven't received their perks into the database as well.

     

    If you're still not receiving your perks, please post here and contact the server's manager you're not seeing it on. Chances are, the manager hasn't updated the server's MySQL settings because I've confirmed everyone is in our database and set to the correct group. I will continue to monitor new Supporters, VIPs, and Members to ensure they are entered into our database correctly as well (already did this with one and confirmed they were entered successfully).

     

    I apologize for the delay on this. I was very busy this week with work and personal things and unfortunately nobody else was able to fix this.


    Thank you for understanding.

  4. Here are my current health issues + my health history:

     

     

    Also more information regarding my current health plus my past, basically I'm feeling stressed pretty much all the time from work and other things such as GFL (constant headaches or migraines, I don't know if I actually have a migraine or not). My stomach constantly runs into issues after eating 99% of any foods. I unfortunately get very little exercise as well which is something I want/need to work on. I had my gallbladder removed last year during February 2018 (emergency surgery since it ruptured unexpectedly), had hernia surgery last July (2018), and had a kidney stone during one of my business trips last September (2018). After getting my gallbladder removed, the doctors believe I have something called BAD (Bile Acid Diarrhea). I haven't been to the doctor since, though (I plan to go back once I get my car issues figured out, which I am making a lot of progress on this week :)). Just curious if these probiotics will most likely help with my current issues, but I do understand I have a more complex situation and I might have things wrong with me that probiotics won't help with. I'm just trying to think of any short-term solutions to make me feel better until I can get back to the doctor so we can dig deeper and find out what issues I'm facing.

     

     

    Putting it in a spoiler since some may find it disgusting or w/e (it's health issues, lol). 

     

    I'm sorry for the rushed post, I have a lot to do right now but wanted to know what people thought from GFL.

     

    Thanks!

  5. Hey,

     

    I was wondering if anyone takes probiotic supplements. I want to start taking them daily along with eating healthier/getting exercise to see if it improves my health. I want to know if anyone has had good experiences with these or not. I purchased Trunature's Probiotics from Costco yesterday. The reviews on it were very positive when I looked them up online. My colleague also said it has good effects on him.

     

    My health currently is really bad. Therefore, I really need to improve it.

     

    Thanks!

  6. Hello,

     

    IMPORTANT - The major networking issue from our physical servers to the Dallas PoP has now been fixed. Please read more about this here.

     

    Our GS09 machine is currently facing a major networking issue with the hosting provider. The first hop is constantly spiking to 400+ms ping which I believe to be one of Nexril's routers. The GS08 machine is operating without any networking issues, but I believe they're being routed through a different Nexril router.

     

    The following are servers this issue is affecting:

     

    • FoF Shootout.
    • TF2 2Fort.
    • CS:GO Surf Timer Beginner 85-tick.
    • GMod TTT Rotation.

     

    We have an open ticket with Nexril and have requested urgency regarding this case.

     

    I do apologize for the inconvenience and once I receive an update, I will let everyone know.

     

    Thank you.

  7. We have resolved this issue ourselves. We stopped announcing to Level 3 and Cogent to the New York City PoP. This then resulted in the physical servers routing back to the Dallas PoP. For some reason, this didn't work the night this started occurring (I basically put in the same BGP communities I had from the other night in). Perhaps it may have needed some propagation time. However, usually things are instant like today was. Either way, I don't really like not announcing to Level 3 and Cogent entirely due to them being Tier 1 peers. Unfortunately, we'll have to do this for now until Vultr improves their networking (which, at this point, doesn't seem like a high possibility).

     

    I do apologize for the delay on this and as soon as we have our own ASN, we are going to be putting measurements in-place to make sure this issue won't occur again. We've been trying to push Vultr to look into this deeper. However, there wasn't much success with that. I seriously can't wait until we acquire our own ASN :P 

     

    Considering most clients that were routing to the Dallas PoP started preferring a route over Level 3 to the NYC PoP, this was ultimately an issue with Vultr's Dallas networking.

     

    Anyways, things should be good now.

     

    With that said, there is currently a major issue with our GS09 machine. I will be posting about that shortly.

     

    Thank you.

  8. Hello everyone,

     

    In the last couple months or so, we've purchased a new web machine. This machine was only 99 cents more a month and has upgraded hardware components. This machine was the same package type as the old machine (SP-64 on OVH) which is why it was around the same price.

     

    Here are the old machine's specs:

     

    • Intel Xeon E5-1630 v3 @ 3.7 GHz.
    • 64 GBs of DDR3 RAM.
    • 2 x 300 GBs SSD.
    • 2 x 2 TBs HDD.
    • 1 gbp port.
    • $144.00/m.

     

    Here are the new machine's specs:

     

    • Intel Xeon E3-1270v6 @ 3.8 GHz.
    • 64 GBs of DDR4 RAM.
    • 2 x 480 GBs NVMe.
    • 2 x 2 TBs HDD.
    • 1 gbp port.
    • $144.99/m.

     

    This was obviously worth the extra $0.99/m, lol. The NVMe storage will certainly help with our MySQL server's performance as well.

     

    We're also restructuring everything with this new web machine. This includes completely rebuilding the back-end servers. We're looking to make things more efficient. I'm hoping this allows us to grant certain permissions to our developers that will allow them to do what they have to while keeping things restrictive and secure. One thing we did poorly with the old machine was over-allocate SSD space to certain servers that didn't need it. This resulted in us running out of SSD space to assign to servers that needed it and it's risky shrinking a disk because it can result in possible loss of data.

     

    Anyways, I've spent most of the weekend setting up the servers on the new machine and things are looking pretty much ready. I've already moved my personal websites to the new machine and things are running great (Browser.TF, Pop.Browser.TF, and g.gflclan.com). @Xy_ has been a great help and we're in the process of moving things over. At some point today, we will be moving this website over to the new machine. This will result in a couple hours of down-time. Last night @Xy_ attempted this already, but ran out of time. Therefore, we will have to resume the move after he gets home today. We will also be moving SourceBans at the same time since it'll make the website transition smoother (i.e. we have functions within the website that relies on the internal connection to the SourceBans database). This will require edits to the database information the game servers point to.

     

    I would like to move everything before the end of the month. If this is the case, we won't have to pay for the old web machine again.

     

    Other things we will have to move include the following (thinking off the top of my head):

     

    • HLStatsX.
    • Beta websites.
    • GitLab.
    • TeamSpeak 3.
    • Other databases used by game servers.
    • Discord bots.

     

    There is a chance we may miss something or bugs will be introduced after these moves. If you discover a bug, please report it to a Director or @Xy_ and I.

     

    Thank you.

     

    View full article

     

  9. Hi all,

     

    I just wanted to let you all know I'm going on a business trip from June 4th to June 7th.

     

    I will still have my laptop on me, but just like the last business trip I had, I won't really be available often during the day and possibly at night since I'll be spending time with my colleagues.

     

    Thanks!

  10. Hello everyone,

     

    Around an hour and a half ago, we started experiencing yet another major issue with Vultr's Dallas network. Our physical hosting provider (Nexril) started routing to the New York City PoP server instead of the PoP server located right in Dallas, TX. I also noticed my home network's route has changed from Dallas, TX to NYC. After trying to mess with Vultr's BGP communities on the NYC and Dallas PoPs to see if I could get it back, I witnessed there was little to no traffic on our Dallas PoP:

     

    466-05-19-2019-aJgJYEoG.png

     

    Therefore, I decided to stop Compressor on the Dallas PoP and run a tcpdump for our Anycast network. There was one poor client who I believed was connected to one of our game servers through the Dallas PoP still (assuming by the amount of port 27015 packets they were sending, which was nearly 30 - 50 per second). They more than likely timed out during this session twice :( Anyways, here's the output of the tcpdump command while excluding the client mentioned before:

     

    roy@da01:~$ sudo tcpdump -i any net 92.119.148.0/24 and not host xxx.xxx.xxx.xxx -nn
    tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
    listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
    06:08:45.369813 IP xxx.xxx.xxx.xxx.55013 > 92.119.148.239.5984: Flags [S], seq 2175052956, win 65535, length 0
    06:08:58.826801 IP xxx.xxx.xxx.xxx.55028 > 92.119.148.10.27015: UDP, length 25
    06:08:59.549474 IP xxx.xxx.xxx.xxx.55028 > 92.119.148.19.27015: UDP, length 25
    06:08:59.862646 IP xxx.xxx.xxx.xxx.55028 > 92.119.148.30.27015: UDP, length 25
    06:09:04.985738 IP xxx.xxx.xxx.xxx > 92.119.148.33: ICMP xxx.xxx.xxx.xxx udp port 7777 unreachable, length 70
    06:09:28.129399 IP xxx.xxx.xxx.xxx.46212 > 92.119.148.38.5984: Flags [S], seq 209625597, win 65535, length 0
    06:09:54.208208 IP xxx.xxx.xxx.xxx.61494 > 92.119.148.56.53: 37944+ A? 153edd71.openresolverproject.org. (50)

    As you can see, we only received 7 total packets in the span of over a minute. Other PoP servers normally have 400 - 500+ packets at this point from what I've seen.

     

    I've submitted a ticket to Vultr about this and requested this to be looked into ASAP. We will be moving off of Vultr once we acquire our own ASN in most locations more than likely due to a number of reasons (I'll write about this more in the next big network update). Once we do, we shouldn't have this issue again because we'll have a PoP server with the same hosting provider as the physical hosting provider.

     

    This is more than likely causing an additional 50 - 100+ms ping for clients connected to our game servers under our Anycast network.

     

    I've also submitted a ticket to Nexril regarding this to see if there's anything they can do. However, I highly doubt it considering there's only a few hosts that is routing to our Dallas PoP server right now, LOL.

     

    I do apologize for the inconvenience this has caused. For whatever reason, Vultr's NYC PoP was experiencing packet loss as well from the network (thanks, Vultr). Therefore, all of our clients were also experiencing this as well. This is a still a new network to us and there are many upgrades we need to make to it in the future (all of this will start happening once we acquire our own ASN which should be very soon).

     

    Thank you for understanding and I will post back here once I receive an update or the issue resolves itself.

  11. Hello,

     

    Our physical hosting provider (Nexril) experienced an issue that occurred twice with the network in the last thirty minutes or so. This resulted in our machines timing out for 1 - 2 minutes and disconnecting pretty much all players from our servers under the Anycast network (92.119.148.0/24).

     

    Here's what we have from our physical hosting provider:

     

    Quote

    James Today at 10:33 AM
    Hivelocity appears to have gone down ... investigating the issue now.
    Seems to hive been a routing blip, everything appears to be fine now. Will investigate the issue and post updates accordingly.

     

    As I receive updates, I will post them here.

     

    I apologize for the inconvenience and thank you for understanding,

  12. Hello,

     

    Compressor on the Miami PoP server went down at some point yesterday (around 19 hours ago). I have rebooted it and it is now accepting connections again. If you weren't able to connect/view the servers on the Anycast network, this might be a reason why.

     

    We won't be running into this issue as soon as we upgrade our network which will be happening at some point soon. With that said, we'll be looking to add some sort of monitoring system so we get notified if one of our PoP servers go down automatically (right now I check the status of Compressor on all PoP servers via a bash script I made).

     

    I apologize for the delay and thank you for understanding.

  13. Hi all,

     

    I just wanted to address a server crash both our 2Fort and High Tower servers experience when performing 2 - 3 map changes over the span of 3 - 5 days. The server crashes as soon as the map changes and there is no engine error before the crash. Here is the stack trace:

     

    Program terminated with signal SIGABRT, Aborted.
    #0  0xf7f17029 in __kernel_vsyscall ()
    [Current thread is 1 (Thread 0xd7cffb40 (LWP 84))]
    #0  0xf7f17029 in __kernel_vsyscall ()
    #1  0xf7d5eea9 in raise () from /lib/i386-linux-gnu/libc.so.6
    #2  0xf7d60407 in abort () from /lib/i386-linux-gnu/libc.so.6
    #3  0xf7d9a37c in ?? () from /lib/i386-linux-gnu/libc.so.6
    #4  0xf7e2a6b8 in __fortify_fail () from /lib/i386-linux-gnu/libc.so.6
    #5  0xf7e288a8 in __chk_fail () from /lib/i386-linux-gnu/libc.so.6
    #6  0xf7e2a5aa in __fdelt_warn () from /lib/i386-linux-gnu/libc.so.6
    #7  0xf65ff276 in CBlockingUDPSocket::WaitForMessage(float) () from /home/container/bin/engine_srv.so
    #8  0xf6724861 in Win32UploadGameStatsBlocking(TGameStatsParameters const&) () from /home/container/bin/engine_srv.so
    #9  0xf67253f7 in CAsyncUploaderThread::ThreadProc() () from /home/container/bin/engine_srv.so
    #10 0xf6725aa1 in CAsyncUploaderThread::CallbackThreadProc(void*) () from /home/container/bin/engine_srv.so
    #11 0xf7eef295 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
    #12 0xf7e1a05e in clone () from /lib/i386-linux-gnu/libc.so.6

     

    This is a very odd crash considering it's calling Linux functions. The only entries that refer to the Source Engine itself is towards the end of the stack trace (e.g. "CBlockingUDPSocket:WaitForMessage(float)"). I might try to learn more about GDB and how to inspect the core dump file. Specifically, I'm curious about how to retrieve the float parameter's value in the CBlockingUDPSocket:WaitForMessage function. I plan to open the engine_srv.so file in IDA and look for this function to see if I can gain more information on what it exactly does.

     

    Thank you!

×
×
  • Create New...