[3-25-19] Physical Machine -> Dallas PoP Routing Issue

[Roy 10,832 / 0]

Hi everyone,

 

I just wanted to let you know that earlier today the routing from our physical machine to our Dallas PoP started having issues. For whatever reason, it started routing from Dallas to NYC to all the way back in Dallas. This is adding ~40ms latency overhead to the network.

 

We have contacted our physical hosting provider and they're contacting the peer they are using who should be able to fix this issue.

 

Once it is fixed, I will reply to this thread.

 

Thanks.

[Roy 10,832 / 0]

MTR example from physical hosting provider to Dallas PoP:

 

 Host                                                                                Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. x.x.x.x                                                 0.0%     8    0.8   0.8   0.6   0.9   0.0
 2. x.x.x.x                                                         0.0%     8    0.8   0.8   0.6   1.0   0.0
 3. 66-206-18-73.static.hvvc.us                                                       0.0%     8   12.4  14.8  11.7  22.2   3.4
 4. ???
 5. ???
 6. ???
 7. ???
 8. ???
 9. nyk-b5-link.telia.net                                                             0.0%     7   46.1  46.9  45.6  53.7   2.9
10. dls-b22-link.telia.net                                                            0.0%     7   63.7  63.7  63.6  63.8   0.0
11. vultr-ic-318447-dls-b22.c.telia.net                                              14.3%     7   65.3  64.7  64.1  65.8   0.6
12. ???

First two hops were removed.

 

As you can see, it is being routed from Dallas to NYC and all the way back to Dallas adding ~65ms latency overhead to the network which is terrible. Our physical hosting provider has confirmed that this is due to Hivelocity[AS29802] (as @Dreae expected) and contacted them regarding this issue.

 

We're hoping this is fixed soon!

 

Thanks.

[Roy 10,832 / 0]

Earlier, the physical server started routing to the NYC PoP for whatever reason. The route was using Cogent[AS174]. We already prepended to Cogent 1x hops. So we tried raising it to 2x hops via:
 

bgp_community.add ((64602,174));

 

In the BIRD configuration export filter on the NYC PoP. This changed the route back to an optimized route of the Dallas PoP!

 

 1. x.x.x.x                                                 0.0%    10    0.7   0.7   0.7   0.8   0.0
 2. x.x.x.x                                                         0.0%    10    0.8   0.8   0.6   0.9   0.0
 3. 66-206-18-73.static.hvvc.us                                                       0.0%    10   17.4  16.3  11.4  21.0   2.7
 4. ???
 5. ???
 6. ???
 7. ???
 8. 66-206-30-6.static.hvvc.us                                                        0.0%     9    0.5   0.5   0.5   0.6   0.0
 9. ae4-703.cr6-dal3.ip4.gtt.net                                                      0.0%     9    0.7   4.4   0.5  30.6   9.9
10. xe-7-2-0.cr0-dal2.ip4.gtt.net                                                     0.0%     9    0.9   0.8   0.7   0.9   0.0
11. ip4.gtt.net                                                                       0.0%     9    3.8   4.6   1.3  14.6   5.4
12. ???
13. ???
14. 92.119.148.15                                                                     0.0%     9    0.9   0.9   0.8   0.9   0.0

 

Let's hope it stays this way

 

This problem should be completely gone when we get our own ASN and get a PoP server using the same hosting provider as our physical machine.

 

Thanks.