Hetzner Falkenstein Packet Loss to Texas USA

#1

Packet loss is now making my Hetzner box almost unusable. Does anyone know much about mitigating this?

About a month ago I started getting massive packet loss between my residence in Texas and my Hetzner box in Falkenstein. Wireshark shows lots of TCP Out-Of-Order and TCP Dup ACK and most of the time I get less than 1Mbit throughput Hetzner to Texas. The traceroute goes from Heztner, telia.net, and then Comcast. Outbound from Texas goes through core-backbone.

I tried playing with TCP Window memory size and it didn’t help and I don’t think that’s the issue anyway after looking at other long distance connections that work without problems.

To confuse things further, I get 100Mbit from Hetzner Helsinki without problems. The IPv4 traceroute to there is via telia.net and I have almost no packet problems, and IPv6 is via core-backbone with lots of packet loss but evidently not enough that’s noticeable and throughput is still 100Mbit +.

So I don’t know what else to try and I’d hate to give up my Hetzner box in Falkenstein because it’s been great and very cheap for over a year now. And it used to work great for streaming and even a slightly laggy desktop VM or two.

Anyone got any ideas to resolve this? Or is it just “Well, that’s the internet. Tough luck.”

#2

Did you customize your sysctl.conf ?

Could simply be SYN / SYNACK values above 15 or SYN cookies disabled.

Especially on a busy box.

#3

Is it your box or all Hetzner FSN?

I can do a traceroute/etc if needed from FSN if it helps work out if it’s just your box or all of FSN

#4

I did that only for the TCP Window memory up to 6MB max.

Not sure how to manage the SYN and SYNACK values; I’ll research that.

I think it’s from all of Hetzner Germany. I’ve tried pulling the 100MB test file from Hetzner and I get the same problems. I’ve also tried cloud servers in Nuremberg and Falkenstein and had the same issues.

#5

Best you can do is take some mtr tests and put in a ticket with Hetzner about the issues you’re having. Maybe they can alter something in the peering or take a look at what might be causing it. I know they did a lot of network twiddling to get things right when they launched Helsinki and were responsive to weird routing reports while that was being sorted, etc.

#6

Wow I completely forgot about mtr. Packet loss 70% & 50% at a couple of telia.net routers.

I’ll run a few more tests both directions and to different servers and open a ticket with Hetzner. Maybe they have some advice or can check with telia.

1 Like
#7

Since I have quite a few :sweat_smile: servers in FSN I’m interested to double check if there are some routing issues to your ISP as well, if you have looking glass or an IP I could mtr I’ll give it a look too

#8

Here’s a comcast router that’s a couple hops from me: 68.86.89.206. An mtr from FSN shows lots of packet loss on a couple of telia.net routers on the US side.

I assume that’s what’s causing problems.

#9

To help give you data points, I’m in east Texas and most of my servers are in that same datacenter. This is an MTR to one:

12%20PM
12%20PM780×452 58.4 KB

The reason I bring this up is that I want you to be sure it isn’t just your ISP or an upstream provider they’re using. This could be a call to your ISP before a ticket to Hetzner. I don’t know if you’ve ruled that out, but it’s the right thing to assume as a possibility before doing so.

2 Likes
#10

Ran another from a server in Dallas and it gave me a good example of when packet loss is okay:

45%20PM
45%20PM780×452 41 KB

You see that high packet loss at point 3. In this case it’s okay because it doesn’t carry forward and the end point still has 0% loss. So this means the level3 routers are not responding to ICMP (go figure, level3). If it carried forward from that point all the way to the end, that’s how you’d know it was most likely the point of blame.

Things you probably already know but I like to leave around for readers.

1 Like
#11

Here are a few mtr -w --first-ttl 2 68.86.89.206 from dedis in FSN

FSN-sa
FSN-sa889×286 60.4 KB

FSN-no
FSN-no896×298 62.4 KB

FSN-de
FSN-de892×303 57.3 KB

FSN-ce
FSN-ce895×300 62.2 KB

(all with the same second hop btw) It seems ok to me
As Jarland pointed out, if it’s carried forward it’s ok, it’s not uncommon at all to have routers displaying some sort of packet loss when doing traceroutes. It’s usually ICMP rate limiting

1 Like
#12

Thanks for the testing @mfs (and Jarland) those match what I get almost exactly. Packet loss is not carried forward past those telia routers.

So it’s not a router; what’s left? Any idea why else would I get all the TCP Out-Of-Order and TCP Dup ACK and TCP Previous segment not captured from any server [I tested] at FSN and NUB but not HEL?

#13

it’s a shot in the dark, but you could check if this applies to you (you’d have to check if you’re using the e1000e driver beforehand)

#14

Looks like I’ve got a Realtek network card. I don’t think it’s my server anyway; I’ve tried from a few of their cloud servers and got the same results.

#15

Mh I see routing is a little different

FI-routing
FI-routing890×362 76.7 KB

Hinting either Hetzner or your ISP to give a look at it may help

#16

I doubt Comcast / Xfinity would even respond. Maybe I’ll bother Hetzner and see what they say.

Sadly my solution so far has been to use a VPN so I get a different route. I usually get between 30-50 Mbit that way.

#17

No issues here, I’m on AT&T Fiber in DFW. I did notice SSH would lag more noticeably than usual recently but I didn’t actually check for packet loss at the time, so I don’t know if it was loss or not (maybe just high load on the systems with the high latency is the problem, idk), but right now there’s no loss after a few minutes ICMP/TCP.

#18

Thank you for running tests/doing some more investigation first. One of the things our networking team often asks for first from customers with these kinds of questions is for mtr in both directions. --Katie

#19

I am not sure myself what they will say here, and if they can help, so I’d suggest writing a support ticket. They focus on answering tickets in their queues and are not active on social media, so please give them the information you’ve already shared here.