Hi.
I have a website hosted in Buffalo, behind Cloudflare. I’m getting alerts from Cloudflare that they can’t connect to the host from Miami.
Is there any way to test the connection from my host to the Cloudflare servers?
Ask them to provide you with a test IP for mtr.
2 Likes
Odd I have Virmach TX with CF and works nicely but Buffalo is like same thing as saying CC HQ 
1 Like
Not even sure if that will work to test it… AFAIK, Cloudflare use anycast with a lot of their IP ranges, so the same IP can actually have different servers in different regions. I used NTT’s and Cogent’s looking glasses to ping a Cloudflare-hosted domain. Both returned 104.31.92.213 and had very low pings (~0.5ms) with short traceroutes (3 hops). In NTT’s case, the router directly before the final hop is still on their network (ae-0.cloudflare.miamfl02.us.bb.gin.ntt.net), so it’s likely a PoP directly on their network, or at least they’re directly peering to Cloudflare within the same DC.
However, when I traceroute that IP from my current location (San Francisco Bay Area) it goes to Seattle, and when I traceroute it from a server in Melbourne Australia, it goes to Sydney.
I’m seeing the same thing for other Cloudflare source IPs in access logs on my server too.
Anyways, sorry for the wall of text… They might actually be able to give you a Miami-specific IP to test with
Otherwise… I don’t know which transit providers Cloudflare use in Miami (or if they’re directly peering with whatever provider you’re using in Buffalo), but you could try traceroute your server’s real IP using Miami-based looking glasses (like the NTT and Cogent ones I linked to earlier) and see if you notice anything odd.
3 Likes
Aye, but even if he got anycast IP he could mtr it to find where the path becomes faulty. Problem is that CF can return different IP when DNS is queries from London and a different one when queried say from Washington, so it would be good to know what to ping…
1 Like
The issue is that the mtr results would differ depending on where you perform it from. Cloudflare could tell you “we’re accessing your server from 104.31.92.213 in Miami and it’s not working”, but then you traceroute that exact same IP and hit a PoP in Seattle, or in New York, or somewhere else, with no issues.
There’s a lot I don’t know about Anycast… One thing I don’t quite understand is how to debug connection issues when just a single PoP is having problems. I can’t think of a way to explicitly route the request to one particular PoP, as it’s reliant on your upstream provider’s routing tables 
. Perhaps you have some ideas? As a server provider, I’m sure you’d have more knowledge about it than I do 
I don’t think the IP address they use to connect to your server from each location are anycast.
The IP they return for your site is.
2 Likes
They should be able to give you a IP to test, one of my site I got on Cloudflare the request for it from home get’s routed to Frankfurt POP.
The connecting IP from CF I got from lsof wasn’t a anycast ip tested traceroute from multiply locations routes to Frankfurt.
You could be right… I’ve seen a few anycast IPs in my logs, but the sites I have behind Cloudflare don’t get a lot of traffic, so it’s possible that not all the IPs are anycast IPs.
These are the ranges they connect from, they appear to be static routes from a few random IPs I tested on the bigger ranges (incrementing by /24 and trying to ping the gateway on each, seeing tokyo, sydney, santa clara, etc)
Not sure where to find miami in that mess though
Yeah it seems like static routes and anycast routes are all mixed together and they don’t clearly say which is which The anycast IP I mentioned earlier in this thread is in one of the ranges on that page (104.16.0.0/12). I’ve also seen 104.18.126.100, which is in the same range and also appears to be an anycast IP. Perhaps just 104.16.0.0/12 is the anycast range? No clue.
I wish they had clearer listings on that page. Like, “for connectivity to Miami, test IP x.x.x.x”. I guess even then it’s hard, as different IPs may be in the same region but have different routing. Hmm.
I’m glad I don’t deal with web hosting any more. Networking and routing is tricky. ¯\_(ツ)_/¯
1 Like
Anycast at CloudFlare is used for incoming public facing traffic. They obviously don’t use anycast for connecting to your site.
2 Likes
TCP handshake requires two way communication, so whatever way this is reaching the content server, there has to be a way to connect back and so use
it for debugging.
This is actually one of the pain points my employer hit with anycast when trying to use it for a CDN. Since multiple nodes have the same IP address, changes to the routing can result in packets from an open connection (or even one currently in the middle of a handshake) suddenly going to a different server, which has no idea about the connection state. From Wikipedia:
The pitfall of this approach is that a connection to an anycast address may fail because the network can change the routing of packets in mid-connection due to congestion or changes in the network, with the result that the destination changes mid-connection, though the new destination is not aware of the connection and is not maintaining the connection state. These conditions are typically referred to as a “PoP switch”. With a normal unicast address a routing change would not be a problem, as this merely results in a different route to the same eventual destination. All packets within a connection do not normally need to follow the same path. But when the address is actually an anycast address masquerading as a unicast address (as in this design), a routing change could be a destination change.
Because of the possibility of a “PoP switch” in IPv4, anycast is generally used with connectionless protocols based on UDP, as a way to provide high availability and load balancing for stateless services.
Cloudflare likely have some proprietary thing to avoid these issues. My employer just ended up using different hostnames for their CDN nodes, based on location.
This is what I thought, too, but one of the IPs I mentioned earlier (104.18.126.100) came from my access logs 
I do see many non-anycast IPs there though, so maybe it doesn’t happen all the time? Hmm. I don’t know. This confuses me, I should probably leave it to the experts
2 Likes
Note that this usually doesn’t happen, internet routing is pretty stable. When this becomes a pain point is when doing per packet load balancing sending packets over different ISPs.
4 Likes
That makes sense. Thanks for the info!
1 Like