Nginx logs two upstream statuses for one upstream
I'm a big fan (and user) of Nginx.
Just occasionally, though, you'll find something that looks a little odd - despite having quite a simple underlying explanation.
This one falls firmly into that category.
When running NGinx using ngx_http_proxy_module (i.e. using proxy_pass
), you may sometimes see two upstream status codes recorded (specifically in the variable upstream_status
) despite only having a single upstream configured.
So assuming a logformat of
'$remote_addr\t-\t$remote_user\t[$time_local]\t"$request"\t' '$status\t$body_bytes_sent\t"$http_referer"\t' '"$http_user_agent"\t"$http_x_forwarded_for"\t"$http_host"\t$up_host\t$upstream_status';
You may, for example, see a logline line this
1.2.3.4 - - [11/Jun/2020:17:26:01 +0000] "GET /foo/bar/test/ HTTP/2.0" 200 60345109 "-" "curl/7.68.0" "-" "testserver.invalid" storage.googleapis.com 502, 200
Note the two comma-seperated status codes at the end of the line, we observed two different upstream statuses (though we only passed the 200
downstream).
This documentation helps explain why this happens.
Example Config
We'll use an incredibly simple nginx config sample here, with a single location
block
location / {
set $backend storage.googleapis.com;
# Force periodic re-resolution of the upstream
resolver 127.0.0.1;
# Send the request
proxy_pass https://$backend;
}
We can see that storage.googleapis.com only returns a single A record
host -t A storage.googleapis.com storage.googleapis.com has address 172.217.168.240
So let's first look at the variable and it's documentation
upstream_status
The Nginx documentation (for http_upstream) describes the variable upstream_status
as follows
keeps status code of the response obtained from the upstream server. Status codes of several responses are separated by commas and colons like addresses in the $upstream_addr variable. If a server cannot be selected, the variable keeps the 502 (Bad Gateway) status code.
Dual Upstream from a Single Host?
We can see from the Nginx documentation that where there are multiple upstreams, $upstream_status
might become comma seperated, but how exactly does this happen when we've got a single upstream configured?
The answer is: IPv6 and IPv4 addresses are treated as separate upstreams and our upstream (in this example Google's API) support both IPv4 and IPv6 (keep reading even if you've disabled IPv6 in your own OS)
host storage.googleapis.com storage.googleapis.com has address 172.217.168.240 storage.googleapis.com has IPv6 address 2a00:1450:400e:80d::2010
If we log $upstream_address
we see that the IPv6 address has first been tried and failed
storage.googleapis.com [2404:6800:4003:c04::80]:80, 172.217.194.128:80 502, 200
Where this gets particularly fun (and why I said to keep reading), is that this can (and will) still happen when Nginx is running on a server with IPv6 disabled.
sysctl net.ipv6.conf.all.disable_ipv6
net.ipv6.conf.all.disable_ipv6 = 1
The flow of events is
- Nginx queries the upstream name
- Both an
A
and anAAAA
record is received (Google's storage API has an IPv6 address) - NGinx attempts to connect to the IPv6 upstream
- The connection fails immediately because the system has IPv6 disabled
- Because it's a failed connection, the status is recorded as
502
(as per the documentation snippet above) - Nginx then connects out to the IPv4 upstream (successfully in this case)
Stop Nginx attempting IPv6 Connections
Preventing this is simple, once you know how.
When using Nginx's resolver
directive, you need to tell it not to query IPv6 records.
resolver 127.0.0.1 ipv6=off;
The documentation for resolver
makes it clear that this is deliberate
By default, nginx will look up both IPv4 and IPv6 addresses while resolving. If looking up of IPv6 addresses is not desired, the ipv6=off parameter can be specified.
Conclusion
So, if you've seen multiple upstream status codes being logged, or an otherwise IPv6 incapable box trying to use IPv6 to connect to NGinx's upstreams, your issues almost certainly lie within your Nginx config.
It's a little non-intuitive when you first look at it, but quickly makes sense when you start logging $upstream_addr
.
Many applications prefer IPv6 over IPv4 connections, so never assume that they won't try and use it just because IPv6 isn't available in the OS (or, indeed, on the network).