Nginx logs two upstream statuses for one upstream

I'm a big fan (and user) of Nginx. 

Just occasionally, though, you'll find something that looks a little odd - despite having quite a simple underlying explanation.

This one falls firmly into that category.

When running NGinx using ngx_http_proxy_module (i.e. using proxy_pass), you may sometimes see two upstream status codes recorded (specifically in the variable upstream_status) despite only having a single upstream configured.

So assuming a logformat of

'$remote_addr\t-\t$remote_user\t[$time_local]\t"$request"\t'
'$status\t$body_bytes_sent\t"$http_referer"\t'
'"$http_user_agent"\t"$http_x_forwarded_for"\t"$http_host"\t$up_host\t$upstream_status';

You may, for example, see a logline line this

1.2.3.4	-	-	[11/Jun/2020:17:26:01 +0000]	"GET /foo/bar/test/ HTTP/2.0"	200	60345109	"-"	"curl/7.68.0"	"-"	"testserver.invalid"	storage.googleapis.com	502, 200

Note the two comma-seperated status codes at the end of the line, we observed two different upstream statuses (though we only passed the 200 downstream).

This documentation helps explain why this happens.

Example Config

We'll use an incredibly simple nginx config sample here, with a single location block

location / {

    set $backend storage.googleapis.com;

    # Force periodic re-resolution of the upstream
    resolver 127.0.0.1;

    # Send the request
    proxy_pass   https://$backend;
}

We can see that storage.googleapis.com only returns a single A record

host -t A storage.googleapis.com
storage.googleapis.com has address 172.217.168.240

So let's first look at the variable and it's documentation

upstream_status

The Nginx documentation (for http_upstream) describes the variable upstream_status as follows

keeps status code of the response obtained from the upstream server. Status codes of several responses are separated by commas and colons like addresses in the $upstream_addr variable. If a server cannot be selected, the variable keeps the 502 (Bad Gateway) status code.

Dual Upstream from a Single Host?

We can see from the Nginx documentation that where there are multiple upstreams, $upstream_status might become comma seperated, but how exactly does this happen when we've got a single upstream configured?

The answer is: IPv6 and IPv4 addresses are treated as separate upstreams and our upstream (in this example Google's API) support both IPv4 and IPv6 (keep reading even if you've disabled IPv6 in your own OS)

host storage.googleapis.com
storage.googleapis.com has address 172.217.168.240
storage.googleapis.com has IPv6 address 2a00:1450:400e:80d::2010

If we log $upstream_address we see that the IPv6 address has first been tried and failed

storage.googleapis.com	[2404:6800:4003:c04::80]:80, 172.217.194.128:80	502, 200

Where this gets particularly fun (and why I said to keep reading), is that this can (and will) still happen when Nginx is running on a server with IPv6 disabled.

sysctl net.ipv6.conf.all.disable_ipv6
net.ipv6.conf.all.disable_ipv6 = 1

The flow of events is

• Nginx queries the upstream name
• Both an A and an AAAA record is received (Google's storage API has an IPv6 address)
• NGinx attempts to connect to the IPv6 upstream
• The connection fails immediately because the system has IPv6 disabled 
• Because it's a failed connection, the status is recorded as 502 (as per the documentation snippet above)
• Nginx then connects out to the IPv4 upstream (successfully in this case)

Stop Nginx attempting IPv6 Connections

Preventing this is simple, once you know how.

When using Nginx's resolver directive, you need to tell it not to query IPv6 records.

resolver 127.0.0.1 ipv6=off;

The documentation for resolver makes it clear that this is deliberate

By default, nginx will look up both IPv4 and IPv6 addresses while resolving. If looking up of IPv6 addresses is not desired, the ipv6=off parameter can be specified.

Conclusion

So, if you've seen multiple upstream status codes being logged, or an otherwise IPv6 incapable box trying to use IPv6 to connect to NGinx's upstreams, your issues almost certainly lie within your Nginx config.

It's a little non-intuitive when you first look at it, but quickly makes sense when you start logging $upstream_addr.

Many applications prefer IPv6 over IPv4 connections, so never assume that they won't try and use it just because IPv6 isn't available in the OS (or, indeed, on the network).