Nginx logs two upstream statuses for one upstream

I'm a big fan (and user) of NGinx

Just occasionally, though, you'll find something that looks a little odd - despite having quite a simple underlying explanation.

This one falls firmly into that category.

When running NGinx using ngx_http_proxy_module (i.e. using proxy_pass), you may sometimes see two upstream status codes recorded (specifically in the variable upstream_status) despite only having a single upstream configured.

So assuming a logformat of


You may, for example, see a logline line this	-	-	[11/Jun/2020:17:26:01 +0000]	"GET /foo/bar/test/ HTTP/2.0"	200	60345109	"-"	"curl/7.68.0"	"-"	"testserver.invalid"	502, 200

Note the two comma-seperated status codes at the end of the line, we observed two different upstream statuses (though we only passed the 200 downstream).

This documentation helps explain why this happens.

Read more ...

Building a Raspberry Pi Based Music Kiosk

I used to use Google's Play Music to host and play our music collection.

However, years ago, I got annoyed with Google's lacklustre approach to shared collections, and odd approach to VMs. So, our collection migrated into a self-hosted copy of Subsonic.

Other than a few minor frustrations, I've never looked back.

I buy my music through whatever music service I want, download it onto the NFS share and Subsonic picks up on it following the next library scan - we can then stream it to our phones (using DSub), to the TV (via a Kodi plugin) or to a desktop (generally, using Jamstash). In the kitchen, I tend to use a bluetooth speaker with the tablet that I use to look up recipes.

However, we're planning on repurposing a room into a puzzle and playroom, so I wanted to put some dedicated music playback in there.

Sonos devices have Subsonic support, but (IMO) that's a lot of money for something that's not great quality, and potentially has an arbitrarily shortened lifetime.

So, I decided to build something myself using a Raspberry Pi, a touchscreen and Chromium in kiosk mode. To keep things simple, I've used the audio out jack on the Pi, but if over time I find the quality isn't what I hope, it should just be a case of connecting a USB soundcard in to resolve it.

There's no reason you shouldn't be able to follow almost exactly the same steps if you're using Ampache or even Google Play Music as your music source.

Read more ...

Nintendo Switch Joycon Analog Stick Replacement

Over time the analog sticks on the Nintendo Switch's joycons tend to fail, usually resulting in drift (i.e. it generates movement despite the stick being dead centre).

This used to be an issue on N64 controllers too, though in fairness to Nintendo the sticks do take a hell of a beating, particularly if you're playing Super Smash Bros. On the other hand, outside of the 2 year warranty period Nintendo are downright predatory on their pricing for what's actually quite a simple fix.

This documentation details how to replace the analogue stick on a Nintendo Switch Joycon - there's a video of the process at the bottom for those who prefer video tutorials

All images are clicky.

Read more ...

Resolving GFID mismatch problems in Gluster (RHGS) volumes

Gluster is a distributed filesystem. I'm not a massive fan of it, but most of the alternatives (like Ceph) suffer with their own set of issues, so it's no better or worse than the competition of the most part.

One issue that can sometimes occur is Gluster File ID (GFID) mismatch following a split-brain or similar failure.

When this occurs, running ls -l in a directory will generally lead to I/O errors and/or question marks in the output

ls -i
ls: cannot access ban-gai.rss: Input/output error
? 2-nguoi-choi.rss ? game.rss

If you look within the brick's log (normally under /var/log/glusterfs/bricks) you'll see lines reporting Gfid mismatch detected 

[2019-12-12 12:28:28.100417] E [MSGID: 108008] [afr-self-heal-common.c:392:afr_gfid_split_brain_source] 0-shared-replicate-0: Gfid mismatch detected for <gfid:31bcb959-efb4-46bf-b858-7f964f0c699d>/ban-gai.rss>, 1c7a16fe-3c6c-40ee-8bb4-cb4197b5035d on shared-client-4 and fbf516fe-a67e-4fd3-b17d-fe4cfe6637c3 on shared-client-1.
[2019-12-12 12:28:28.113998] W [fuse-resolve.c:61:fuse_resolve_entry_cbk] 0-fuse: 31bcb959-efb4-46bf-b858-7f964f0c699d/ban-gai.rss: failed to resolve (Stale file handle)

This documentation details how to resolve GFID mismatches

Read more ...

Python3 - TypeError: encoding without a string argument

I thought I'd document this as although the cause/fix is fairly simple, searching for the error string encoding without a string argument gives a lot of hits for a similarly structured but different error - string argument without an encoding.

An example backtrace might be:

Traceback (most recent call last):
  File "./", line 346, in 
    meta['config_files']['pdns'] = zip_and_compress(read_file_content('/etc/powerdns/pdns.conf'))
  File "./", line 289, in zip_and_compress
    gz = gzip.compress(bytes(s,"utf-8"))
TypeError: encoding without a string argument

With the example code being fairly simple

def read_file_content(path):
    ''' Read the entirety of a file into a variable
    file_content = None
    with open(path, 'rb') as content_file:
        file_content =

    return file_content

def zip_and_compress(s):
    ''' Config files can get quite sizeable. To keep the size of our output DB down
    we gzip and then ascii armour them
    gz = gzip.compress(bytes(s,"utf-8"))
    return base64.b64encode(gz).decode("utf-8")


Read more ...