Docker's default configuration doesn't perform log rotation.

For busy and long running containers, this can lead to the filesystem being filled with old, uncompressed logging data (as well as making accidental docker logs $container invocations quite painful).

It is possible to configure docker to rotate logs by editing daemon.json, but the rotation threshold options are fairly limited:

• max-size: size at which to rotate
• max-file: max number of rotated files

Whilst these options do help to reduce filesystem usage, being purely size based they fail to support a number of extremely common log rotation use-cases

• Log rotation at a specific time based interval (e.g. daily log rotation)
• Maximum retention periods (to comply with GDPR retention policies etc)

Unfortunately, json-file isn't the only logging driver to suffer from this limitation, the local driver has the same restrictions. It looks like there's an implicit decision that anyone who wants to follow common rotation practices should just forward logs onto syslog, journald or some other logging infrastructure (such as logstash). In practice, there are a variety of use-cases where this may be undesirable.

However, as json-file simply writes loglines into a logfile on disk, it's trivial to build a script to implement the rotation that we need.

This documentation details how to set up interval based log rotation for docker containers

Read more…

Pi-Hole provides simple tooling for managing lists of ad domains to block, but sometimes simple blocklists don't provide enough coverage on their own.

Blocking Xiaomi's Tracking

The mobile phone manufacturer Xiaomi is a good example of why a more flexible blocking approach is sometimes called for.

Various views within the MIUI system UI contain tracking/ads with a broad range of regionalised addresses used to support these data harvesting activites.

For example, Xiaomi phones sometimes contact the domain tracking.intl.miui.com, but there are also regionalised variations such as tracking.rus.miui.com and tracking.india.miui.com.

Once known, these domains are easy to block, but a purely reactive approach means that there will always be periods where data is collected unimpeded.

It's far preferable, then, to be able to predict what their other tracking domains might be. Unfortunately the regionalisation of Xiaomi's services isn't particularly consistent:

• There are services at fr.app.chat.global.xiaomi.net
• But there are none at tracking.fr.miui.com
• There are also no services at tracking.gb.miui.com but DNS lookups for it behave differently to those for tracking.fr.miui.com

This inconsistency makes effective blocking of Xiaomi's tracking domains via blocklists quite difficult: not only do we need to be able to enumerate all current domains, we're also reliant on Xiaomi not launching stalkerware services in a new region.

Enter Regex

Regular expressions (regex) provide a tool by which we can improve the effectiveness of our blocks.

Rather than needing to enumerate every variation of tracking.$foo.miui.com we can instead provide a pattern to match against

^tracking\..+\.miui.com$

For those not familiar with Regex, this breaks down as follows

• ^tracking. : queried name must begin with tracking (the ^ signifies start of the input)
• .+ : allow an unlimited number of any characters
• \.miui.com$ : the queried name must end with .miui.com (the $ signifies end of the input)

As if this wasn't powerful enough, PiHole also supports approximate matching allowing things like stemming to be used.

For example, this allows us to trivially create a regular expression that'll accept TLD substitutions:

^tracking\..+\.miui.(com){#3}$

This expression will match any of the following

• tracking.foo.miui.com
• tracking.foo.miui.org
• tracking.foo.miui.net

Managing Regex in Pi-Hole

So, why do we need an entire post for this?

Adding regex blocks to Pi-Hole individually is trivial as they can be added through the web interface

However, adding a bulk list or linking a remotely maintained list provides a bit more of a challenge.

Older versions of Pi-Hole referenced a file called regex.list on disk, allowing easy automation of updates.

But support for that file was dropped when Pi-Hole Version 5 was released last year and regexes now need to be written into the gravity database.

This post details the process of automatically fetching and refreshing lists of regular expressions for Pi-Hole versions later than version 5.

Read more…

I started to curate my own adblocking scripts back in 2014, making them available in the directory /adblock/ on my site.

At the time of their creation the lists were poorly controlled and pretty sparsely documented:

In 2018, I got my act together a bit: I moved the lists into a Github repo and implemented project management to track additions to the lists.

Whilst project management improved, the publishing of the lists continued to rely on some increasingly shonky bash scripts. Those scripts modified and compiled in third party lists, stripped duplicates and generated various output formats. They were never engineered so much as spawned.

Because the lists used third party sources, the compilation process needed to run at scheduled intervals: it couldn't simply be triggered when I made changes, because the lists were presented as near-complete alternatives to others.

Despite their awful hacky nature, those scripts managed to compile and update my block lists for nearly 8 years.

However, they've long been overdue for replacement.

This post serves as a record for the deprecation of my original adblock lists, as well as providing details of their replacement.

Read more…

Like many on this here Sceptred Isle, I use my kettle regularly throughout the day.

I live in a hard water area, and the prevailing wisdom is that you should refill rather than reboil your kettle in order to reduce the rate that limescale builds up at (and by extension reduce energy usage).

The logic is that during the first boil, the denser minerals move to the bottom of the kettle, so after you've made your cuppa, the adulterants in the water left in the kettle are much more concentrated, leading to an increased rate of scaling in each subsequent boil (and, it's been suggested, possible increased health risks).

From an energy use perspective, this is an issue: Limescale adds mass to the inside of the kettle, so over time more energy is required in order to boil the same volume of water (though, strictly speaking, if you're using the gauge on your kettle you'd actually be boiling a smaller volume water because the limescale will have displaced some measure of it) because you're having to heat the limescale layer too.

Emptying and refilling reduces the rate of build-up, but, if the kettle is used even semi-regularly it comes at a cost: the residual warmth of the remaining water is lost and the new water has to be brought to boil from (tap) cold instead.

It's the cost of that temperature gap that I was interested in: I wanted to see how big a difference refilling made in energy usage (both per boil and over time).

Over the course of a few days, I stuck to my usual routine (best summarised as: want tea, make tea) but used different approaches to kettle filling to see what the effect on energy consumption was.

Read more…

I use a free Uptime Robot account to help keep an eye on the availability of www.bentasker.co.uk.

Every 5 minutes, UptimeRobot places requests to my site (and it's origin) and reports on how long those requests take to complete and updates my status page if there are issues.

The free tier only tests from a single location, but is usually a good indicator of when things are going (or starting to go) wrong. I use my uptime-robot exec plugin for Telegraf to pull stats from UptimeRobot into InfluxDB for use in dashboards.

Because I test against my origin as well as the CDN, it's usually possible to tell (roughly) where an issue lies: if CDN response time increases, but origin doesn't, then the issue is likely on the CDN.

Earlier in the week, I saw a significant and sustained increase in the latency UptimeRobot was reporting for my site, with no corresponding change in origin response times.

This suggests possible CDN issues, but the increase wasn't reflected in response time stats drawn from other sources:

• There was no increase in the response time metrics recorded by my Privacy Friendly Analytics system
• The Telegraf instance (checking the same endpoints) on my LAN wasn't reporting an increase

Given that pfanalytics wasn't screaming at me and that I couldn't manually reproduce the issue, I felt reasonably confident that whatever this was, it wasn't impacting real users.

But, I decided that it would be useful to have some other geographically distributed availability system that I could use for investigation and corroboration in future.

I chose to use Amazon's Elastic Container Server (ECS) with AWS Fargate to build my solution.

This post walks through the process of setting up a serverless solution which runs Telegraf in a Fargate cluster and writes availability data into a free InfluxDB Cloud account.

Read more…

Version: 0.1

Project Info

The tor-daemon plugin is an exec plugin for Telegraf allowing statistics to be captured from the Tor daemon on Onion services and relays.

Details on usage can be found at Monitoring the Tor daemon with Telegraf.

Release Notes

Version 0.1 implements the basic functionality of the plugin

Release Revision

This version was released in commit 8b60035

Release issue tracking

Issues assigned to this release can be viewed in GILS

Plugin

The plugin can be downloaded from Github or from here.

My services have been available via Tor .onion for around 7 years now, but my monitoring of their availability has always been relatively limited. I did previously have smokeping running reachability tests, but other than that there's been a reliance on me noticing that things weren't right (or perhaps receiving reports that an .onion was misbehaving).

Part of the reason for this is that there's never (to my knowledge) been a good centralised way to monitor the health of a Tor install. Nyx is a fantastic command line tool, but relies on the operator logging into their box: it's akin to relying on top to monitor CPU usage.

I've always figured that it should be possible to monitor the tor daemon more effectively, but never really quite got around to do anything about it.

This week, I decided to take a pop at it, and a quick scan over Tor's control port spec revealed how easy it should be to collect stats.

This documentation details how to use my new Tor Daemon Plugin for Telegraf to collect metrics from a Tor daemon.

The full list of statistics collected can be seen in the plugin's README, but they include

• bytes_rx: total bytes received by Tor
• bytes_tx: total bytes transmitted by Tor
• uptime: Tor daemon uptime
• version_status: Tor's assessment of whether the installed version is OK to use
• Accounting information: is a quota set? If so, how much is left?
• Reachability test statuses
• Guard node states

Although my main focus is on monitoring the availability of my onion services, the plugin can be used to monitor tor relays, bridges and exit nodes too.

Read more…

Earlier in the week, I saw an article about running Python in the browser via a port of the Python interpreter to WebAssembly.

I couldn't think of an immediate need for using it over Javascript, but often learn unexpected things when playing around with new technologies, so I wanted to have a play around with it.

An obvious start point for me seemed to be to see whether I could use the Python InfluxDB Client to connect to InfluxDB Cloud and query data out.

This proved to be more challenging than I expected, this post follows the process of tinkering around with Pyodide and Pyscript in order to get the client running and ultimately build a semi-interactive webpage that can query InfluxDB using Python in the browser.

Read more…

It's so easily done: you cook a nice roast dinner and someone "helps" by cleaning your roasting tray.

You find out, far too late, that they did this by putting it in the dishwasher, so the next time you see your trusty roasting pan all the seasoning's been stripped and it's a rusty mess

Unless you're partial to added rust in your food, this is an unmitigated disaster - anything cooked in the pan is going to stick and your roast spuds will pull apart when you try and take them out of the pan.

It is, however a recoverable disaster - the pan can be re-seasoned using much the same method as you'd use to Season cast-iron cookware.

Essentially, what it involves is coating the tray in (cooking) oil, and then holding that oil at it's smoke point for an extended period so that it leaves a protective residue on the base of the pan.

In this post, I'm going to re-season what was my best roasting tray. The same process can be used for cast-iron pans too.

Read more…

A few days ago, news broke that $34,000,000 had become forever irretrievable as the result of a bug in a smart contract created by AkuDreams.

It's a good example of an issue that's pervasive across multiple industries, rather than being something that's cryptocurrency specific (though I'm no fan of crytocurrencies)

The reason that $34m in ethereum has been lost isn't simply because someone made a mistake in a smart contract: that's just the inevitable result of an underlying naivety in the smart contract model itself. As we shall see, that same naivety is all too common in the wider technology industry too.

Read more…