Musings on Home Automation

I've dabbled with elements of Home Automation in the past.

In a previous rental, we only had storage heaters, so I equipped each room with an Oil Radiator and an energenie RF plug socket (like these https://www.amazon.co.uk/Energenie-Remote-Control-Sockets-Pack/dp/B004A7XGH8) using a Raspberry Pi and the Energenie remote control header board to set up an effective heating schedule.

However, aside from that, and mild "wouldn't it be nice too..." ideas, I've not really been overly interested into it until relatively recently.

Having spent a bit of time dabbling, I thought I'd write a post on my experience - not least in case it helps people with some of the things I struggled with.

 

 

NEST Thermostat

Our boiler died at the beginning of spring, so we had a new Combi fitted, which also meant we needed a new controller and thermostat. 

We went with the NEST Gen 3 Self Learning Thermostat.

Aside from a few annoyances about Google's attempts to shove NEST into Google Home, all was fine over the summer, as we didn't actually need any heating, but as winter approached, things started to go awry and our gas usage increased far, far more than I was expecting (and was significantly higher than last years heating, despite having a more efficient boiler).

I had a schedule set up to try and save energy, turning the thermostat right down during weekdays when it was only me home - meaning the heating should generally be off for 8 hours during the day.

Except, I came down one day, to find that NEST's "True Radiant" had started pre-warming the house 5 hours before the temperature was configured to increase. I turned it back off, then came down later and turned the temperature back up to the target and the display said it'd only take 1.5 hours to get there.

So, after a few days of experimentation tinkering with True Radiant's setting, I ultimately turned it off entirely, and adjusted my scheduled times forward about 45 mins so the house would feel warm (if not be up to temperature) at the configured time.  -1 point for NEST's "smart" featureset.

A few days later, we were sat watching a film and started to feel cold. When I checked in the NEST app, the thermostat had decided that we were out of the house, and so had switched to Eco mode (essentially setting the target temperature to 9). Although this could be addressed by allowing NEST to track our phones, that doesn't seem like a worthwhile privacy trade-off.

So, Home/Away Assist got disabled too.

Which left me with a NEST thermostat with no smarts turned on - essentially an overpriced dumb thermostat with a weekly programming function.

I'd lost (and, still haven't regained) all trust in NEST and it's "smart" qualities.

But, at the same time, I didn't particularly relish the idea of paying out to change the heating programmer just 9 months after having it installed, especially when there's no guarantee that HIVE would be any better (and I've heard some horror stories about Tado too).

Searching around, I could see that NEST has an API, so I decided to take another approach - have it supervised by something I had control over.


 

HomeAssistant

HomeAssistant (also known as Hass.io) is an OpenSource Home Automation solution.

It has a wide range of "integrations" allowing you to pull data from, and send commands to a wide range of third party products/solutions.

It's often run on a Raspberry Pi and/or in docker, but I didn't have a Pi spare and didn't want to use docker for this, so opted to run in a VM instead. They provide pre-built VM images on their download page. This later led to a period of grumbling when I found that the VM image runs an OS, docker and the HomeAssistant Docker image (who puts docker into a vm....), though in fairness I've not really had an issues with it.

 

HomeAssistant and NEST


NEST is amongst those.

Unfortunately, NEST was acquired by Google and the APIs have been subsumed into Google's API ecosystem, the result of this is that getting API up and running is an unintuitive mess, relying on you knowing where the correct pages are.

Whilst the HomeAssistant docs try to help, they suffer from Google's habit of moving and changing things. Initially the integration didn't work, because I hadn't gone back into my OAuth acceptance page config and clicked "Publish" (because having it be created private by default is so useful...).

However, once that pain was out of the way, it all showed up quite nicely, and you get graphs showing recorded temperature, target temperature and whether the boiler was on, as well as the ability to control the temperature

So, I had visibility and basic control of NEST.

However, I wanted to be able to use additional inputs in order to make decisions.

 

Zigbee

One of the things I was fairly firm on, at the outset of this, was that I wanted as much control as possible to be local only. I was OK with relying on remote services whilst prototyping the setup, but ultimately wanted to remove dependancy on my internet connection as far as possible.

This, of course, means I needed to pick an IoT protocol to run with - the contenders really being Z-Wave or Zigbee.

In terms of functionality, there's probably not too much between them, but there were a few deciding factors

  • Z-Wave uses different frequencies in different regions of the world - when ordering kit, you'd need to make sure you're getting the right one
  • Zigbee now has some big names attached to it, with Amazon, Apple and Google working with the Zigbee Alliance, suggesting a better chance of long-term support and stability

Ultimately, I went with Zigbee because I thought there was a better chance of everything working without too much frustration.

In order to communicate with Zigbee devices, you of course need hardware to speak to them - known as a Zigbee Co-ordinator. Many HomeAssistant (and it seems OpenHAB) users use Zigbee USB sticks for this purpose, but I didn't want to invest time/money in getting that working without first proving Zigbee could do what I needed.

Newer Amazon Echo's actually have a Zigbee hub built into them, but Alexa is persona non-grata in this house, and examination of the Echo integration documentation doesn't suggest that HomeAssistant can actually access the devices linked to the Echo (though I may be wrong).

So, instead, I spent £15 on a Tuya Zigbee gateway, and activated the Tuya Integration in HomeAssistant. Simples, right?

 

The Zigbee Crapshoot

Errr, no.

I'd gone with Zigbee on the basis that it should have easy hardware compatability, but unfortunately it turns out that Zigbee is a complete shitshow of incompatible devices.

It's not that the devices can't talk Zigbee, it's that they often won't speak to Zigbee devices made by other manufacturers - sometimes because one or the other has done something non-standard, or because the manufacturer has made an explicit decision not to support "other" devices.

For example

  • There are reports over the net that various Zigbee plugs/repeaters cannot repeat signals originating from Xiaomi/Aqara devices because Xiaomi added some non-standard stuff, extending Zigbee in a non-compatible manner
  • Devices like the SONOFF Zigbee gateway will refuse to talk to non SONOFF devices downstream
  • My Tuya gateway refused to pair with some Zigbee smart sockets made by FeiBit, but would talk to some other non-Tuya devices
  • Although the Tuya gateway (and app) would see my non-Tuya temperature sensor, it wasn't exposed via the Tuya API, so HomeAssistant couldn't see it. Tuya Temperature sensors on the other hand, were exposed just fine

Still trying not to have to run my own Zigbee hardware, I bought a £10 SONOFF Zigbee gateway, with the intention of flashing Tasmota onto it, to use as my Zigbee coordinator.

Although Tasmota flashed on fine, HomeAssistant couldn't talk to it (in the Tasmota console, I could see the connection string coming in, but no reply went out). So, ultimately, I moved that onto the "to revisit" pile.

Ultimately, I bit the bullet and bought a CC2531 USB Zigbee Co-ordinator pre-flashed with Zigbee co-oridinator firmware.

I plugged it into my VM host and passed it through to the VM, before hitting HomeAssistant's web interface to configure zha - setup was fast, and trivial.

I was able to pair every Zigbee device I had with it, with no issues.

Lesson learned: If you're going Zigbee, run your own co-ordinator as the manufacturers don't seem to have received the interoperability memo

 

HomeAssistant General

HomeAssistant itself is a fantastic piece of kit, and works well if you're willing to spend some time feeling fairly frustrated, and exploring docs.

Although there have been times when I've thought I had, I've not yet found anything I need that it can't do, only things that I didn't yet know how to implement. Sometimes the answer is that there's an integration that you don't know about, and just need to search for the right thing.

 

Learning to do temperature control properly

As part of my experimentation, I wanted to read from the temperature sensor in the shower room, and

  • turn the towel rail (connected to a smart socket outside the room) on if it was too cold
  • turn it off if too warm
  • Do neither if not in a scheduled period

I went about creating an automation for each task, and it didn't really work very well.

However, when trying to work out how to improve it, I found that HomeAssistant has an integration called generic_thermostat

climate:
  - platform: generic_thermostat
    name: Shower Room
    heater: switch.sonoff_10007b4ee7
    target_sensor: climate.bffb2d1d241441b73edwxp
    min_temp: 7
    max_temp: 21
    target_temp: 19
    min_cycle_duration: 00:05:00
    hot_tolerance: 0.5
    cold_tolerance: 0.1
    away_temp: 10
    initial_hvac_mode: "off"
    precision: 0.1

This configures so that

  • Use the smart switch as a heater
  • Read from the temperature sensor 
  • The min temperature you can set to is 7
  • The max is 21
  • The default target temp is 19
  • If powering on, wait a minimum of 5 mins before turning off (and vice-versa) - preventing excessive cycling
  • Wait until we're half a degree over target before turning off
  • Turn on if we're > 0.1 degree below target
  • Default is off

This gives us a climate controller

Shower Room Controller

(yeah, it gets cold in there)

We could now manually switch the heater into auto mode so that it'd try and maintain temperature.

It was then just a case of creating automations to switch it between auto and off at specific times to achieve a schedule.

Actually, when I first set it up, it wasn't quite that simple - the temperature sensor was exposed via the Tuya API, but the Tuya integration used On, Off, Auto for the state attribute, but generic_thermostat expects it to be the temperature, so I had to create a psuedo-sensor to translate it

sensor:
    - platform: template
      sensors:
         shower_state_state:
           unique_id: "shower_room_temp_abcedfg"
           friendly_name: "Shower Room Temperature"
           value_template: " state_attr('climate.bffb2d1d241441b73edwxp', 'current_temperature') "
           attribute_templates:
               state: " state_attr('climate.bffb2d1d241441b73edwxp', 'current_temperature') "
 

Once I'd moved to my own Zigbee co-ordinator, though, that wasn't an issue.

 

HomeAssistant and Zigbee Events

Another challenge I ran into was around setting HomeAssistant up to re-act to the push of a Zigbee button.

I mean, that's the whole point of a Zigbee button - you link an automation to it's push event(s) and do something (in my case, switch the towel rail climate controller into auto mode).

It was fairly clear that the automation trigger was an event, but what event? How do you find out which event fires when you push the button?

It's not an easy thing to search for either, things like Homeassistant Zigbee Button push will tend to lead you to posts relating to UI elements, but eventually I found this Gist which pointed me in the right direction.

The event I needed was zha_event, you then filter this by the device's identifier (device_ieee).

To get this information, you hit Developer Tools -> Events and then set to listen to the event zha_event, and then press your button, giving you a dump of JSON

{
    "event_type": "zha_event",
    "data": {
        "device_ieee": "5c:02:72:ff:fe:49:f2:ef",
        "unique_id": "5c:02:72:ff:fe:49:f2:ef:1:0x0006",
        "device_id": "c7b4b40ba23f686e71de9da76e8231ef",
        "endpoint_id": 1,
        "cluster_id": 6,
        "command": "on",
        "args": []
    },
    "origin": "LOCAL",
    "time_fired": "2020-12-23T16:55:24.746293+00:00",
    "context": {
        "id": "1dd16fa23d60c6d85135917608b48575",
        "parent_id": null,
        "user_id": null
    }

The button supports sending 3 different commands (based on how you push it), so that can be checked within the automation too, ultimately giving something like

- id: '1608742755182'
  alias: Shower Room Button Press
  description: ''
  trigger:
  - platform: event
    event_type: zha_event
  condition:
  - condition: template
    value_template: '{{ trigger.event.data.device_ieee == "5c:02:72:ff:fe:49:f2:ef"}}'
  action:
  - choose:
    - conditions:
      - condition: template
        value_template: '{{ trigger.event.data.command == "on" }}'
      sequence:
      - service: script.towel_rail_power_on
        data: {}
    default:
    - service: script.towel_rail_power_off
      data: {}
  mode: single

 

HomeAssistant and Scheduling

One of the things that HomeAssistant seems to lack (or, perhaps I've yet to find it) is a good scheduling mechanism for things like heating.

For all of it's other faults, NEST does have quite a nice scheduling interface

(I mentioned above that I drop the temperature in the mornings - the above is taken during the holiday season, so I don't).

Obviously you can just specify the times in your automations, but that feels a bit clunky and not very user friendly.

So, as an intermediate point, I created a bunch of helpers in order to expose a panel like this (if the times are a bit weird, it's because it's used to control an oil radiator in an underheated bedroom)

Heating Schedule

The automations then reference the value of these settings, allowing for easy tweaking from our phones.

 

Getting back to NEST

So, all of this started out with wanting to have my smart thermostat work in a way that was actually smart, and bill friendly, rather than melting ice-caps with an unjustified pre-warm period.

There are a few "wants" on my list

  • The ability to have a "boost" button - i.e. turn the heating on for (say) an hour, then back off
  • The ability to have rooms within the house call for heat (perhaps by triggering the boost function)

I'm not quite ready to proceed with this, as I'm waiting on a few more temperature sensors to arrive - I want to properly understand the rate at which heat dissipates from each room in the house before I start implementing things (as the approach will likely be different for each room).

So, for now, this next bit is all fairly theoretical, and unfortunately, implementing "boost" isn't as straightforward as it sounds.

The NEST thermostat is, well, temperature based and doesn't incorporate a boost function (Hive thermostats, on the other hand, do) - so if it believes the house is at temperature, it's not going to turn the boiler on.

So in order to affect a "boost" what's needed is

  1. read the current target temperature
  2. Call the API to Increase it by n degrees
  3. Wait for the boost period to elapse
  4. Call the API to reset the target temp to it's original value

The problem here, is that if HomeAssistant is restarted in between API calls, it'll lose state and never turn the heating back down (which isn't the end of the world, the existing schedule should adjust it back down, but it might mean some rooms in the house get uncomfortably warm in the meantime - hence wanting to wait on temperature sensors).

The challenge here is that HomeAssistant doesn't (appear to) support variables as such, so instead you need to hack around that by pushing state into some inputs (which act as globals).

As a quick initial effort, the following script appears to work

boost_heating:
  alias: Boost Heating
  sequence:
  - service: input_number.set_value
    data:
      value: '{{ state_attr(''climate.downstairs'',''temperature'') }}'
    entity_id: input_number.current_heating_target
  - service: climate.set_temperature
    data:
      temperature: '{{ float(states(''input_number.current_heating_target'')) + float(states(''input_number.heating_boost_degrees''))
        }}'
    entity_id: climate.downstairs
  - delay: 00:20
  - service: climate.set_temperature
    data:
      temperature: '{{ float(states(''input_number.current_heating_target'')) }}'
    entity_id: climate.downstairs
  mode: single

However, it is somewhat racey:

  • If the second API call fails, or if HomeAssistant is restarted, then the boost temperature will be maintained
  • If the NEST schedule lowers the temperature before the boost period elapses, the script will set the temperature back to it's original scheduled value rather than the new one

There doesn't appear to be a good way around the second - ultimately it's a shortcoming of the NEST API (and product, really, it's not like the NEST app supports it either). The answer, down the line, is probably to switch to a thermostat with a more complete feature set.

In the short term, though, it should now be possible to assign automations to temperature sensors in important rooms so that they can effectively call for more heat (subject to time constraints etc).

Of course, in the longer term, the setup will probably include smarter thermostatic heads on the radiators themselves, but as I'll need to change the TRV's themselves for a more modern thread, that's something that will have to wait. Ultimately, we'll want those heads to be able to call for heat, which either means a non-NEST thermostat, or having something like the above implemented and working. 

 


Conclusion

My ambitions for a "smart home" have never been huge - I don't particularly need or want motion detection based triggers (yet), I just want my heating to be smarter and not burn gas when it doesn't need to - without having the house constantly feel cold as a result.

NEST promised to be the solution to that, but utterly failed to deliver, in fact, graphing out my daily gas usage, it's fairly clear which red block was me turning True "Radiant" off entirely

Daily Gas Usage

Apart from a brief spike (because a door was left open - I guess we add door sensors to the wishlist) usage has consistently been much lower, despite us also having crossed into the holiday period, where temperatures are set higher.

HomeAssistant, quite frankly, is a fantastic bit of kit, though it does require some patience and dedication when you're trying to implement new things. Error messages aren't always helpful, but so far the gaps have mostly been in my own knowledge rather than in the platform's capabilities.

With a little setup, Zigbee works as it should, but my foray into that started out as quite a disappointment. Various vendors have managed to take all the promise of a common standard, and smother in in an attempt to effect vendor lock in.

As a result of a bit of work, I've now got a fairly simple automation setup (with niceties like living room lamps turning themselves on at sunset), and better visibility over the heating decisions being made

 
 Share