Our Solar Battery Started Kicking In Late
Over the past 9 months or so, I've written fairly extensively about how we've tried to improve the savings that our Pylontech US2000C solar battery is able to yield.
Using the battery to shave pricing peaks relies on schedules to control when the battery kicks in, ensuring that it supplies the house when prices are most likely to be high.
Getting that timing right is all the more important over winter, with the majority of savings being achieved by charging cheaply and discharging at peak rather than coming from the panels.
Recently, though, our savings attempts have been somewhat thwarted because the battery hasn't been kicking in when it's supposed to.
Over the course of January, I attempted to troubleshoot this whilst also engaging with our inverter's manufacturer.
This post details what we experienced, what was checked and how it was ultimately resolved.
Routine Charge Schedule
We currently use a fixed schedule:
This schedule says:
- We should charge between midnight and 6am or until the battery is full (whichever comes first).
- We should charge between noon and 4pm or until the battery is full
The battery cannot supply the house during a scheduled charge period and so will (well, should) switch to supplying at 6am/4pm whether the battery charged throughout or finished charging early.
Noticing the Issue
It had never really occurred to me that this could be an issue, so I hadn't set up any alerting capable of catching this.
One morning I noticed that the battery wasn't supplying (and hadn't supplied anything that day):
When I checked my graphs, I could see that the battery charged as planned but, when the schedule should have ticked over, it never started to supply household usage (in the graph below, there should be a green line from 6am and isn't):
Part of the reason that I noticed the issue was that, the day before, I'd made a config change to increase the charge rate (in an attempt to ensure that the battery reached 100% at least once a day).
Because I'd made a change, I initially assumed that it must be something to do with that: Although my change was simple, maybe Solis had made a platform change which had effectively been pushed out to the inverter alongside my change?
The other thing that stood out was that, as a result of the charge-rate change, there was now a 2 hour gap between the battery reaching full and when it should come into use, so maybe the battery had gone to sleep or otherwise timed out in some way?
Whatever the cause was, the battery kicked in whilst I was fiddling with settings (I really didn't want to have to go and get the ladder so I was seeing if switching between modes would encourage it back to life).
With the battery supplying again, I set the charge rate back to where it had been, grumbled a little on social media and hoped to get on with life.
At the 4pm switch-over, it started supplying the house just as scheduled.
Failed Again
The next morning, though, it failed to switch over again.
This time, no amount of mucking about with settings would get the battery to kick back in (the day before was probably a fluke). I waited it out and eventually the battery came back into use, some 3 hours late.
With an Octopus power-up due the next morning, I changed the schedule to take advantage of the free electricity. Unfortunately, though, the scheduled charge didn't happened at all.
Concerned that this seemed to be getting worse, I raised a ticket with Solis.
They quite quickly found that, whilst changing settings to try and get the battery to kick back in, I'd screwed up and forgotten to turn the time-of-use schedule back on. As a result they (quite reasonably) assumed that the other issues was probably also a case of PEBKAC and offered to set the schedule correctly for me.
The next day, the battery again failed to charge... at least it wasn't just me.
When I checked, it turned out things were actually slightly worse than first thought. Our panels were generating more than the household needed, but rather than being stored, the extra energy was being exported
So, not only was the battery not supplying the house when it should, it wasn't reliably storing the free electricity we were generating, leaving us to have to buy energy back at higher prices later.
Not Just Mornings
Needing to dig into this further, I decided to start by looking at what might be different in the afternoons: why was it able to start reliably then?
It... it wasn't.
Although not running as late as the morning switch-over, on closer inspection some afternoon switch-overs were definitely tardy.
Give Me A Pattern To Work With
I like patterns because their presence make troubleshooting much easier. I do not like stuff that pretends to be random and fails in whatever chaotic manner takes it's fancy.
Unfortunately, this issue falls into the latter group: there was no apparent consistency nor pattern to the lateness in morning or afternoon. As far as either I or Solis could see, the time at which the battery would come back into service was random:
- 2024-01-11: first reported discharge
07:10 - 2024-01-10: first reported discharge
06:30 - 2024-01-09: first reported discharge
09:00 - 2024-01-08: first reported discharge
08:50
Checking a wider array of dates just added more times to the list.
Discounted Theories
Not having access to the source of the firmware, I did the only thing I could - threw theories at the wall to see if anything stuck (or at least revealed some kind of internal consistency).
Unsurprisingly, there were quite a few things that didn't pan out:
- The firmware applies the schedule by polling the RTC rather than by counting ticks (the latter could lead to discrepancies if the MIPS runs behind)
- This wasn't clock skew: The inverters RTC was correct (or at least, consistently the same)
- It's not triggered by connectivity disruption: Schedules are applied locally
- There was no apparent correlation between recovery and the inverter's internal temperature (batteries behave differently in the cold)
- There was no correlation between recovery times and the outside temperature
- There was no correlation between recovery and the nearest indoor temperature monitor (suggesting it's not our central heating driving timing)
- The battery, apparently, will lower a reported value if it feels a need to adjust to account for temperature - that hadn't changed
- There was no correlation between reported battery charge level at time of recovery
- There was no correlation with panel output levels
- The meter in the fusebox uses raw values and not the delta between reads when deciding whether to demand from the battery (so our baseload being an almost constant level wouldn't have prevented switchover)
- We updated the firmware ( on the basis that there was newer available) with no change in behaviour
The one thing that I kept coming back to was temperature. I hadn't been able to substantiate it, but it felt like it was the only known variable that would consistently change (potentially giving an appearance of randomness).
On the off-chance that they might have configured something unusual, I also emailed our solar installers: they agreed that the behaviour was odd but were unsure of the cause.
Catching the Battery Napping
At this point, I still hadn't actually been up into the attic to look at the battery - I'm not the most mobile of people and a trip up there can lead to days of being in more pain than usual. But, (un)fortunately a day came along where I didn't have a choice.
Whilst I was up there, I looked in on the battery units and found them all lit up, happily absorbing electrons during the afternoon charge.
A few hours later, around 5pm, I noticed that we were still drawing from the grid, so popped back up into the attic to take a look. Rather than the happy array of lights that had greeted me earlier, I found the attic dark with only a sad occasional flash coming from each of the battery packs
Note: there were actually 3 lights, it's just that converting to GIF has dropped that frame.
The LED in question is circled below:
The light(s) that were flashing were labelled RUN. There's a red Alarm LED next to it which was not illuminated. The lights that had previously been illuminated are the charge level lights above the gradient symbol.
Our battery was sleeping on the job!
Checking the display on the inverter didn't yield anything of interest: the battery showed as being available for comms and with a charge level of 91% (but with no energy being supplied).
I flicked each of the power switches off, only to find that the inverter also powered off: clearly it had been able to draw some power out of the batteries to run off.
I turned the system back on by toggling the power switches and then holding down the SW button on the 1st battery until the lights came on. The inverter started to boot and each of the battery packs lit up before starting to supply household usage.
Clearly, then, there wasn't some kind of persistent issue preventing supply.
The eagle-eyed amongst you might have noticed that the battery has a port labelled CONSOLE. Having a serial console in this situation sounds ideal. However, Pylontech's documentation warns that they may invalidate your warranty for using it
Given that the issue could yet have proven to be something that needed fixing under warranty, I decided not to risk it and instead emailed them (unfortunately, they were very slow to respond, so I didn't hear back until far too late - I finally received an email from them whilst I was proof-reading this post).
I also remembered previously seeing a thread which talked about some very similar behaviour. The only problem was, the issue described in that thread had been resolved by an inverter firmware update (which we'd already tried).
The inverter's settings include an Auto Battery Wakeup option (which I assume works by putting a charge voltage across the battery), so I emailed Solis and then enabled it.
Our installer said that they didn't think it would help:
your batteries should be doing that anyway.
They were right, it had absolutely no impact.
A New Firmware Update Arrives
Eventually, I received an email to inform me that my firmware had been updated again, this time to version 470045-010000.
With the stats that Soliscloud provides, there isn't an easy way to identify exactly when the firmware update landed, but it is possible to infer it by using the readingAge value that my Soliscloud Telegraf plugin calculates.
The inverter writes metrics into Soliscloud every 5 minutes and readingAge is the number of seconds between the reported time of that write and Telegraf fetching stats:
SELECT
readingAge
FROM "Systemstats"."autogen"."solar_inverter"
WHERE
time > '2024-02-02T03:00:00Z'
AND time < '2024-02-02T09:00:00Z'
At 06:19 the value spikes to 384 seconds (6 minutes, 24 seconds), suggesting that the inverter missed a write. There are no other spikes of quite that scale that day, so, there's a reasonable chance that the firmware update landed around then.
If we look at the battery rates graph we can also see that the battery kicked in just after that and that it has kicked in dead on time at the changeovers since:
Although the graph only shows a small sample, the battery has continued to kick in exactly on schedule since.
Identifying Scale of Issue
With a light at the end of the tunnel, I wanted to check how often the battery had left us exposed to the morning pricing peak (I should probably have checked before, but didn't want to risk irritating myself by potentially watching money fly out of the window).
The following Flux query checks the battery's rate of discharge at the start of peak (0700 - 0715), if no discharge is reported, we know that the battery hasn't yet kicked into service. The query outputs a count of days where the battery was in service during that time (hit) and a count where it was not (miss):
import "date"
from(bucket: "Systemstats")
|> range(start: -90d)
|> filter(fn: (r) => r._measurement == "solar_inverter")
|> filter(fn: (r) => r._field == "batteryDischargeRate")
// We now have battery output in kW
// Filter down to the period just after the scheduled morning
// start time
|> filter(fn: (r) => date.hour(t: r._time) == 7
and date.minute(t: r._time) < 15 )
// Strip any unnecessary columns to ensure we're dealing with
// a single series even if some state changed
|> keep(columns: ["_time", "_value"])
// Extract a daily max
|> aggregateWindow(every: 1d, fn: max)
// reduce to two counters
// hit: battery was supplying
// miss: battery was not supplying
|> reduce(
identity: { miss: 0, hit: 0},
fn: (r, accumulator) => ({
miss: if r._value < 0.1 then accumulator.miss + 1 else accumulator.miss,
hit: if r._value > 0.1 then accumulator.hit + 1 else accumulator.hit
})
)
The battery failed to kick in before morning peak on 33 days out of the last 90.
Of course, the morning pricing peak is relatively low, so although there is a cost associated with it, it's not huge.
What about evening peak, which isn't nearly so small and has a much greater potential cost? We use the same query, but with a slight change in filter:
|> filter(fn: (r) => date.hour(t: r._time) == 16
and date.minute(t: r._time) < 15 )
We can see that the battery left us exposed to the start of peak pricing on 41 days.
By removing the date.minute() component of the filter, we can also have the the query consume readings for an entire hour, enabling us to look at just how late the battery came into service and on how many days:
|=================================
| Discharge | 1 hr | 2 hr | 3 hr |
|=================================
| Evening | 30 | 14 | 5 |
| Morning | 27 | 17 | 10 |
----------------------------------
The resulting stats indicate that, on 5 evenings out of 90, the battery was at least 3 hours late to come into service.
Powerups
But... there is still a slight complication to account for in those stats.
Some of those times, we probably chose not to use the battery because we were getting free grid electricity as a result of an Octopus Powerup event.
Unfortunately, I only keep the more granular data needed for this for a short time, so we can't check the full 90 days:
import "date"
from(bucket: "Systemstats/autogen")
|> range(start: -45d)
|> filter(fn: (r) => r._measurement == "octopus_pricing" )
|> filter(fn: (r) =>r._field == "override_price")
|> filter(fn: (r) => r.reason == "octopus_powerup")
|> filter(fn: (r) => date.hour(t: r._time) == 16)
|> aggregateWindow(every: 1d, fn: count, createEmpty: false)
|> keep(columns: ["_time", "_value"])
By running this query against the same hours that we used to assess lateness (7, 8, 9, 16, 17, 18), we can count the number of days where a power-up was active in those periods:
|=================================
| Discharge | 1 hr | 2 hr | 3 hr |
|=================================
| Evening | 6 | 2 | 0 |
| Morning | 2 | 2 | 3 |
----------------------------------
So, we can safely discount some of those "late" days as being the result of us adjusting schedules to take advantage of an Octopus Powerup. The vast majority, though, are not explained by this.
Weather
What's interesting, is that adjusting the original query to cover 45 days instead of 90 doesn't change the number of morning misses:
import "date"
from(bucket: "Systemstats")
|> range(start: -45d)
|> filter(fn: (r) => r._measurement == "solar_inverter")
|> filter(fn: (r) => r._field == "batteryDischargeRate")
// We now have battery output in kW
// Filter down to the period just after the scheduled morning
// start time
|> filter(fn: (r) => date.hour(t: r._time) == 7
and date.minute(t: r._time) < 15 )
// Strip any unnecessary columns to ensure we're dealing with
// a single series even if some state changed
|> keep(columns: ["_time", "_value"])
// Extract a daily max
|> aggregateWindow(every: 1d, fn: max)
// reduce to two counters
|> reduce(
identity: { miss: 0, hit: 0},
fn: (r, accumulator) => ({
miss: if r._value < 0.1 then accumulator.miss + 1 else accumulator.miss,
hit: if r._value > 0.1 then accumulator.hit + 1 else accumulator.hit
})
)
This indicates that the issue with mornings started within the last 45 days, bringing me straight back to my suspicion of temperature.
As alluded to in "Discounted Theories", temperature as a factor has been quite well covered:
- The battery finally kicking back in doesn't align with any increases in outdoor temperature
- The battery finally kicking back in doesn't align with any increases in indoor temperature
- The battery finally kicking back in doesn't align with an increase in inverter internal temperature
- Solis said that the battery will lower a reported value if it feels a need to account for temperature and no such change was seen
To summarise: there's been no hard evidence supporting temperature as a factor and, in fact, I've had explanations from the manufacturer as to why it can't be temperature.
Despite that, I just can't get the suspicion out of my mind, even though I'm conscious that I am starting to sound a little bit like Maddy from The Traitors:
Now that we know about the 45 day window, though, we can more effectively look at broader trends.
Unfortunately, the temperature theory suffers an immediate hit: with the exception of a few days, the last 45 days have not really been much colder outside than it was at times in November (when we know that this issue didn't occur):
If we saw similar outdoor temperatures in November and the issue didn't occur then it can't be temperature, right?
However, the graph below shows that the last 45 days really have been quite a bit windier than it was in November:
The weather temperature sensor (unsurprisingly) is outdoors and so, perhaps, isn't the best guide of temperatures within the attic. Could it be that, despite similar outdoor temperatures, the addition of high winds led to the attic being colder than it was in November?
I had actually considered deviations in attic temperature previously, but didn't get very far in pursuing the idea: I installed a Zigbee temperature sensor to try and find out what things looked like, but found that Zigbee is unable to pass through ~30cm of rock wool insulation (meaning that no readings could actually be collected).
But, if we instead look at the internal ambient temperature reported by the inverter over the same time period:
We can see that it has tended to report lower in the last 45 days than in the period before it. It stands to reason, then, that the battery packs have probably also been colder during this period.
So, although it's not been possible to tie battery recovery to specific changes in temperature, there does seem to be a correlation between the period of battery misbehaviour and average temperatures. That's further supported by the perception that, over the last 5 days or so, as average temperatures have risen, the battery has kicked in much closer to the scheduled time.
As a result, I remain fairly convinced that the root of this issue somehow relates to the ambient temperature that the batteries have been trying to operate in.
Although I haven't been up the ladder with a clamp meter to check, there are threads online which imply that battery-sleep issues have been addressed by periodically putting 20W across the battery (presumably to keep it from going to sleep in the first place).
IF that's true, it would also explain how a firmware update has been able to overcome a temperature related issue.
Conclusion
Whatever the actual cause, a firmware update seems to have stopped the unwanted behaviour.
It's been a pretty frustrating issue to deal with, spending a month watching expensive grid electricity come in whilst the battery sits hoarding electrons and sticking two terminals up at us
(if you're wondering, then, no Bing's AI has not really got any better at filtering than it was 9 months ago)
Solis, for their part, have been pretty helpful and professional throughout - patiently tolerating my throwing out & testing of theories despite me having no real insight into the workings of their firmware (yeah... I'm afraid I became one of those customers).
Our installers were also pretty great. They were not in a position to do detailed software analysis (and you wouldn't really expect them to be) but they offered useful insights and were the ones who prompted for the firmware update which ultimately put things back on track.
With the new firmware in place, we can get back to squeezing savings out of the thing.
