Hi! DIY Home automation system build.

[paulca]

433Mhz key fobs (if you mean car key fobs) are all digital. As are most other things that use 433Mhz like weather stations.

It's been a very long time since any sort of remote control was analogue...

Technically no RF is digital, not Wifi, not GSM. Mother nature does not do digital.

But I take your point that the payload is digitally encoded.

It is the analogue nature of RF that scares me from an electronics perspective. Dealing with I2C or SPI control over an interface that ends up on Wifi or whatever I'm fine with, but if I had to design a board with an RF chip on it, the mathematics becomes intense very quickly.

[jvallis]

Very interesting. A few things I'd mention:

1. Your system seems to be operating at quite a high resolution. Boilers, particularly oil burners don't like to be cycled too much - it's best to limit boiler cycles to between 3 and 6 per hour (gas boilers can cycle more). Starting a boiler is a very energy intensive operation and the trick with boilers is to get the condenser working as long as possible once it's heated up. A condenser is best thought of as a turbocharger for a boiler. In this, I would consider using a PID to control the boiler if you don't have the capability to run Opentherm. Alternatively, since you're replacing your boiler and are a geek, I'd strongly suggest you get an Opentherm capable boiler and use Cyril's Opentherm Gateway to monitor and control the boiler.

2. Given you already have ESP-based sensors, it may be an idea to poll humidity as well and use the humidex to assuage temperatures which is a relatively simple calculation. This is the 'feels like' temperature. Humidity changes when humans are in a room and shaded rooms are different to those with solar radiation (big windows).

3. Cacti is very old, and has largely been superceded by Influx DB and Grafana, two products that work together very well. This will give you pretty HTML5 graphs that you can zoom in and out etc, and Influx is already a JSON db that's optimised for time-based data.

4. Please share your work! ESP programs, controller programs etc on github!

[paulca]

Hi, thanks for your thoughts.

Asides from the experiment of controlling the heating loop temp at 40*C, which I don't think I really need to do, the boiler really only cycles at most 2 or 3 times an hour. Usually it's once once or twice. The current boiler is an old 1970s oil burner, so no condenser. I'd honestly prefer to have independant control of the burner and pump, but I am upgrading it to gas this autumn anyway. Getting a Bosch-Worchester 30kW condensing gas combi-boiler fitted. I will hopefully be able to talk the spark installing the electrics into giving me a plug socket beside the boiler for gadgets as the remote switch on it is "Volt free", ie. a live switch only, no neutral, no power.

I'll look into OpenTherm.


Humidity. I currently record outdoor humidity, although I have a TODO list to move those outdoor/garage sensors off the RPI and onto an ESP8266 as the PI has become unreliable. It's not just Wifi either, the log is full of fails to read sensors AND no connection to Wifi when it goes funny, so something more fundamental is happening to the hardware, could be spiders, could be cold, could be humidity.

I would see the Humindex as more of a thing for cooling, rather than heating, but as next summer I might try automating the air con in the bedroom so it's already cool for heading to bed I keep it as an option.

Cacti has released a fairly new version, 1.2.5 which has an all new UI. I do have reservations about it as it's gone "single page webapp" with stateful buttons which I never liked. If I click button X to go to page X, then pressing button X again should refresh page X, but single page web app developers seem oblivious to this concept and that urks me.

The question about graphana et. al. Will they read raw RRD files? I'll maybe have to take a look at these, but cacti currently works fine enough. Note I don't use Cacti for data gathering, I manually write RRDs from Python and just point Cacti at the RRDs. RRDs are fairly perfect for timeseries data, but I'm not in that reluctant to try other means, I can always dump the RRD to XML and reimport to something else.

On an unrelated point, while I am very happy with the timed, expiring, demand based nature of the heating control, I think it's the right solution for safety, it is however a little cumbersome for control of things like lights. Mostly the polling nature of demands and the single sided state machines (turn it on, just let it time out), don't work well for lights. When you hit a switch you want an instant response from the lights and you probably don't expect the light to go off 15 minutes later unless you press the switch again. My current thoughts are to move to MQTT for lights.

My Source code is here: https://gitlab.com/paulcam/home_heating

The ESP code in: https://gitlab.com/paulcam/home_heat...master/esp8266 (some legacy)
The demand scheduler etc in here: https://gitlab.com/paulcam/home_heat...ster/scheduler

[paulca]

A small update, purely graphical. I spend half an hour working with Cacti to combine the graphs:

Ambient Temperatures
Heating Status
Presence Home

To create this beauty. All the information in one place. Green shading is "Presence" and Blue is "Heating status"


You can see an instance of my Raspberry PI in the garage bugging out for a few hours on Thursday morning.

As another update, I successfully processed the Heating Pipe graph using the command line to give me a figure of "total hours boiler time", which is a more accurate figure for how much oil/gas I am using given that the boiler is cycling itself to maintain it's loop thermostat. This was done by dumping the RRD for a given time range and when the current loop temp is higher than the previous it adds the time between the current and previous data points to the total run time. If the temperature was the same or lower the boiler is not running.

I have yet to make this re-usable. it was just an experiment requiring a bit of shell foo like cat, cut, grep and a 10 line python script to do the detection and counting.

[jvallis]

Hi, thanks for your thoughts.

Asides from the experiment of controlling the heating loop temp at 40*C, which I don't think I really need to do, the boiler really only cycles at most 2 or 3 times an hour. Usually it's once once or twice. The current boiler is an old 1970s oil burner, so no condenser. I'd honestly prefer to have independant control of the burner and pump, but I am upgrading it to gas this autumn anyway. Getting a Bosch-Worchester 30kW condensing gas combi-boiler fitted. I will hopefully be able to talk the spark installing the electrics into giving me a plug socket beside the boiler for gadgets as the remote switch on it is "Volt free", ie. a live switch only, no neutral, no power.

I'll look into OpenTherm.

I can understand why you'd want to mess around with a pump and burner on such an old boiler, but modern boilers - like cars - are much more clever and advanced. It's best instead to think of them as a 'black box thing' and use an interface to get the information to monitor and control it, while using it's internal firmware to manage itself, the burner and pump etc.

That's why I mentioned Opentherm, a standard developed by Honeywell to do just that. You 'demand' heat by specifying a temperature and the boiler will automatically manage what the burner needs to do to optimise the gas usage to deliver you water at that temperature, and thus depending on the heat loss of your pipes you can generate just the right amount of heat to satisfy the demand at that specific time.

Opentherm is a 2-wire serial interface (similar to I2C) that has a set of predefined messages for communication. Cyrils OTGW is on an OrangePi (a Chinese RPi which has Wifi) and a HAT, but it's based on the work done on the PICS-based OTGW (http://otgw.tclcode.com). On that site, you can go to: http://otgw.tclcode.com/matrix.cgi#boilers, and you will see all the message ID's of Opentherm (rollover the number on the left column to see the details of the message). It's best to think of OT as a sliding scale of control rather than just a binary on/off.

Worcester Bosch boilers use a proprietary standard called EMS, and you'll need an intermediary device (https://myboiler.com/opentherm/worce...sch-opentherm/) to work with OT. The myboiler site has a few other boilers that support OT.

Humidity. I currently record outdoor humidity, although I have a TODO list to move those outdoor/garage sensors off the RPI and onto an ESP8266 as the PI has become unreliable. It's not just Wifi either, the log is full of fails to read sensors AND no connection to Wifi when it goes funny, so something more fundamental is happening to the hardware, could be spiders, could be cold, could be humidity.

I would see the Humindex as more of a thing for cooling, rather than heating, but as next summer I might try automating the air con in the bedroom so it's already cool for heading to bed I keep it as an option.

It's up to you. Older houses feel colder because they suffer from humidity even in winter. Internal humidity gets changed a lot with heating as it dries it out, and if you get any mildew or condensation, you have a humidity problem usually as a result of over or under ventilation. At the same time you'll get a very crusty nose when waking up in a super dry house. Indoor humidity should always be under 60%, but the range of 30-50% is 'comfortable'.

Cacti has released a fairly new version, 1.2.5 which has an all new UI. I do have reservations about it as it's gone "single page webapp" with stateful buttons which I never liked. If I click button X to go to page X, then pressing button X again should refresh page X, but single page web app developers seem oblivious to this concept and that urks me.

The question about graphana et. al. Will they read raw RRD files? I'll maybe have to take a look at these, but cacti currently works fine enough. Note I don't use Cacti for data gathering, I manually write RRDs from Python and just point Cacti at the RRDs. RRDs are fairly perfect for timeseries data, but I'm not in that reluctant to try other means, I can always dump the RRD to XML and reimport to something else.

RRD is Round Robin Database. It's a database in itself, albeit a very simple one. Influx is another more powerful database, and it's an actual server that communicates with JSON over HTTP. No XML, no RRD no other intermediary format. You can get your ESPs to write directly to Influx by simply POSTing and HTTP request and GETing the result. Furthermore, it's much more powerful than RRD because you can do more complex queries much like SQL (though you again send a special JSON message to do so instead of a SQL string). Just think of it as RRD v2 and saving you the need to translate between JSON, XML and RRD all the time.

Grafana is a replacement to Cacti, and can easily be made to read RRD files with an intermediary translator (e.g. https://github.com/doublemarket/grafana-rrd-server). But it works natively with Influx.

On an unrelated point, while I am very happy with the timed, expiring, demand based nature of the heating control, I think it's the right solution for safety, it is however a little cumbersome for control of things like lights. Mostly the polling nature of demands and the single sided state machines (turn it on, just let it time out), don't work well for lights. When you hit a switch you want an instant response from the lights and you probably don't expect the light to go off 15 minutes later unless you press the switch again. My current thoughts are to move to MQTT for lights.

My Source code is here: https://gitlab.com/paulcam/home_heating

The ESP code in: https://gitlab.com/paulcam/home_heat...master/esp8266 (some legacy)
The demand scheduler etc in here: https://gitlab.com/paulcam/home_heat...ster/scheduler

You might want to also have a look at what Adrian Kennard (the owner of an ISP called AAISP) is also doing with ESP widgets: https://www.revk.uk/

The reason I'm encouraging you is that I think ESP is really the way to go, and if we can get to the point that we can have smart ESP sensors and a simple controller to manage them I think this is the next big thing for IoT-based home automation and be completely open source in the process. So many thanks for your engineering so far!

[paulca]

On OpenTherm. You are right that brute forcing the relay ON/OFF on the power feed to the boiler is certainly not a wise option on a new gas boiler, even though it might appear to work. The option I was considering was placing a relay on it's "Volt less" thermostat lines as if it were using a single wired mechanical thermostat. I'm fairly sure this is still supported.

Set-Point temperatures concern me though. This is how things like HoneyWell work. However to implement what I currently have with such a system would cost me hundreds or closer to a thousand pounds in sensors and hubs and .. of course... only work with HoneyWell devices, although I believe there are some hacks into the APIs and integrations with more open hubs.

Maybe taking too simplistic a view on set-points I formed the following questions.

Which set-point? Living room? Office? Bedroom?
What sensor - where does the data come from for it to decide?
Different set-points? I don't want all my rooms to be at the same temperature. Nor do I want it turning the boiler down because the living room is 21*C when I'm trying to heat the garage that is -2*C.
Foreign control in the system - What if I have more than one way to heat things and the gas boiler is only one of them, the rest beyond it's knowledge or control? Can I tell it to "stay out of it".
How does multi-zone control work and what if the radiators valves are not specifically designed for that boiler's proprietary protocols?

What if I want to do my own set-point logic, monitor the sensors and tell the boiler to target a "loop temperature" rather than a room temperature? Then control the radiators myself etc.

Maybe OpenTherm has all this covered I don't know. If it's just a remote control for the heating hub built into the boiler and I have to have sensors and radiator valves designed for that boiler I'm probably not interested. Not unless OpenTherm provides a way to bridge and or interface outside devices into that system, even then I'd be resistive, although I can always send it faked readings to control it, but that's hacking.

On InfluxDB. The thing you seem to point to as a failure of RRD is it's greatest strength and why it is the industry standard for such storage and has been for many decades. It's simple, lightweight, fast, does ONE single job and does it well. It doesn't collect data, it doesn't display data (though you could argue as rrdgraph is part of package that it does, but it's different application and you are not required to use it), it doesn't have a server etc, it just stores data and provides ways to query it. In this sense it follows the Unix convention of small components that do small jobs extremely well that can be built up together to produce much more complex solutions. As an engineer and tinkerer this is the ethos I subscribe to and it is why that even after nearly 50 years Unix-like OSes are still going strong.

InfluxDB is more the Windows ethos, or one big application that does n dozen different things, jack of all trades, master of only a few.

InfluxDB page one. Run the server. So it requires a network server. Why? I don't need one. Compared to RRD this multiplies the complexity immediately. So it will need to provide something in return that makes it worth it. I can think of a few use-cases that a central server with a query engine could solve for me, so I'm not ruling this out. One such use case is gradient analysis for pre-ramp and over-shoot on heating targets. Querying historic data to work out how quickly the heating can raise the temperature by 2*C and turning the heating on early is an open item on my list.

One very serious black mark is that it has "Enterprise" and "cloud" versions. While those "could" just be support and management agreement options to make a bit of money, they often end up with "This feature is only supported in the Enterprise edition" clauses. If not now, when the bean counters get involved and get all Golham on it. Then instead of Enterprise requirements alone it starts to be useful and popular features that move to the licensed model.

I will definitely consider it. I need to review the data layer of my system anyway, to open it up to things like HomeAssistant and intergrate with other systems. Moving away from my in memory JSON "blob" will be a step up in complexity, but it might be a good move.

On humidity, you might be right. The home survey when bought the place was able to see evidence of mildew on ceilings, even thought they were freshly painted. He suggested "life style choices" around ventilation and heating. With the place currently being sub standard in terms of insulation with only 100mm attic insulation, no cavity wall insulation and no underfloor insulation it will be an interesting winter and I may have to increase the amount of heating I do even while not at home to keep damp at bay.

Currently the states are Outside: 83% Living Room: 44% (from the wall clock only as I have on indoor humidity sensor).

[paulca]

On ESPs. Yes I totally agree they are an amazing platform to do all manor of things.

The one downside of them in power consumption. They can consume around an average of 30-40mA of current. This pretty much excludes them from being battery powered. Even a large 3000mAh Lithium Ion would be dead in a day or two.

That said there are products such as the SOnOff RF Bridge which adds RF support to the ESP eco-system and allows battery operated switches and sensors that have battery life of months.

To that end I did this on Saturday.
https://www.youtube.com/watch?v=aZ-BMd1MXkw

It's RFSwitch->ESPurna(RFBridge) -> MQTT -> Custom MQTT subscriber -> Raises GroupLightDemand and my normal mechanics take over.

I'm not overly thrilled with the delay and for lights I need a more event based structure than the polled "demands".

[paulca]

Just one final comment on InfluxDB.

It is not round robin. I see some people see this as an advantage. I like the round robin nature, the files never grow. I never need to worry about running out disk space on a a 4Gb SD card and I don't care that older data drops in resolution and eventually dies. I prefer it that way. I don't think I care to have 1 minutes accuracy of data 10 years old.

I think my current RRDs are good for 10 years of data and consume exactly and always 4.3Mb of disk space.

I wonder how much diskspace InfluxDB will consume in 10 years, if it's even still around then.

[paulca]

So an update after a long pause.
In November I had my heating system changed to natural gas and a combi boiler. I spent the winter with just a 3 point timer.

With the Covid situation emerging in March and me working from home I found the time to reimplement the heating controls. This time I put my SOnOff in parallel with the standard day timer.

Until July it was just bug fixes and tweaks, in particular balancing my default target temperatures so that the heating would not be coming on when unnecessarily on spring and summer mornings. To do this I lowered the target temp for the living room which is the coldest room in the house when it's not in use.

I added an "Override" feature which takes a zone name, a target temp and a number of seconds for it to apply. This overrides all scheduled target temps for that zone. So I can set the living room to 16*C and it won't waste heating while I'm in the office upstairs all day. But when I've finished work or want to go watch a movie I can press a button to set the target temp for the living room to 20*C for 3 hours, knowing it will revert to 16*C automatically I don't need to remember to turn it back down. - Keeping the design ethos of - revert to safe and efficient policy with everything timing out on it's own unless action is taken to persist it -.

However late July my SDCard in my Raspberry PI failed. It didn't just start producing errors, it died instantly and completely. "No Medium Found - Please insert disc in drive F:"

While my code was all in gitlab, the RRDs with my 3 or 4 years of data were lost. As was all the configuration on Cacti for the graphs.

So, I decided I would give the modern alternatives suggested above a go. InfluxDB and Grafana.

To be honest, while I do still see some disadvantages of this tech stack the advantages probably out weigh them. Grafana is so much easier to set up graphs for. The default template graph panels are much better and the configuration beyond that much more intuitive than Cacti.

So here are some pretty pictures of the new setup:
Basic "gauge" dashboard.


New Temperature graphs:


Solar Panel:


The brand new power monitor with a ShellyEM - So new it's graph is less than 24 hours and it already proved useful as it indentified I had an electric supply fault that I was able to report and was fixed before it caused damage across the whole street.

[paulca]

And finally the visualisation I have been trying to produce for a while and finally got something nice working. The active demands for heating/lighting.


You can see around 7:30 when the heating came out of overnight mode into day mode, both the living room and bedroom demanded heating. As I have yet to put thermo-electric valves in yet, the only thing the system can do is demand the actual boiler come on to meet the demand.

(The presence graph is noisy because the phone keeps going to sleep every hour or so and dropping off Wifi for a while). EDIT: It (presence) was original made a Demand as demands have an expiry life stamped onto them, where are data points do not. But I'm not exactly sure it needs to be both a datapoint and a demand, so I might remove it from demands.

One of the largest changes behind the scenes was moving all the python micro-services that were running on the PI as basic Unix services into Docker using Docker compose. So now it's kind "deploy anywhere". I didn't even have to reflash my devices, I just moved the Raspberry PIs IP address to my Docker host and it all started working again.