Hi! DIY Home automation system build.

[paulca]

Hi guys,

I'm Paul, 44, software engineer from Northern Ireland. I have been slowly hobbying up a smart home system for a while now, I thought I would share some of it with you guys and share some of the ideas for where it goes next.

So far I have a data network in the home. It's very simple, as complex as it currently needs to be. Devices send JSON blobs via UDP to a hub. The hub merges the JSON into a single lump of data and makes it available to any other device that is interested. Presently the whole dataset is sent if you as much as connect on it's TCP reading port, but this could become a REST API.

So for example I have a few Raspberry Pis and a few ESP8266's with temp sensors shoved in discrete places. Every 5 seconds they wake, read the temp and send a JSON string to the hub. Something like:

{"type": "float", "units": "'C", "key": "livingRoom", "value": "21.81",  "shortName": "LR", "name": "Living Room"}

the hub then adds this to a larger tree of data and adds a timestamp to the json object to allow stale data detection.

I also have a monitor on my solar panel controller, which uses an RS485 to serial converter and an ESP8266 to transmit it's parameters every 5 seconds to the hub.

The only client I have currently is a small RaspberryPi enclosure with a 16x2 screen which just displays all of the data 2 at a time on the screen in a loop.

Additional to collecting and sending data, the hub also, every minute, dumps a snapshot of the data into RRA archives. If new data arrives it will automatically create a new RRA for it. I have a web application called "Cacti" which provides management for graphing this RRAs on demand, allowing me to view graphs of data over the web.

RRAs are round robin archives. They are fixed size, multi layer, sequence database. Mine store 1 minute resolution for 2 years, 5 minute resolution for 10 years and 1 hour resolution for (I can't remember... a long time). They never get larger, never fill you disk space.

I am currently moving on to the first real automation. The heating controller.

I will follow up with a post of the first version approach to heating scheduling and then the future ideas for making it more of a multi-zone, presence aware, demand based system for when I upgrade my heating a bit.

Here are a few images:
Solar panel, RS485 to Wifi monitor box:


Raspberry PI Monitor/Display:


Graph output from RRAs

[paulca]

So as i said the first thing I intend to control is the heating. Presently I have oil fired heating where the boiler is switched directly by the heating controller relay. The wiring diagram for an external thermostat is, regrettably, just putting another relay in series with the heating power feed. So this means switching current, but not that much. The start up surge current, as best as I can gather is much less than the 10A capacity of a SonOff or an opto-isolated relay. I prefer the SonOff, though I have never used one yet, at least it doesn't involve frankenstiening up an ESP8266 and relay board. The SonOff is just simple, contained and fairly safe.

So scheduling the heating. Version 1 will just be a true/false output for heating.

So far I have a basic Python application running on the hub, however due to the distributed nature of the system I already have, the heating scheduler does not need to be on the hub and will actually connect to get it's data, even if that is "localhost" just.

The general theory/design of version 1 is a Scheduler which is a collection of Schedules. Schedules are in order of priority. The first one to return "Active" sets the heating to ON, if none return active the heating is OFF. The heating requirement will be published back into the JSON data array. A further script/program/MCU will periodicly poll this data looking for something like {"heating": {"state": "True"}} and (assuming the SonOff) send an event to turn the relay on.

A Schedule is a collection of conditions and their modifiers. I have already written conditions for "Time of day", "Day of week", "Temperature minimum". So, for example, my weekday morning schedule might be:

TimeOfDay 06:30 - 07:45
AND DayOfWeek 1, 2, 3, 4, 5
AND Temperature livingRoom 20

A few utilitarian schedules will exist with much higher priority, for example:
Temperature livingRoom 10
OR
Temperature bedroom 10
OR
Temperature garage 2

Another, slightly more complex, schedule will need the last time the heating was run placed in the dataset and based on this value, run the heating for 15 minutes if the heating has not been run in 23:44 minutes. Call it the "exercise schedule".

There are obviously a few over simplifications and I have a few implementation details proving somewhat tricky, but it should be functional. This is mostly coded and I'm just going to order the SonOff to test it out, with a table lamp at first, then with the heating itself.

There are also some fail safe items to provide here. For example the timestamp on the "Heating On" data will allow the heating controller to detect the "command" stale is stale and the scheduler should have been refreshing it every minute and will ignore it and turn the heating off. With the SonOff, I can flash the firmware to do similar, to make a single "ON" command only be valid for 1 hour before switching off.

I have to get to work, so I'll post the Version 2 design, which is were it get's interesting.

[paulca]

So before I outline my design for the heating control scheduling, a small aside.

I got the SOnOff basic switches in the post. They didn't work at all with the supported software. This was only slightly irritating as I fully intended to flash them with my own software, but I would liked to have seen them function as intended first.

So the current R2 models use an ESP8285, not an ESP8266 which renders most of the custom firmware options a little more than drop in replacements. The software itself will work, but there are differences in the flashing process etc.

Anyway I took the default webserver code that comes with the stock ESP8266 and implemented a basic REST api on to the switch. I have three end points:
http://the.switch/on - turns in on
http://the.switch/off - turns in off
http://the.switch/status - tells you if it's on or off

For the heating switch this will be extended to have:

1. The button single press will turn the heating on or off, of course the scheduler might just turn it off or back on again. This is primarily for if the hub is down or offline for whatever reason and I need heat.
2. A long press on the button will override the automation for 1 hour and run the heating regardless of what the hub asks for.
3. Anti-short cycle. Even though this will be implmented by the gas boiler, I choose to add it anyway. So if the heating has been turned off, it will refuse to turn back on within 1 minute. This prevents broken software from rapidly toggling the heating on and off.
4. "ON" request timeout after 1 hour. So if the hub crashes when it was about to turn the heating off and leaves the heating running it will timeout after 1 hour unless the /ON command is refreshed periodically.

These ITEAD.cc devices are almost all based on ESP8285 and a small amount of research, a £5 USB Serial/TTL converter and the Arduino IDE and you can put your own custom software on them.
https://www.itead.cc/

[paulca]

Some rather pretty graphs from my system:

Ambient temperatures. Note the huge spike in the garage temp when the clothes drier is running!


This should make detecting day and night very easy, solar panel voltages!

[paulca]

So heating control. Now bear in mind I am kind of deliberately doing this "clean room" style. That means I am not really looking at what others are doing or how they do it. I am trying to design this from scratch based on what my view of heating control is.

Version 1, which is more-or-less coded and needs maybe an evening to finish is basically a more advanced 7 day timer with temperature awareness. However the schedules themselves would be much more complex as I do not need to provide a UI for them, I can simply log into the Raspberry PI hub and edit a few lines of code to change them. Though there is more on that later.

I would say the code is "simple", but that's subjective. It's aiming to provide simple building blocks from which complexity can be constructed.

A Scheduler contains a set of Schedules. It basically moves through each Schedule in turn, hands it the current data from the network and asks it if it's active or not. For future proofing the whole set of active schedules is returned in the order they were queried. Order is important as it provides priority. However currently the first active schedule is enough and "wins" control. (Again more on this later).

A Schedule then is, in it's basic form, is a set of Conditions which the Schedule "evaluates" to determine if it should be active or not. Conditions come in a variety of forms; TimeOfDay, DayOfWeek, DayOrNight, MinimumTemperature, TemperatureBand, etc. etc. Even, though not obvious, HeatingOn condition. Conditions are attached to an operator, such as AND and OR. So when placed in an ordered list these give an expression.

So an example. Morning, week day schedule:
OR TimeOfDayCondition 6am - 8am
AND DayOfWeekCondition Mon-Fri
OR MinimumTemperature LivingRoom 18*C
OR MinimumTemperature BedRoom 18*C

The logic is such that a "False" immediately stops the evaluation. It's a bit brittle this and I'm not happy with it. It takes quite a bit of thought to work out what that series will or won't do for a set of circumstances. But it's version 1.

If the full expression returns True the Schedule considers itself active and returns "Active"

If you are still reading... Returning active in version 1 will simply re-publish a new JSON blob, or update the existing one with "HeatingDemand": "True" and also "ActiveSchedule":"Morning Weekday"

So single zone, single active schedule, single on/off. For now.

Another script or completely separate device can poll that demand status and send/activate the correct device to turn the heating off.

There is logic in my madness here. It's keeping things modular. In fact there are the following "modules":

Sensors and devices.
The data hub / logging. The heart of the thing.
The scheduler. As discussed here.
The heating (in my case) controller.
The SOnOff device doing the switching.

The layered separation means that, for example, I can present an HID (interface) to the heating controller allowing me to completely override the heating and take "manual" control. It also allows a layer in which to evaluate "acceptable use" of the device being controlled. Short-cycling, on to long, cycling, minimum time on, etc.

I can also intervene at the SOnOff switch itself of course.

The "More on that later"'s will have to wait to the next post, but are mostly about Version 2 which I try to address various issues with the above approach.

One I'm actually struggling with is negative demands. A real world example. The set up is doing it's job on a Friday evening, it's 22*C, but I'm going out and staying at a friends, so I want a way to say, "Heating off.", or "I'm out" mode. I can, intervene at the "Heating controller", but I want something smarter than just switching it off.

I think it might be better to play with the target temperatures overriding the active schedule itself.

[paulca]

A little off-topic from heating control, but a cool feature that popped up.

I have 4 lights in my living room. All are plugged into wall sockets. On an evening I typically have to go around the room and switch each one on at the wall. When it's bedtime I have to go and switch them all off again.

A light bulb moment (excuse the pun) was when I realised I could have any one of those switches (if a smart switch) raise a demand for it's peers to be on too.

An afternoon coding and I have created what I call "Grouped Lights".

My custom firmware on a SOnOff switch allows you to:
1. Switch it on/off by pressing the button.
2. Switch is on/off by REST http request
3. Have it publish a demand for it's group to the network by long pressing the button.

The demands are published as JSON and a "GroupLightController" Python script polls these and when it sees a change in demand for the group of lights it commands they to swtich on/off via REST.

Also, here is the code for the whole system as it stands now.
https://gitlab.com/paulcam/home_heating

[paulca]

So I finally got the heating control implemented and functioning. I did have to pull a feature or two out of the Version 2.0 stack, but as "Demands" had already been used for the group lighting I figured I would add it to the heating control too to keep it consistent. It adds so much more flexibility and makes the schedules more dynamic.

Previously the schedules where a set of conditions that returned true if those conditions where satisfied. If any returned True the heating came on.

Now with demands, any and all schedules return a "Demand" object. The Demand object has:

* name
* state ON/OFF etc.
* expiry
* timestamp

Demands are then processed to determine what action can be taken to address them. This adds a decoupling between the schedules and the actual outcomes, meaning that schedules don't need to know what they are controlling or how.

Schedule demands are not published, but recalculated and processed every 30 seconds. They "could" be published to the network hub, but currently aren't. All this means is that the Demand Processor for schedules has to run in the same process, in fact currently the Demand Processor is directly called by the scheduler. This just makes things easier.

A schedule might raise a demand for a single room, here is an example:

{"heating":{"key":"livingRoom", "demandState":"ON", "expiry": "0"}}

Expiry is not important for scheduled demands as they are instantaneous, like events.

Seeing this demand, the demand processor in my current real world home has no way to provide heating just to the living room and so all it can do is put the whole heating on. This brings us to another demand, for the heating to be on. Stay with me here The schedule raises a demand for the action it wants, "heating -> livingRoom", the processor adapts this to what it can do about it and raises a demand for that to happen.

Presently there is a single switch for the heating, but that might not always be the case, so the relay on the boiler does NOT subscribe to this demand. Instead, in another layer of abstraction, the "Heating Controller" polls the demand for the boiler to be on and sends the actual command to the boiler relay. So the chain is:

With the network data a Schedule decides heating is needed in the living room, it returns a demand.
The Scheduler sees the demand and passes it to the Demand Processor.
The Demand Processor publishes a network demand for "HEATING", "ON"
The Heating Controller see this demand and, if appropriate, turns the heating on.
The SOnOff custom firmware decides if the relay should or should not be allowed to toggle and switches the heating relay on.

The SOnOff Wifi relay is running custom software that provides seamless transparency to manual operation. It supports:

* Comes on when power up. When power goes off it goes off. Makes the SOnOff "disappear" and allows full manual control (with caveats).
* Single press of the button toggles the relay, if short cycling rules are obeyed!
* Long press publishes a demand for one hour in the opposite state the relay is currently in (Manual 1 hour boost with automation help of course)

So if my Raspberry PI dies in the dead of winter, corrupting it's SD card, while I have backups it might take a day or two to find the time to restore it to health. That's fine. I can just use the normal boiler control panel scheduler. If I turn it off the SOnOff resets so it will come one when powered. It's in series with the normal wall panel controller. I can still also directly send HTTP on, off, status requests to it.

[paulca]

So... presence awareness.

I didn't like that the following schedule:

6:30-8:00 Mon-Fri 17:00-23:00 Mon-Fri - Living Room - 20C

would not account in anyway for me to be "out" during the evening. Nor did I like it would go off and not provide heating if I was working from home one day.

So I needed to know if I was home or not. My first thought was my phone. It always connects to my Wifi within seconds, so all I need to do is detect it there. First I tried pinging it, figuring I could give it a static IP. This worked until the phone went to sleep, then it stopped responding. So then I tried rumaging around with nmap and arping. Still no response when sleeping and it could sleep for 20-30 minutes without answering a single ping or probe. I then tried a Tasker profile to send a notification of presence every minute while connected to the Wifi, but ... the Android over ruled it and shut the network off so that it still couldn't send the notifications. Pheewww.

Finally on a long shot I used "SNMP" to scan the TP-Link router and BINGO I found a dynamic list of connected DHCP assignments. When my phone connects it is listed within 10 seconds. When my phone disconnects it is unlisted in 20 seconds or so. So I wrapped this in a bit of python, ran it as a service and it publishes (you should see a trend ) a notification of my presence, either 1 (in), or 0 (out).

Finally tying it all together with a MultiRoomTempSchedule and a PresenceCondition I have the following:

target_temps = {
        "default": {"livingRoom":6, "bedroom":6, "garage":1, "office":6},
        "eco": {"livingRoom": 18, "bedroom": 16, "garage": 2, "office": 18},
        "preferred": {"livingRoom": 20, "bedroom": 18, "garage": 2, "office": 20},
        "max": {"livingRoom": 30, "bedroom": 30, "garage": 30, "office": 30}
    }

A maintenance schedule without presence detection, active 24/7, 365 is told to use "default" temps.
A routine schedule without presence detection, active when I am 'routinely home', is told to use "eco" temps.
Finally a presence aware schedule active "all day" only if I am present, that is told to use "preferred" temps.

Sample logging out put at key times:

Aug 27 06:25:46 raspberrypi python3.5[22639]: Maintenance Schedule Active - checking temps default
Aug 27 06:25:46 raspberrypi python3.5[22639]: office: 22.31 < 6
Aug 27 06:25:46 raspberrypi python3.5[22639]: garage: 16.5000 < 1
Aug 27 06:25:46 raspberrypi python3.5[22639]: livingRoom: 19.12 < 6
Aug 27 06:25:46 raspberrypi python3.5[22639]: bedroom: 21.56 < 6
Aug 27 06:25:46 raspberrypi python3.5[22639]: Active demands []

At 06:30 however....

Aug 27 06:30:30 raspberrypi python3.5[22639]: Weekday Morning Routine Active - checking temps eco
Aug 27 06:30:30 raspberrypi python3.5[22639]: office: 22.31 < 18
Aug 27 06:30:30 raspberrypi python3.5[22639]: garage: 16.5620 < 2
Aug 27 06:30:30 raspberrypi python3.5[22639]: livingRoom: 19.06 < 18
Aug 27 06:30:30 raspberrypi python3.5[22639]: bedroom: 21.50 < 16
Aug 27 06:30:30 raspberrypi python3.5[22639]: Presence Schedule Active - checking temps preferred
Aug 27 06:30:30 raspberrypi python3.5[22639]: office: 22.31 < 20
Aug 27 06:30:30 raspberrypi python3.5[22639]: garage: 16.5620 < 2
Aug 27 06:30:30 raspberrypi python3.5[22639]: livingRoom: 19.06 < 20
Aug 27 06:30:30 raspberrypi python3.5[22639]: bedroom: 21.50 < 18
Aug 27 06:30:30 raspberrypi python3.5[22639]: Maintenance Schedule Active - checking temps default
Aug 27 06:30:30 raspberrypi python3.5[22639]: office: 22.31 < 6
Aug 27 06:30:30 raspberrypi python3.5[22639]: garage: 16.5620 < 1
Aug 27 06:30:30 raspberrypi python3.5[22639]: livingRoom: 19.06 < 6
Aug 27 06:30:30 raspberrypi python3.5[22639]: bedroom: 21.50 < 6
Aug 27 06:30:30 raspberrypi python3.5[22639]: Active demands [HEATING livingRoom True 0 0]
Aug 27 06:30:30 raspberrypi python3.5[22639]: Existing heating demand NONE heating ON 1566805485.240631 300
Aug 27 06:30:30 raspberrypi python3.5[22639]: Raising demand: {"heating": {"key":"heating", "demandState":"ON", "expiry":"300", "category": "CONTROL"}}

Presence Schedule Active raised the demand for heating at the living room is under 20C.

The "expiry" 300 is important. The heating commands have a minimum ON time for 5 minutes. This prevents it toggling on, off a lot and works well. The expiry also means that nothing has to switch the heating off again, it's demand will simply expire, a schedule is of course free to demand it for another 5 minutes every 30 seconds to keep it on.

The heating controller ended up doing this:

Aug 27 06:30:56 raspberrypi Heating Controller[22859]: {"heating": {"category": "NONE", "key": "heating", "demandState": "ON", "timestamp": 1566883830.6425545, "expiry": "300"}, "presenceHome": {"category": "NONE", "key": "presenceHome", "demandState": "ON", "timestamp": 1566883735.8001761, "expiry": 120}}
Aug 27 06:30:56 raspberrypi Heating Controller[22859]: Heating demandState ON demandExpiry 300 timestamp 1566883830.6425545 currentTime 1566883856.7965205
Aug 27 06:30:56 raspberrypi Heating Controller[22859]: Demand state True current state False
Aug 27 06:30:56 raspberrypi Heating Controller[22859]: Commanding heating on http://10.0.0.9/on
Aug 27 06:30:57 raspberrypi Heating Controller[22859]: <Response [400]>
Aug 27 06:30:57 raspberrypi Heating Controller[22859]: <Response [400]>
Aug 27 06:30:57 raspberrypi Heating Controller[22859]: Sending JSON datum to 10.0.0.3: {"heating": {"name": "Heating", "value": 1, "units": " ", "key": "heating", "timestamp": 0, "shortName": "HT", "type": "bool"}}
Aug 27 06:31:27 raspberrypi Heating Controller[22859]: {"heating": {"category": "NONE", "key": "heating", "demandState": "ON", "timestamp": 1566883830.6425545, "expiry": "300"}, "presenceHome": {"category": "NONE", "key": "presenceHome", "demandState": "ON", "timestamp": 1566883735.8001761, "expiry": 120}}
Aug 27 06:31:27 raspberrypi Heating Controller[22859]: Heating demandState ON demandExpiry 300 timestamp 1566883830.6425545 currentTime 1566883887.8853893
Aug 27 06:31:27 raspberrypi Heating Controller[22859]: Demand state True current state True
Aug 27 06:31:27 raspberrypi Heating Controller[22859]: Commanding heating on http://10.0.0.9/on
Aug 27 06:31:27 raspberrypi Heating Controller[22859]: <Response [200]>

This shows a bug. Because there is a minimum "off time" on the boiler, and the fact the heating controller sends an off signal every minute if the heating is off.... the first "on" command is rejected with a Response <400>. The second one, 1 minute later works fine. The SOnOff is now "on".

Justification for this 'feature'... OFF means OFF. So the OFF is always accepted and resets the "offTime" for short cycle detection. I think I can work around it though.

Obviously it send "on" a few more times, but as the sun was rising to come in the living room window it was set back to OFF pretty quickly.

Note at 08:00 when I was supposed to be leaving for work this happens:

Aug 27 08:30:17 raspberrypi python3.5[22639]: Presence Schedule Active - checking temps preferred
Aug 27 08:30:17 raspberrypi python3.5[22639]: office: 22.44 < 20
Aug 27 08:30:17 raspberrypi python3.5[22639]: garage: 16.7500 < 2
Aug 27 08:30:17 raspberrypi python3.5[22639]: livingRoom: 20.06 < 20
Aug 27 08:30:17 raspberrypi python3.5[22639]: bedroom: 21.81 < 18
Aug 27 08:30:17 raspberrypi python3.5[22639]: Maintenance Schedule Active - checking temps default
Aug 27 08:30:17 raspberrypi python3.5[22639]: office: 22.44 < 6
Aug 27 08:30:17 raspberrypi python3.5[22639]: garage: 16.7500 < 1
Aug 27 08:30:17 raspberrypi python3.5[22639]: livingRoom: 20.06 < 6
Aug 27 08:30:17 raspberrypi python3.5[22639]: bedroom: 21.81 < 6
Aug 27 08:30:17 raspberrypi python3.5[22639]: Active demands []

The morning routine schedule has ended, but the presence aware schedule is keeping the "preferred" target temperatures active.

Next steps have changed a bit now that I have it "in service". What I badly need now is a UI. First up probably the most helpful will be on the monitoring side of things. The current 16x2 LCD is having issues showing things like the heating status and presence, so that needs fixed. Then it would be nice to see the target temps for the various "modes" and to change them. Also the ability to manually hack the active demands and other low level stuff from my phone would be nice.

Then I need to return to the over-ride problem. The schedules are fairly dynamic in terms of temperature, presence etc. but they are still fixed. Things change, routines change and while they are easy to edit or change via a UI I would prefer a way to "over ride" certain things until conditions are met or time has passed etc.

I can think of several ways to achieve these things at different layers in the system, but haven't exactly what yet. An emerging idea is basically "Profiles", which are a set of things like Schedules, Conditions, Target temps, Controller config etc. That can be switched manually and have an expiry set. Consider:

Profile NORMAL
Profile ALL OFF
Profile ECO
Profile HOLIDAY
Profile DEMO - this could be cool, especially with lighting

They could be manually enabled and have a kind of exit condition, much like Tasker profiles on Android.

[paulca]

Quick update on the first proper test of the presence detection as I was working from home yesterday.

So it passes on both fronts. Yesterday when I was in all day it remained active and kept my temps at my preferred targets.

Today I left for work somewhere around 08:10 and I see in the logs at 08:15 the presence data went to 0 (AWAY). I can't confirm how accurate that was as I didn't note my exact leaving time, which I will try tomorrow. I know it responds to a administrative Wifi disconnect on the phone within a matter of seconds, but not sure how long it takes if I just go out of range. Based roughly on my morning this morning I'd guess it took at most 4 minutes to mark me as "AWAY". I can live with that. Not quick enough for "security" automation, but quick enough for heating/environmental.

I also see in the logs that within seconds of arriving home the presence schedule activated.

[DerekWilliamsUK]

Please keep your progress/updates coming. This makes great reading. Thanks.