Good progress. The PIC and hardware side is just about there now - only some tidying up. Focus has shifted away from that and towards host-side software.

Firmware

The key things that have been settled are:

• De-glitch has been added which has massively reduced the amount of dirty transitions being reported and allows better separation between noise and real signals
• De-glitch is now integrated with the RX Enable command so that the minimum event length is specified along with the Prescaler config
• Re-jigging output buffer handling in a way that disables interrupts through critical parts to avoid corruptions to the output buffers... no evidence that this was actually happening, but it is a risk when there are a lot more devices around.
• Since the improvements and avoiding running the host software in a VM there have been no more USB overruns detected
• Fixed an interrupt deadlock
• Added multi-packet sending capability (see below)
• Refactored the code to make it fit in the ROM

There is still a fairly substantial code cleanup needed at this point before it will be in a state that it can be released.

Never expected that...

One of the things I never anticipated when doing this was the very poor design of the protocols used by most devices. I had assumed that established engineering practices of a packet structure and encoding combined with some kind of validation would be the norm. That assumption has turned out to be about as wrong and is possible.

Every controllable device, and with the exception of the OWL Energy Monitor, all devices I currently have seen follow none of the engineering best practices I had assumed. A sad state of affairs indeed! I might look at designing my own protocol and my own firmware for devices in the longer term (make them more secure, reliable and safe), but for now I have to work with what I have.

The practices used in designing the devices I have resulted in an unexpected problem: it appears that with the asymmetric encoding and negligible preamble to get receivers settled in a consistent state, timing of packets (specifically the gaps between them) is rather important as early packets will often act as the preamble to settle the receiver enough to reliably catch a later packet. This explains the brutal approach taken with these protocols of sending large numbers of packets.

I had not made provision for accurate timing of the gaps between packets on transmission with the result that I can't reliably (OK, these devices will never really be reliable) control devices of this sort. For that I've had to extend the firmware to send multiple packets with precise gaps when needed.

Host Software

The basic Plug-in infrastructure is in place with a stable API, plus Symbol decoder classes for Manchester (OWL Energy Monitor), Pulse Width (STATUS Plugs) and Pulse Gap (Imagintronix Temperature & Humidity Sensor) which can be leveraged by other Plug-ins (eg. Mercury Extension) over time.

One unexpected side-effect of having a generic logging Plug-in (useful for catching new protocols when other Plug-ins don't pick them up) is that it also picks up failings in existing code. One case was 4 logged packets which turned out to be from the OWL Energy Monitor and showed that noise before the signal start was getting the Manchester decoder class in a twist. After making it more robust no further stray packets have been logged for that, but I did catch another device of some sort (higher data rate)... presumably something a neighbour has.

I also have a Test Harness that allows Plug-ins to be run against logged packets and this is very useful as a development tool and debugging robustness problems.

Another unexpected consequence of starting to decode things properly is that I can observe the trickier devices better. I've made significant improvements to the decoding OWL Energy Monitor leaving only a few things unknown about the protocol.

Considerable work has gone into avoiding anomalies generated by some devices. The STATUS plug is one case: the last packet transmitted by the remote lacks the last (stop?) bit and hence is ignored by the plug. If we however don't check for this we are not decoding the same way as the plug, but likewise we don't want to let the logging Plug-in pick it up. To achieve that there is now an ignore entry in the return hash which allows the Plug-in infrastructure to reset other Plug-ins but not to return the (invalid) result.

I've also spent time tidying up and refactoring what started off as quick test code into useful classes.

Next Steps

More Devices

Now that I've got the basics becoming more solid I'm starting to buy some extra devices. First up is another Temperature sensor which will hopefully be more robust than the Imagintronix. The aim is to be able to put it outside and unlike my existing USB sensors won't be constrained by where I can get a cable to. That's been a problem so far with sun hitting the side of the building where the sensor is and giving false readings.

I'm also looking for some more remote controlled plugs, in particular multi-way extensions which can be individually switched would be very useful (got one now).

Network Daemon & Hooks

At this point I'm looking towards making things more useful so that the system can be integrated into existing Home Automation tools. The obvious thing is to turn the tool into a network daemon which will allow multiple nodes to be put all over the place and remotely addressed over the network.

By default the daemon will be in Receive mode waiting for activity, but when transmissions (via Plug-ins) are requested it can flip to Transmit mode transmit the requested packets and revert back to Receive mode.

One key thing is to make the daemon as flexible as possible, so I've decided that we need a <Plug-in>.d/ directory and any packet received can then trigger an appropriate script to decide if it should be acted on and how. This will mean that I can easily hook sensors (Temperature, Energy etc.) into Cacti, and automatically create further events in response to them (send emails, turn things on/off etc.)