Seven Days of Data

We put an Outlier Node on a motor over Christmas break. The plan was simple: leave it running for three weeks, come back, pull the SD card, and find out what three weeks of vibration data from a real machine looked like. It was the first time the board would run unattended anywhere other than a bench.

When we pulled it off, the log ended after seven days.

The board had not crashed and it had not been knocked loose. It had run out of power. A sensor built on the promise of a six month battery managed one week, and the fortnight of data we most wanted, the part where you learn what a motor does when nobody is watching it, did not exist.

The failure that does not announce itself

A device that dies loudly is easy. You see it, you fix it, you move on. This one died quietly. Every signal available at the moment of installation said it was working: it powered on, it wrote to the card, the status LEDs did what they were supposed to. The failure only became visible three weeks later, and only because we went looking for the data.

That is the part worth sitting with. The whole premise of the product is that these sensors go on machinery nobody visits, in places with no network, and get collected by a technician walking past weeks later. A sensor that stops logging silently is worse than one that never starts, because you make plans around data you believe you are collecting.

Measuring instead of reasoning

The temptation was to argue it out on paper. Datasheets have sleep currents in them. You add those up, divide the cell capacity, and produce a number that says everything is fine. We already had that number and it said six months. The board had just spent a week demonstrating the number was wrong, so more arithmetic on the same assumptions was not going to find the problem.

So we bought a coulomb counter and put it inline with the battery. A coulomb counter integrates current over time, which is precisely the question we had: not what is the board drawing at this instant, but where did all the charge actually go. It converts an argument about datasheets into a measurement.

What we found

The microcontroller was sleeping correctly. The things around it were not. The SD card module and the MPU-6050 accelerometer stayed powered straight through sleep, drawing steadily the entire time the system believed it was doing nothing. Neither looked alarming on its own. Added together and multiplied by twenty four hours a day, they were the entire budget.

This is the trap in a low power design. The sleep current that matters is not the microcontroller's, which is the figure printed in large type on the front of the datasheet. It is the sleep current of the whole board, including every part that has no idea the system is supposed to be asleep.

The fix

Three changes across the following board revisions. High side MOSFET switches, so a GPIO can cut supply completely to the parts that do not need to be alive between samples rather than trusting each one to power itself down politely. Diodes to block a reverse path that was letting current back in where it had no business being.

The third change is the one I did not expect to matter as much as it did. The original SD card module needed more than 3.3 volts, which meant the board had to carry a higher rail just to feed it. Moving to a smaller module that runs happily at 3.3 volts let the entire circuit collapse onto a single rail. One fewer rail is one fewer regulator, and a regulator costs you quiescent current every second of every day whether anything is using it or not. Choosing a part that matches the rail you already have beats adding a rail to match the part.

The principle behind all three is the same, and it is what I would tell anyone starting a battery powered design: do not ask a component to power itself down. Take the power away.

What I would actually change

The leakage was a bug and we fixed it. The deeper problem was that the device could not tell us it was dying, and that is a design fault rather than a component one.

A sensor meant to sit alone for months should log its own health alongside its measurements: battery voltage, or coulombs consumed, written into the same record as the vibration data. The seven day gap would not then have been a mystery discovered in January. It would have been a line trending downward that we could see, and the fix would have landed a revision earlier.

We lost two weeks of data from a motor. What we got back was the reason the six month figure was fiction, which was worth more, though it did not feel that way in the moment.