nodemcu/thingspeak IOT data analytics

This is a more detailed build post about my crabgrass soil temperature monitor.

The general goal here was to get widely accessible data with configurable visualizations on the web, with a minimum of $ and backend development and very long (~3 months or more) life on a pair of lithium AA batteries.

Hardware: I have been using arduino nanos for this kind of thing a lot, but getting onto the network has always been a pain, as has the small space available for programs and data storage.  With this in mind, I started looking into ESP8266-based boards. The best combination of performance and price seems to come in knock-off nodeMCUs, so that's what I chose.

For a sensor, I was planning initially to use a regular old thermistor, BUT for long battery life, I want to avoid voltage regulators.  That means that I can't feed my thermistor half-bridge with a known input voltage, so I need to log input voltage as well as output voltage (and battery life).  This is a problem, because the nodeMCU only has one analog input.  So, either I need to add on an I2C ADC (which would be a pretty good solution), or I need to get away from so many analog inputs.  I then came across the very-common, cheap, and mostly-waterproof dallas onewire DS18b20.  This is perfect, as I can feed it anything from 3-5v and it should work ok.  I plan to also to do a bit of testing at lower voltages to see what it does to the temperature readings).

So, anyway, it's a nodeMCU (with the arduino IDE) with a DS18b20 for temperature, a pair of AA batteries connected directly to the 3.3v line for power to both the nodeMCU and the DS18b20, and also to the analog input to monitor battery condition.

IOT system: Initially I thought I could do it with mozilla's host-it-yourself IOT system, but at the moment it seems to be much more tailored to IOT control systems (like light switches), and really not there yet for data collection and presentation. There are MANY other IOT data systems out there, but most of them have a very restricted free tier.  I narrowed it down to Adafruit's IO.adafruit.com, and mathworks' thingspeak.  Both have a good free tier, but when I dug in a bit I decided that I wanted longer-term data hosting than the 30 days that the free adafruit tier comes with, and I also like the integrated matlab analytics that comes with thingspeak.  So to thingspeak I went.

Once that decision was made, I put a script together to read the temperature and battery voltage, send them to thingspeak, and go into deep sleep for some time (the logging interval).  This worked fine, but if the logging interval is anything shorter than about an hour, it burns through the batteries quite fast. When the wifi is connecting I get pulses of up to 170ma current draw, and while sending/receiving data it seems to pull about 70ma.  When it's in deep sleep, it draws about 20ua, and when it's reading the temp sensor with the wifi off, it only draws about 20ma.

For this application, a few temp readings a day is probably plenty, but in general that's not a very satisfying solution.  The next idea was to LOG data more often, but only TRANSMIT it a few times a day. The problem with this is that when the board deep-sleeps, it starts the sketch over from the setup section, and loses any data stored in variables. However, the nodeMCU has a bunch of flash memory, that can be read and written to by the sketch, and does not get cleared on a reset!  Perfect!

But (isn't there always a "but"?) thingspeak timestamps data by when it is transmitted.  And now, if I'm transmitting data at times other than when it was logged, I have to supply a timestamp for each data point.  Probably the best way to do this is to add on an I2C real-time clock, but in an effort to simplify the assembly (and since I don't need high precision timekeeping), I decided I would just dead-reckoning the time during the datalogging and sleeping periods, and resynchronize with network time every time I connect to upload data.  This is going to give me awful time precision, but at least the drift will be reset twice a day.

So, now what happens is, the datalogger wakes up and looks at the dead-reckoned time and the time at which it should upload data. If it's not data-upload time yet, it reads the battery voltage and temperature and adds them to the flash datalog file. Then it goes back to sleep (in this case for another 15 minutes).  If it IS time to transmit, it connects to wifi, sends the logged data line by line, clears the log, resets the time to network time, logs a new line of data, and goes to sleep.  The only real hangup here is that the free tier of thingspeak will only accept 1 transmission every 15 seconds, so the wifi is on for a long time while transmitting data. If I pay the $95/year that the "home" thingspeak license costs, that goes down to 1 second, which will help me save a lot of battery power.

Then, once thingspeak has all the data, it converts the unix timestamp to a normal date/time and makes all the plots.  I plan to also set it up to issue some kind of warning (email, ideally), when the temperature thresholds are crossed.

Code is here (I'm still learning to github properly, but the code should at least be available):
https://github.com/Matt-K/IOTsoiltemp