Back from the Brink

I have my weather website running again.  It was a major PITA to get things going.  I am partially to blame for not having a good backup system in place.  But since it is such a hassle to get things working, when I finally do, I'm almost afraid to mess with it.

As previously mentioned, I believe the SD card on the Raspberry Pi failed.  It is actually not that easy to know for sure, but the Pi itself seems OK with another card installed.

So the challenge was to get Weewx up and running again with a new SD card.  I had to reload all of the required software.  Of course the latest version of Weewx was 5.1, and I had been running 4.10.  Here is a list of software I had to load:

• Weewx
• Apache
• DDClient (for dynamic DNS)
• Fswebcam

Settings for those often had to be added, as well as adding some udev rules to workaround the flakey Davis serial to USB converter, setting all kinds of permissions.  Weewx 5 also had some major changes to get used to.  So I spent a ton of time visiting the Weewx website and guides.

Unfortunately, many needed files were on the "bad" microSD card.  I figured I probably could recreate most of what I needed, except for 16 years of weather data stored in a sqlite database.

I tried so many things, I can't remember most of them.  The files I needed could not be read by windows.  I attempted to read the files on the Raspberry Pi using a SD to USB adapter with no luck.  I eventually found some software (and tried many) that would run on Windows such that I could recover some files from the SD card.

Sadly, the main one, the weather database, initially appeared corrupted for both Weewx and SQLite.  Again, I found another tool that "recovered" the database, though this whole process took hours and hours of Google searches and trial-and-error.

Eventually I had a usable database again, but missing some data.  The last few years of data was missing, or partially missing (some months).  Again, I was able to get most of that back.  Still waiting on the image graphs for the last year, not sure when they are generated.

In all, it was the better part of two days work.  I'm no Linux expert, so I basically need to refamiliarize myself to the basics every time I need to delve into it.  

It seems like there is some major problem/failure/etc that happens every couple of years that requires a huge effort to overcome.  At least it hasn't cost me anything but time, so far.

Raspberry-Pi Issues

 For a couple of days I was unable to access my web page.  I was getting similar errors off and on, so I thought maybe it was a temporary problem.  Finally, I checked the Raspberry-Pi and there were some problems.  I had a desktop screen, but most everything was missing.  I pulled up a command window, and it didn't seem to recognize any commands.  I really was confused as to what was going on, so I did a power shutdown.

After powering back up, it seemed to like it was trying to boot, but eventually the screen just went black.  I did a few recommended checks on the Pi, and it seemed it was OK.  Then I installed a new SD card with a fresh version of Raspian and it booted up fine.

It indicates that the SD card is corrupted, which I understand is a common issue.  I don't think I ever got an automatic backup working right, so no backups stored anywhere.

My next step was to try and recover files from the SD card.  That is in process.  But I also will need to reload all the software to the new SD card and get everything configured.

So no WX for a few days.

New Temp/Humidity Sensor Installed

After several nights of working, the sensor had a fairly lengthy period of no readings of several hours on the 18th and 19th of July.  So it was clear that it did not suddenly fix itself.  The new sensor arrived on July 19th, and I was able to install it that afternoon.

So far, so good.  Hopefully the problem is resolved and I can get another 15 years out of this one!

Intermittent - Now Worked 5 Nights Straight

 So of course the next day after ordering the new sensor, it has worked through the night for 5 days.  Not that it would have changed my mind, though I might have held off a few more days.  It did drop out for somewhat shorter periods for a couple of days, but now has had a streak of solid readings.  It might help that the nights are short, as seems voltage related, but other sensors are OK.

I still feel it is intermittent, and probably on the way to a complete failure.  The new sensor just shipped today from California, so it won't be here until next week.  It will give a few more days of readings so it will be at least interesting to watch.

Ordered Temp/Humidity Sensor

The first night after complete disassembly and cleaning of the ISS, the temperature and humidity worked through the night.  Still,  I wasn't too hopeful the problem was resolved.

I had done some research and found this website:  https://www.manula.com/manuals/pws/davis-kb/1/en/topic/general-please-read

This is not a Davis site, but I believe a UK reseller of Davis products.  But they have supplied a good amount of information, better in many ways than Davis.

They indicate that a temperature/Humidity Sensor has an average life of about 5 years, less if near the ocean.  They also say with maintenance, a Davis weather station can expect to last about 10 years with some maintenance.

Interesting that other than replacing the super capacitor once for a failure, my other failure was the console, always indoors and out of the weather.

Since my anemometer seems to be performing better with some lubrication, I decided to order a new temperature/humidity sensor.  I'm sure I will also need a ISS Cover/Solar cell in the near future.  Hopefully no other failures for a few years.

Complete Disassembly of ISS

 After another night of Temp/humidity not reporting, I completely disassembled the ISS and looked it over.  I did not see anything that looked wrong, but I cleaned it up.  I didn't see anything with the temp/humidity sensor obvious, but I also cleaned the cover and filter screen.

While I was at it, I lubricated the anemometer and wind vane.  The wind vain was definitely not turning easily.  The anemometer wasn't bad.  Though it certainly seemed like under 5 mph it was not spinning the last couple of years.

Since the rain gauge and wind speed/direction seem to be working, things are pointing to the temp/humidity sensor.  It is as if it works, but when the voltage drops below some level where it should work, it doesn't work.

I am still struggling with whether to spend $60 on a new sensor.  I would if I knew the station would work for several more years, but the anemometer is marginal, and the solar cell / ISS cover looks like it has been out in the weather for 15 years.

The Ambient WS-2902 I have been looking at requires some magic to work with Weewx.  Not sure I want to attempt a major new project.

As much as prices for the Davis weather stations has increased the last 15 years, and the trouble I have had with my Vantage Pro 2, buying a Vantage Vue ISS would be plug and play with my current setup.

Well That Didn't Help

 A couple of days after replacing the "Super Capacitor", the temperature and humidity readings are still dropping out intermittently.  Mostly at night, but this morning they stopped for an hour or two after sunrise, though I think it was mostly overcast.

So I wondered if the ISS battery was really OK.  Weewx reports "signal quality" of the ISS and actually graphs it, and I wasn't seeing any issues.  It also reports a "Battery Status" in real time (at the 2.5 sec update rate, though web page only refreshed every 5 minutes.  It normally reports just as "OK".  I assume if there is an issue, it would report "Not OK" or something similar.

Then today I spent some time trying to figure how to know if the Battery Status ever was not OK, like in the middle of the night.  I didn't think so, but wanted to know. I search through some logs, but didn't see any battery issues.  Then I did some searching in the Weewx Google Groups pages, and eventually saw that one could make Weewx create an alarm send an email if battery was ever low, or not OK.

That was not quite so easy.  It took me several hours to get it working, but I believe it will now send me an email if the ISS battery level drops below OK.

One other bit of information that really should answer the question, is that it appears wind speed and direction is still being reported.  A least it was yesterday.  Today shows no wind speed, but maybe it is just calm, or very low.  I don't think the anemometer works great below 5 mph.

Then tonight, just after dark temp and humidity have stopped reporting.  It is looking like maybe it is just a temp/humidity sensor issue.  Looking on line, I see some people say they have to replace them every 3 or 4 years.  I'm at 15, so maybe I'm lucky?

The Vantage Pro 2 is supposed to be one of the best personal weather stations.  I have had quite a few issues with mine, and it looks like I will need to spend some more to keep it running.  The Temp/Humidity sensor is $60.  Parts to replace anemometer bearings are not available for my vintage unit, so a whole new one is $184 as of today.  A complete weather station from Ambient that has similar features is less than $200.  The vantage Pro 2 updates readings more often, and it has a better wireless transmitter.  But it is not a bulletproof station.  It needs regular upkeep and parts will fail.  Another option is to buy a slightly lower cost Vantage Vue ISS only.  My console will work with it.  It's around $300.

A different station will mean doing some updating to make it work with Weewx.  Maybe I could sell the Vantage Pro 2 console and logger/pc interface.

Maybe I'll just buy the sensor and hope everything else lasts a few more years.

Integrated Sensor Module Issues

The last few nights the outdoor sensors stopped responding in the middle of the night.  They would begin sending data again a while after sunrise.  I had this problem before, maybe 2012-2014, not sure of date, where I replaced the "super capacitor", which helps maintain power during the night.

The battery reads over 3 V, so battery is not likely the problem.  Fortunately I had a spare capacitor from the last time I ordered one, and installed it today.  Hopefully that is the fix, otherwise maybe the solar cell is bad.

I don't think I want to put too much money into another Davis unit.  Price has soared and I haven't exactly had good reliability with it.  The anemometer may need new bearings as well.

I might opt for one of the Ambient systems at this point.  Maybe I could sell the Davis console and datalogger.

Various Issues Popping Up Again

 For some reason the Raspberry-Pi stopped working last night.  It had power and was connected up, but I had nothing on the monitor, and could not connect remotely.  I had no choice but to hard power-down.  It came back up fine, and weewx started running, but after one update, it was no longer updating the wx data.  After trying many things, I had to to a memory dump, then clear memory on the Vantage logger.  Luckily these tools are built into weewx along with a good troubleshooting guide, otherwise I'd be still floundering.

One issue I have had with this Pi, is the HDMI connector.  I don't know if it is the cable, or the connector, but the connection is intermittent.  This seems to only affect the monitor, so I don't think it is related.  But if any of the other USB connectors were intermittent, it could be.

This is similar to the problems I was having a couple of years ago with my previous Raspberry-Pi.  I wasn't sure if the Raspberry-Pi (memory card) was part of the problem, but I was due for a refresh of everything anyway.  It did give me two years of no problems.  I'm hoping this isn't a recurring trend.

711 Days without Interuption

Well the streak had to end sometime.  It was not a failure of the hardware or software, just a failure on my part .  I accidently killed power to the Raspberry Pi,  But it gave me the opportunity to investigate possibly adding an extension in Weewx.  Ultimately it didn't work out, but I've wanted to try it for a long time but didn't want to shut it down.

But it back up and running, but I'll probably be quicker to do updates or make changes without trying to break any uptime records.

Weewx and Website Running Over 1 Year w/o Interruption

 

In the past, while running Wview or Weewx, I have had times where the station has run for a few months before something would happen, like a power blip or outage.  I've had the Davis console fail, batteries and capacitors in the integrated sensor unit fail, USB hubs, fail, serial port glitches, memory problems on the Raspberry-Pi, etc.  But after a new Raspberry-Pi in Dec 2020, and a UPS last February 2021, the Raspberry-Pi and Weewx have been running over a year without a restart.  I have had connection to the internet fail a few times, but the system keeps on recording data and updates when it is restored.

The UPS has been the biggest improvement.  I'd guess we have had 3 or 4 power outages in the last year.  Most were short, one was a couple of hours.  But it is nice to not have to connect up to the Pi and start everything up again for every power glitch.

I have a couple of things I'd like to do on the Pi, but almost don't want to shut it down and break this streak.

Inside Humidity

One of my hobbies is playing guitar.  I have several, both electric and acoustic.  Humidity plays havoc with the wood, especially the neck of a guitar, affecting the tuning and playability.  The seasonal changes in humidity is when it is most noticeable.  We have air conditioning, and I have reduced the number of times that we keep windows open in the summer, as I felt going back and forth was not great for the guitars.  Last winter, after some very cold dry weather when the indoor humidity was in the low 30% range, I finally had a humidifier added to my hot-air furnace.

The humidifier has a rough adjustment, but I have been depending on my Davis Vantage Pro II console for the indoor humidity.  Weewx, by default, does not include indoor humidity in it's standard report (Seasons).  It does include indoor temperature.  I guess it is because not all weather stations include an indoor humidity measurement.  But I wanted the ability to track the indoor humidity so I can adjust the humidifier to ensure it is keeping the house in the proper range.

So today I delved into trying to figure out how to include Indoor Humidity in the Weewx generated web pages.  It wasn't particularly difficult if I don't include the couple of hours to figure out how to do it.  The actual changes took maybe 15 minutes.  I was surprised that there wasn't specific instructions on how to do this.  I searched the User Guide, Customization guide, and User Forums.  Eventually I narrowed in on a solution, but it wasn't easy.  Essentially it required changes to the skin.conf and index.html.tmpl in order to display the Indoor Humidity.  At least those changes didn't require a Weewx restart, and I am still going on 335+ days without a restart.

One of my issues is that I'm not a Linux expert or super user.  My Raspberry Pi has been running for 335 days, and that is how long it has been since I actually typed any commands in Linux.  So I have to re-learn everything when it has been that long.

It would be nice if Weewx had options in a config file for what to display by default.  There are specific options for Vantage stations, why not make a switch?  Or at least semi-detailed instructions on how to add the Indoor Humidity.

Anyway, it is working now, and since the data has been recorded all along, I can see the history of the indoor humidity for a year, and see the difference it made when my humidifier was installed.

Station Up for 200 Days

Just noticed on my website that the wx station has been running 200 days straight without a hitch.  I haven't kept track, but pretty sure this is a record for me.  The UPS has helped, though I'm not sure we've lost power, but the new Raspberry-Pi and SW probably has helped as well.  I also noticed the battery in the sensor unit was low, so I replaced that before it caused an issue.  Anyway, happy to have this working well.

USB Hub

 Very odd.  The problem with the webcam seems related to the USB powered hub I was using for the webcam.  The webcam was the only thing connected to the hub.  I have been using it for years, and why it decided to act up is unknown.  I disconnected the webcam from the powered hub and now going straight to the Rasberry Pi.  We'll see if it keeps working, but so far the webcam is back online.

Webcam Issues

 I just noticed earlier today that my webcam image has not been updating since January 27.  Nothing has been touched, everything has been running fine.  So when I went to look at the Raspberry-Pi UI, the monitor will not read the signal from the Pi.  So I set up another PC to attach remotely.  Restarting the webcam software did not help.  Then I just took a photo with the webcam, and it seems to be working, so luckily it is not the webcam itself.  So I looked at the syslog for the Pi and I am getting a bunch of USB related messages that look like errors - disconnecting/reconnecting, etc.  It's late, I'll have to look at it tomorrow.

Finally a UPS

 I finally got around to purchasing an Uninterruptable Power Supply for the Davis console and the Raspberry Pi.  If you know Linux, it isn't good to just drop power from the PC.  Probably more so when the operating system is on a SD card.  I'm not really running much else with it, so I should have a bit of run time, though the internet would probably be down, but the Davis should collect wx data for a bit.

It will definitle help with the short 2-10 second power outages that happen a few times a year. 

New Raspberry-Pi and Web Page Skin

 Early in 2020, I had a rash of issues.  In troubleshooting and reading up on the Weewx website and forums, I was either having issues with the Davis Datalogger or my SD card in the Pi.  I had been contemplating getting a newer Pi anyway, so I figured I'd go ahead and get one and see if it helps.  This was maybe April 2020.

Wouldn't you know, after getting it, the problems mostly went away.  So here it was December 2020, and this new Raspberry-Pi is still sitting there.  I had dreaded making the switch, because I knew it would be a few days of work, getting all backed up and transferred over, plus I had this other skin and numerous customizations.  But I went ahead and started it last Sunday.

I initially got it going after about 10 hours, and realized there was a new default skin in Weewx named "Seasons".  It was much improved over the old standard one.  The Bootstrap one I was using before has some features this one doesn't, but I thought it was time to simplify things by using the default skin.  I'm still doing a few customizations, but not too many.

Here it is Wednesday, and I'm still tweaking here and there, but it is running and looks good.  I finally ordered a UPS, so that will hopefully handle the occasional power glitches that can corrupt the file system and SD card.

I think I may need to revisit the datalogger serial to USB converter issue.  I may have messed up the udev rule or I may need to add ferrites to the USB cable, as my power arrangement for the Pi has changed.

I try to keep this up-to-date as time permits and the situation changes.

Vantage Pro2 Console Failure

 Things had been running well for 20 months or so. Then on around Oct. 11, 2016, the weather station stopped updating. The Vantage Pro2 console was frozen at whatever time it failed. Repeated attempts to reset, power-off/on failed. Eventually the display went completely blank. A call to Davis confirmed it needed to be repaired. $82 and a couple weeks later I had the console up and running, but the weather wasn't updating. Searching the Weewx forum, it appeared my datalogger likely had some corupted data, and probably not surprising since the console went thru many power resets. Weewx has a utilty to clear data from the datalogger, which solved the problem.

More Missing History

 Making some progress in possibly restoring the bulk of my station history. I have been uploading data to Weather Underground since the beginning (2008) and found there is a couple ways to get data back from them. I have downloaded my history in a series of CSV files, merged them together, and manipulated the data to match up with the Wview/Weewx archive database. The biggest issue was converting date time back from YYYY-MM-DD HH:MM:SS to Unix Epoch time. I've made some progress and imported the CSV to SQLite, so things may be looking up. Still a bit of work to do.

Missing History

 Well I guess I really screwed up. Looks like I lost my weather history from 2010-2014. I had thought that I copied to wview archive file from my Ubuntu PC, but apparently the one I copied was an older one from 2009. It was in the location described as the place where Wview archives were stored, but I guess the location moved depending on what version of Wview was installed. I had looked at the data briefly, and saw data from 2009, so I thought I was good. But, still it all could have been avoided if I didn't decide the give Linux Mint a try on the old PC. I should have waited until I knew for sure I didn't need any files from that machine. I wonder if there is a way to extract data back from Wunderground. At least daily min/max values.