publish new post

content/posts/2023/59-bye-netatmo-welcome-ecowitt/index.md

@@ -0,0 +1,318 @@
+---
+title: 'Bye Netatmo – welcome Ecowitt'
+summary: >
+  I'm totally done with Netatmo at this point and I am more than happy to
+  show you my new setup at my home. I'll start with a
+  [WS69](https://shop.ecowitt.com/products/ws69?variant=41418884743330)
+  (all-in-one) sensor and the Gateway [GW1100](https://shop.ecowitt.com/products/gw1100?variant=41418887987362).
+  Combined with WeeWX we will publish our data to some online weather networks/services.
+date: 2023-12-26T14:06:19+0100
+lastmod: 2023-12-26T13:27:34+0000
+categories:
+- amateur-radio
+- computerstuff
+tags:
+- aprs
+- netatmo
+- raspberry-pi
+- selfhost
+- weewx
+- wx
+
+---
+
+Finally. I've seen this coming for a long time but I didn't want to remove the
+"somewhat pricey" Netatmo stuff[^1] just like it costs nothing.
+
+[^1]: The sensor are not very expensive, but they are not cheap either.
+They use a lot of batteries (usually type AA or AAA) and they perform quite good, but
+all data is stored on Netatmo servers and your weather station has to use their API
+to download actual values.
+
+"Luckily" the weather station wasn't able to get actual data from their servers and a
+complete re-installation (with fresh api keys) didn't resolve the problem. Also some
+manual requests from my computers (yep, I used two with different operating systems)
+weren't able to obtain any data through their API.
+
+That was finally the point where I decided to remove this ~~shit~~ stuff. And some
+outdoor cameras will follow as well, when I have found new and affordable hardware.
+I'm somewhat looking towards [Axis](https://www.axis.com/products/network-cameras).
+
+The following article describes my new weather station setup at home but I wrote this
+down nearly two weeks after the installation and I might forgot something -- read this
+with a grain of salt.
+
+## Installation of the gateway
+
+This is super easy, there is a temperature and humidity sensor with a ~70 cm long cable
+(and a ~1 m USB cable, which I don't use). I plugged the gateway into my Raspberry Pi 4
+directly and finished the setup on my smartphone. I won't cover this here, the process
+is really super easy and straightforward.
+
+I will continue with the main software, the logic of most personal weather stations:
+[WeeWX](https://www.weewx.com/).
+
+## Installation of WeeWX on the Raspberry Pi 4
+
+As a little sidenote, I am using the **testing branch** of Debian, which is
+_Debian GNU/Linux trixie/sid_ at the moment.
+
+![/etc/motd file showing WX RASPI4](wx-raspi.png "I do like `/etc/motd` files showing to what I connect")
+
+### First off, a database is needed
+
+I use a MySQL database for the weather data which should be installed before
+we connect WeeWX to a database.
+
+~~~console
+$ sudo apt install mariadb-server
+$ sudo mysql_secure_installation
+$ mysql -uroot -p
+~~~
+
+Login with the new password and create the necessary users and databases. I will name them
+both **weewx**.
+
+You can also use graphical management software to manage your mysql databases, but you would
+need to secure your installation yourself.
+
+~~~console
+$ sudo apt-get install phpmyadmin
+~~~
+
+That would also pull in a whole lot of packages like PHP.
+
+{{< alert >}}
+Don't allow access from the internet to your management software!
+{{< /alert >}}
+
+As I had a backup of my old installation I will import the old data into my fresh database
+setup:
+
+~~~console
+$ mysql -uweewx -p'{fancy-password}' weewx < weewx-backup.sql
+~~~
+
+The backup was done like:
+
+~~~console
+$ mysqldump -u weewx -p weewx > weewx-backup.sql
+~~~
+
+I had data from 2022-01-01 to 2023-12-10, which were about 120MB.
+
+### Some information about security
+
+{{< alert >}}
+If your setup is focused on security, my approach is probably not suited for you!
+{{< /alert >}}
+
+I will install the packaged version of WeeWX -- which is version 4.10.2 at this moment
+but the main concern is the user which runs the installed application. The installed
+application will be run by the **root** user.
+
+Since this is a dedicated Raspberry Pi that will never run something else (it also has
+its own MariaDB server running)... _I will live with it_.
+
+In contrast, my last WeeWX installation was on my webserver (it only had to pull the
+weather data via an API from a cloud network) and was setup with a dedicated user for
+that particular task (running the daemon and publishing the website directly on the
+server).
+
+The new setup will also publish directly to my webserver and again, a dedicated user is
+used for this on the webserver. Authentication is done via ssh and a dedicated
+ssh key on the Raspberry Pi.
+
+Back to topic, I was about to install WeeWX on the Raspi. Follow the
+[instructions](https://weewx.com/docs/debian.htm) on their website and all is fine :wink:
+
+Nonetheless, I leave the code snippets over here:
+
+~~~console
+$ wget -qO - https://weewx.com/keys.html | sudo gpg --dearmor --output /etc/apt/trusted.gpg.d/weewx.gpg
+$ wget -qO - https://weewx.com/apt/weewx-python3.list | sudo tee /etc/apt/sources.list.d/weewx.list
+$ sudo apt-get update && sudo apt-get upgrade && sudo apt-get dist-upgrade
+$ sudo apt-get install weewx
+~~~
+
+The third line is quite extended because I had to update several packages on my Raspberry Pi first.
+
+They suggest to look at the WeeWX logs via `sudo tail -f /var/log/syslog` -- that is
+actually not the best idea on newer linux distributions. Newer distributions use
+[systemd](https://systemd.io/) and it comes with its own userspace tools to obtain the
+journal (the logs).
+
+I use either `journalctl -fe` or specific to WeeWX `journalctl -feu weewx`.
+
+They also use the init files for starting and stopping; if you want to use systemd for that, you
+would typically start and stop WeeWX like this:
+
+~~~console
+$ sudo systemctl start weewx(.service)
+$ sudo systemctl stop weewx(.service)
+~~~
+
+As you can see, you can append `.service` to the service name or just not use it. This is not needed though,
+as there is no unit file as per se, but systemd knows the actual init file and internally generates
+a unit file.
+
+There is actually a unit file at [Github](https://github.com/weewx/weewx/wiki/systemd).
+
+As my old installation was on my webserver I had to adopt another script to work properly.
+You can find that script on my older article about
+[moving WeeWX to OpenBSD]({{< ref "41-moving-weewx-to-openbsd" >}}).
+
+But we drift off again...
+
+### Install weewx-gw1000 driver
+
+~~~console
+$ wget -P /var/tmp https://github.com/gjr80/weewx-gw1000/releases/download/v0.6.0b2/gw1000-0.6.0b2.tar.gz
+$ sudo wee_extension --install=/var/tmp/gw1000-0.6.0b2.tar.gz
+~~~
+
+Taken from the [README](https://github.com/gjr80/weewx-gw1000) file, go and have a look at it,
+as you might need to change the config.
+
+I tested the driver with
+
+~~~console
+$ PYTHONPATH=/usr/share/weewx python -m user.gw1000 --test-driver
+~~~
+
+and finally updated the WeeWX configuration
+
+~~~console
+$ sudo wee_config --reconfigure --driver=user.gw1000
+~~~
+
+{{< alert "circle-info" >}}
+You should check the WeeWX configuration file if everything got updated correctly
+and you may probably fill in an IP address.
+{{< /alert >}}
+
+I tried to use it as a weewx-service but ended with the normal driver setup.
+
+A final restart of WeeWX whenever you change your config file.
+
+~~~console
+$ sudo systemctl restart weewx
+~~~
+
+### Install the Weather Data Center Skin
+
+~~~console
+$ wget -O "/tmp/weewx-wdc.zip" https://github.com/Daveiano/weewx-wdc/releases/download/v3.4.0/weewx-wdc-v3.4.0.zip
+$ mkdir /tmp/weewx-wdc
+$ unzip /tmp/weewx-wdc.zip -d /tmp/weewx-wdc/
+$ sudo wee_extension --install /tmp/weewx-wdc/
+~~~
+
+### Install the weewx-xaggs extension
+
+Install and configure as shown on the
+[wiki page](https://github.com/Daveiano/weewx-wdc/wiki/Support-for-weewx-xaggs).
+
+### Install the weewx-forecast extension
+
+There is also a [wiki page](https://github.com/Daveiano/weewx-wdc/wiki/Support-for-weewx-forecast)
+for this. Read carefully and my observation is: use Openweathermap.
+
+I had shitloads of problems trying to use the forecast of WUnderground and I finally
+ended up using Openweathermap (which worked out of the box).
+
+#### WUnderground
+
+I tried to install this initially because it was the cheapest solutions. You get free
+forecast information if you provide your weather data to their network. I'm not sure if this
+is the official way because I read this somewhere.
+
+#### Openweathermap
+
+This is also a good solution because it works instantly and it is not as expensive as the other
+options, the free available API calls should be enough for a personal weather station.
+
+The default options in the `skin.conf` file work fine for me, don't forget to comment them out.
+
+## Uploading data to other weather websites/services
+
+### Ecowitt
+
+Ecowitt publishes the data on their network: <https://www.ecowitt.net/home/index?id=163378>
+
+You need to be logged in to view other stations though.
+
+### WUnderground
+
+Because I already registered an account for the forecasts I decided to keep the account and
+push my weather data to their network.
+
+I thought I could upload my historical data with `wunderfixer` but it did not got imported.
+
+~~~console
+$ for month in {1..12}; do for i in {1..31}; do sudo wunderfixer -d 2022-$month-$i; done; done
+$ for month in {1..12}; do for i in {1..31}; do sudo wunderfixer -d 2023-$month-$i; done; done
+~~~
+
+This is very theoretical because I wasn't able to import all my past data. I won't look into this
+any further because I really don't care and I personally don't like the WUnderground website that much.
+
+My station can be found here: <https://www.wunderground.com/dashboard/pws/ILNGEN12/>
+
+### AWEKAS
+
+AWEKAS is an Austrian weather map system (Automatisches Wetterkarten System). I found this site
+actually while looking for information about the different forecast models mentioned in the
+forecast plugin above (well, not direclty...).
+
+Anyway, they provide a paid website for your weather data, so if you don't want to host your own
+website this might be interesting to you. The price is currently at 3.83 € (excl. VAT) per month
+(that would be ~50 € a year) -- forecasts are pulled from
+[World Weather Online](https://www.worldweatheronline.com/) and include a 3-day preview (only).
+
+What I liked on AWEKAS is, that they blocked my air pressure measurements (only those) **within a few
+hours** as they were off a bit (~9 hPa higher). I then calibrated the sensor and made a quick check,
+then I unblocked them on my stations control panel. I've seen they are still a bit high and low sometimes,
+but now I know I should keep an eye on them for a while.
+
+The station can be found here: <https://www.awekas.at/en/instrument.php?id=32962>
+
+And if it is still online, the webstation can be found here:
+<https://stationsweb.awekas.at/index.php?id=32962>
+
+I've included a preview here because I registered recently and they gave me one month of a free
+webstation (that is how they call the website that publishes the data).
+
+![awekas01](awekas01.png "The look at the index page showing current values as well as min/max values")
+![awekas02](awekas02.png "Comes in very handy: a stylish forecast")
+![awekas03](awekas03.png "Sun and Moon rise and set times, the visualisation speaks for itself")
+![awekas04](awekas04.png "Plots all values into one diagram -- very nice")
+![awekas05](awekas05.png "The [interactive map](https://www.awekas.at/en/map.php) from the website")
+
+### CWOP/MADIS
+
+Let's write these short words out one time:
+
+CWOP
+: Citizen Weather Observer Program
+: <https://en.wikipedia.org/w/index.php?title=CWOP>
+
+MADIS
+: Meteorological Assimilation Data Ingest System
+: <https://en.wikipedia.org/w/index.php?title=MADIS>
+
+Of course I upload my data to the CWOP as my station is a registered CWOP station. CWOP sends their
+data to MADIS, who then rate my station in relation to my neighboring stations.
+
+You can have a look at my station [AV859](https://weather.gladstonefamily.net/site/AV859) if you like --
+hough, it seems the information there did not get updated yet as it still shows Netatmo as my hardware and
+also other people reported problems with the CWOP (and sometimes FindU) sites.
+
+Recent packets can be seen here: <http://www.wxqa.com/sss/search1.cgi?keyword=oe7drt-13>
+
+## One last view at the old page
+
+Unfortunately I only have the statistics page captured.
+
+![old stats page (very, very long)](old-netatmo-stats.png "The screenshot has been taken on my phone -- hence the small paragraphs...")
+