pywws¶
Python software for USB Wireless Weather Stations.
pywws is a collection of Python modules to read, store and process data from popular USB wireless weather stations such as Elecsa AstroTouch 6975, Watson W-8681, WH-1080PC, WH1080, WH1081, WH3080 etc. I assume any model that is supplied with the EasyWeather Windows software is compatible, but cannot guarantee this.
The software has been developed to run in a low power, low memory environment such as a Raspberry Pi. It can be used to create graphs and web pages showing recent weather readings, typically updated every hour. It can also send “live” data to services such as Weather Underground and post messages to Twitter.
- The development version of pywws is hosted on GitHub.
- “Snapshot” releases of pywws are available from the Python Package Index (PyPI).
- Documentation is hosted on Read the Docs.
- Documentation is available in the following languages (the non-English versions are not complete or up to date):
I have written this software to meet my needs, but have tried to make it adaptable to other people’s requirements. You may want to edit some or all of the modules, or write some new ones, to get it to do exactly what you want. One of the reasons for using Python is that it makes such alterations so easy. Don’t be afraid, just jump in and have a go.
Requirements¶
The software needed to run pywws depends on what you plan to do with it. You’ll need some of the following.
- Essential: Python 2.7 or 3 (also see legacy version below).
- Essential: USB library python-libusb1 or PyUSB or, for MacOS, hidapi and a Python interface to it.
- Graph drawing: gnuplot.
- Secure uploading to your web site: Paramiko.
- Posting to Twitter: python-twitter or Tweepy.
- Posting to other web services: python-requests.
For more detail, see the documentation - dependencies.
Legacy version¶
If for some reason you are stuck with Python 2.5 or 2.6 a “legacy branch” of pywws can be installed with pip and is available on GitHub. The most recent version of this branch is 18.4.1.
Documentation for the legacy version is also available on Read the Docs: http://pywws.readthedocs.io/en/legacy/
Installing and upgrading pywws¶
pywws can be installed directly from the Python Package Index (PyPI) using the pip command. See How to get started with pywws for full instructions.
Documentation¶
Documentation is included with pywws downloads, and is also available online. A good starting place is How to get started with pywws which describes in more detail how to install pywws.
If you have questions not answered in the documentation, please join the pywws Google mailing list / discussion group and ask there. Note that your first message to the group will not appear immediately – new posters have to be approved by a moderator, to prevent spam messages.
Contents¶
Dependencies¶
The list of other software that pywws depends on looks frighteningly long at first glance. However, many of these packages won’t be needed by most users. What you need depends on what you want to do with pywws. Remember, it’s a “kit of parts” rather than a monolithic application.
Some of the requirements are Python packages that can be downloaded from the Python Package Index (PyPI). I recommend using pip to install these.
You should be able to install the remaining dependencies using your operating system’s package manager. This is a lot easier than downloading and compiling source files from the project websites. Note that some Linux distributions use different names for some of the packages, e.g. in Ubuntu, pyusb is called python-usb.
Note: some of these libraries may have their own dependencies that you may need to install. Follow the links to read more about each library’s requirements.
Essential¶
- Python version 2.7 or higher, preferably Python 3
If you are stuck on Python 2.5 or 2.6 for some reason you can use the older “legacy” branch of pywws. This will not be further developed so you’ll be missing out on any improvements made to the main branch.
You will probably be able to install pip with your system’s package manager, where it may be called python-pip or python3-pip or something similar.
If not, download and run the get-pip.py file from the pip web site.
In either case you should immediately use pip to install the latest version of itself:
sudo pip install --upgrade pip
Make sure you install the correct Python version’s pip. If you want to install pywws for both Python 2 and Python 3 you will need pip2 and pip3.
This is a handy little module that provides information on your local time zone.
It’s best installed with pip:
sudo pip install tzlocal
USB library¶
To retrieve data from a weather station pywws needs a python library that allows it to communicate via USB. There is a variety of USB libraries that can be used. Not all of them are available on all computing platforms, which may restrict your choice.
On MacOS X the operating system’s generic hid driver “claims” the weather station, which makes it very difficult to use any other USB interface. Unfortunately, you will need to download and compile hidapi yourself.
If you can’t install ctypes then you can try the Cython interface to hidapi instead:
- cython-hidapi
- cython (your package manager may know it as python-Cython)
Other systems use a Python interface to the libusb system library. There is a choice of interface and library version - install the latest that is available for your computer.
- libusb version 1.x (should be available from the package manager)
- python-libusb1 version 1.3+
pip install libusb1
or
pip install pyusb
If neither of these options works for you then you may be able to use hidapi – see the Mac OS X instructions above.
Changed in version 15.01.0.dev1265: added ability to use python-libusb1 interface.
Flexible timed tasks¶
The pywws.regulartasks module can do tasks at particular times and/or dates.
This requires the croniter library.
(Simple hourly, daily or ‘live’ tasks don’t need this library.)
pip install croniter
Running as a daemon¶
The pywws.livelogdaemon module runs pywws live logging as a proper UNIX daemon process.
It requires the python-daemon library:
pip install python-daemon
Graph drawing¶
The pywws.plot module uses gnuplot to draw graphs.
If you want to produce graphs of weather data, e.g. to include in a web page, you need to install the gnuplot application:
- gnuplot v4.2 or higher (should be available from the package manager)
After installing gnuplot you should edit weather.ini (see weather.ini - configuration file format) and set the gnuplot version config item.
Finding out the installed gnuplot version is easy:
gnuplot -V
Weather “service” uploading¶
Uploading to HTTP based “services” such as Weather Underground requires the Python requests library:
pip install requests
Secure website uploading (sftp)¶
The pywws.towebsite module can use “ftp over ssh” (sftp) to upload files to your web-site.
Normal uploading just uses Python’s standard modules, but if you want to use sftp you need to install these two modules:
sudo pip install pycrypto paramiko
Twitter updates¶
The pywws.service.twitter module can be used to send weather status messages to Twitter.
Posting to Twitter requires these modules:
- python-twitter v3.0 or higher
- python-oauth2
sudo pip install python-twitter oauth2
or
- tweepy v2.0 or higher
- python-oauth2
sudo pip install tweepy oauth2
Note that tweepy appears to be the less reliable of the two.
If you have problems, e.g. with character encoding, try installing python-twitter instead.
MQTT¶
New in version 14.12.0.dev1260.
The pywws.service.mqtt module can be used to send weather data to an MQTT broker.
This requires the paho-mqtt module:
sudo pip install paho-mqtt
To create new language translations¶
pywws can be configured to use languages other than English, as described in How to use pywws in another language. The Babel package is required to extract the strings to be translated and to compile the translation files.
sudo pip install babel
Copying files to or from Transifex requires the transifex-client package.
sudo pip install transifex
Translating the documentation using local files needs the sphinx-intl package.
sudo pip install sphinx-intl
Changed in version 14.05.dev1209: pywws previously used the gettext package.
To ‘compile’ the documentation¶
The documentation of pywws is written in “ReStructured text”. A program called Sphinx is used to convert this easy to write format into HTML for use with a web browser. If you’d like to create a local copy of the documentation (so you don’t have to rely on the online version, or to test a translation you’re working on) you need to install Sphinx, version 1.3 or later.
sudo pip install sphinx
Comments or questions? Please subscribe to the pywws mailing list http://groups.google.com/group/pywws and let us know.
User guides¶
Contents:
How to get started with pywws¶
Installation¶
First of all you need to install Python and a USB library (to allow Python to access the weather station). See Dependencies for more detail.
Create a directory for all your weather related files and change to it. For example (on a Linux or similar operating system):
mkdir ~/weather
cd ~/weather
The easiest way to install pywws is with the pip command:
sudo pip install pywws
Upgrading pywws is also a one line command:
sudo pip install -U pywws
Now you are ready to Test the weather station connection.
If you prefer not to use pip, or you want easy access to the pywws source files (e.g. to translate the documentation – see How to use pywws in another language), you can download and extract the files into your weather directory.
Visit https://pypi.org/project/pywws/ and download one of the .tar.gz files. Put it in your weather directory, then extract all the files, for example:
cd ~/weather
tar xf pywws-18.4.1.tar.gz
This should create a directory (called pywws-18.4.1 in this example) containing all the pywws source files.
It is convenient to create a soft link to this awkwardly named directory:
cd ~/weather
ln -s pywws-18.4.1 pywws
Upgrading a downloaded snapshot is the same process as the first installation. Download the .tar.gz file, extract its contents, then delete the soft link pointing to the old download and create one pointing to the new download. Once you are satisfied the new version is working OK you can delete the old download entirely.
The PyPI files contain a snapshot release of the software - a new one is issued every few months. If you want to use the very latest version of pywws, e.g. to work on fixing a bug, you can get all the files you need from the GitHub repository. Install git and use it to clone the repos:
cd ~/weather
git clone https://github.com/jim-easterbrook/pywws.git
To upgrade you use git to pull any changes:
cd ~/weather/pywws
git pull
If you have downloaded or cloned the pywws source files, you need to use setup.py to install it:
cd ~/weather/pywws
python setup.py compile_catalog
python setup.py build
sudo python setup.py install
The python setup.py compile_catalog step is only needed if you want to use pywws in a language other than English.
See Test the pywws translations for more detail.
Compile documentation (optional)¶
If you’d like to have a local copy of the pywws documentation (and have downloaded the source or cloned the repo) you can “compile” the English documentation. This requires the sphinx package:
cd ~/weather/pywws
python -B setup.py build_sphinx
To compile the documentation in another language you need to set the LANG environment variable.
For example, to compile the French documentation:
cd ~/weather/pywws
LANG=fr python -B setup.py build_sphinx
The compiled documentation should then be found at ~/weather/pywws/doc/html/index.html.
See How to use pywws in another language for more detail.
Test the weather station connection¶
Now you’re ready to test your pywws installation.
Connect the weather station (if not already connected) then run the pywws.testweatherstation module:
pywws-testweatherstation
If everything is working correctly, this should dump a load of numbers to the screen, for example:
0000 55 aa ff ff ff ff ff ff ff ff ff ff ff ff ff ff 05 20 01 51 11 00 00 00 81 00 00 0f 00 00 60 55
0020 ea 27 a0 27 00 00 00 00 00 00 00 10 10 12 13 45 41 23 c8 00 32 80 47 2d 2c 01 2c 81 5e 01 1e 80
0040 96 00 c8 80 a0 28 80 25 a0 28 80 25 03 36 00 05 6b 00 00 0a 00 f4 01 18 03 00 00 00 00 00 00 00
0060 00 00 4e 1c 63 0d 2f 01 73 00 7a 01 47 80 7a 01 47 80 e4 00 00 00 71 28 7f 25 bb 28 bd 25 eb 00
0080 0c 02 84 00 0e 01 e3 01 ab 03 dc 17 00 10 08 21 08 54 10 03 07 22 18 10 08 11 08 30 10 04 21 16
00a0 26 08 07 24 17 17 08 11 01 06 10 09 06 30 14 29 09 01 06 07 46 09 06 30 14 29 09 01 06 07 46 08
00c0 08 31 14 30 10 05 14 15 27 10 01 26 20 47 09 01 23 05 13 10 01 26 20 47 09 01 23 05 13 10 02 22
00e0 11 06 10 02 22 11 06 08 07 07 19 32 08 12 13 22 32 08 09 07 08 48 01 12 05 04 43 10 02 22 14 43
There are several reasons why this might not work. Most likely is a ‘permissions’ problem. This can be tested by running the command as root:
sudo pywws-testweatherstation
If this works then you may be able to allow your normal user account to access the weather station by setting up a ‘udev’ rule.
The exact method may depend on your Linux version, but this is typically done by creating a file /etc/udev/rules.d/39-weather-station.rules containing the following:
ACTION!="add|change", GOTO="weatherstation_end"
SUBSYSTEM=="usb", ATTRS{idVendor}=="1941", ATTRS{idProduct}=="8021", GROUP="weatherstation"
LABEL="weatherstation_end"
Unplug and replug the station’s USB connection to force udev to apply the new rule.
This allows any user in the group weatherstation to access the weather station.
You need to create this group and add your normal user account to it – many Linux systems have a GUI for user and group management.
If you have any other problem, please ask for help on the pywws mailing list: http://groups.google.com/group/pywws
The pywws-testweatherstation command has options to decode the numbers shown above, or to show you the station’s logged data.
The --help option prints a usage message:
jim@firefly ~/weather/pywws $ pywws-testweatherstation --help
Test connection to weather station.
usage: /usr/bin/pywws-testweatherstation [options]
options are:
--help display this help
-c | --change display any changes in "fixed block" data
-d | --decode display meaningful values instead of raw data
-h n | --history n display the last "n" readings
-l | --live display 'live' data
-m | --logged display 'logged' data
-u | --unknown display unknown fixed block values
-v | --verbose increase amount of reassuring messages
(repeat for even more messages e.g. -vvv)
jim@firefly ~/weather/pywws $
Using the --decode option shows the decoded “fixed block” data:
jim@firefly ~/weather/pywws $ pywws-testweatherstation -d
10:02:16:pywws.logger:pywws version 18.4.2, build 1523 (092cf26)
{'abs_pressure': 1001.6,
'alarm': {'abs_pressure': {'hi': 1040, 'lo': 960},
'dewpoint': {'hi': 10, 'lo': -10},
'hum_in': {'hi': 65, 'lo': 35},
'hum_out': {'hi': 70, 'lo': 45},
'illuminance': 0,
'rain': {'day': 150, 'hour': 3},
'rel_pressure': {'hi': 1040, 'lo': 960},
'temp_in': {'hi': 20, 'lo': 0},
'temp_out': {'hi': 30, 'lo': -10},
'time': '12:00',
'uv': 0,
'wind_ave': {'bft': 0, 'ms': 11.2},
'wind_dir': 0,
'wind_gust': {'bft': 0, 'ms': 22.3},
'windchill': {'hi': 20, 'lo': 0}},
...
'read_period': 5,
'rel_pressure': 987.2,
'settings_1': {'bit3': False,
'bit4': False,
'pressure_hPa': True,
'pressure_inHg': False,
'pressure_mmHg': False,
'rain_in': False,
'temp_in_F': False,
'temp_out_F': False},
'settings_2': {'bit5': False,
'bit6': False,
'bit7': False,
'wind_bft': False,
'wind_kmph': False,
'wind_knot': False,
'wind_mph': True,
'wind_mps': False},
'timezone': -1,
'unknown_01': 0,
'unknown_18': 0}
jim@firefly ~/weather/pywws $
The --history option shows recent “logged” data, starting with the current “live” record:
jim@firefly ~/weather/pywws $ pywws-testweatherstation -d -h 2
10:06:28:pywws.logger:pywws version 18.4.2, build 1523 (092cf26)
...
Recent history
0x30a0 2018-04-27 10:06:00
{'abs_pressure': 1001.5,
'delay': 2,
'hum_in': 48,
'hum_out': 88,
'rain': 2178.6,
'status': {'rain_overflow': False, 'lost_connection': False},
'temp_in': 18.4,
'temp_out': 8.5,
'wind_ave': 0.7,
'wind_dir': 5,
'wind_gust': 1}
0x3090 2018-04-27 10:04:00
{'abs_pressure': 1001.6,
'delay': 5,
'hum_in': 47,
'hum_out': 88,
'rain': 2178.6,
'status': {'rain_overflow': False, 'lost_connection': False},
'temp_in': 18.4,
'temp_out': 8.5,
'wind_ave': 1.7,
'wind_dir': 6,
'wind_gust': 3.4}
jim@firefly ~/weather/pywws $
These options are useful if you ever need to examine the raw data as stored by the station, before pywws does any processing.
Set up your weather station¶
If you haven’t already done so, you should set your weather station to display the correct relative atmospheric pressure. (See the manual for details of how to do this.) pywws gets the offset between relative and absolute pressure from the station, so this should be set before using pywws.
You can get the correct relative pressure from your location by looking on the internet for weather reports from a nearby station, ideally an official one such as an airport. This is best done during calm weather when the pressure is almost constant over a large area.
Set the weather station logging interval¶
Your weather station probably left the factory with a 30 minute logging interval.
This enables the station to store about 11 weeks of data.
Most pywws users set up their computers to read data from the station every hour, or more often, and only need the station to store enough data to cover computer failures.
The recommended interval is 5 minutes, which still allows 2 weeks of storage.
Use pywws.setweatherstation to set the interval:
pywws-setweatherstation -r 5
Note that the weather station will not start using the new interval until the current 30 minute logging period is finished. This may cause “station is not logging data” errors when running pywws logging. If this happens you need to wait until the 30 minute logging period ends.
Log your weather station data¶
First, choose a directory to store all your weather station data. This will be written to quite frequently, so a disk drive is preferable to a flash memory stick or card, as these have a limited number of writes. In most cases your home directory is suitable, for example:
mkdir ~/weather/data
This directory is referred to elsewhere in the pywws documentation as your data directory.
Make sure your computer has the right date & time, and time zone, as these are used to label the weather station data. If you haven’t already done so, it’s worth setting up NTP to synchronise your computer to a ‘time server’.
The first time you run pywws.logdata it will create a configuration file in your data directory called ‘weather.ini’ and then stop.
You need to edit the configuration file and change the line ws type = Unknown to ws type = 1080 or ws type = 3080.
(If your weather station console displays solar illuminance you have a 3080 type, all others are 1080.)
Then run pywws.logdata again.
This may take several minutes, as it will copy all the data stored in your station’s memory.
The pywws.logdata program has a ‘verbose’ option that increases the amount of messages it displays while running.
This is useful when running it manually, for example:
python -m pywws.logdata -vvv ~/weather/data
(Replace ~/weather/data with your data directory, if it’s different.)
You should now have some data files you can look at. For example:
more ~/weather/data/raw/2012/2012-12/2012-12-16.txt
(Replace the year, month and day with ones that you have data for.)
Convert old EasyWeather data (optional)¶
If you had been running EasyWeather before deciding to use pywws, you can convert the data EasyWeather had logged to the pywws format. Find your EasyWeather.dat file and then convert it:
python -m pywws.mergeewdata EasyWeather.dat ~/weather/data
(Recent versions of EasyWeather may use a different file format which pywws.mergeewdata cannot handle.)
Set some configuration options¶
After running pywws.logdata there should be a configuration file in your data directory called ‘weather.ini’.
Open this with a text editor. You should find something like the following:
[paths]
work = /tmp/pywws
[config]
usb activity margin = 3.0
ws type = 1080
pressure offset = 9.3
logdata sync = 1
(Don’t worry about the order of items within each section.
Re-ordering them has no effect.)
You need to add a new entry in the [config] section called day end hour.
This tells pywws what convention you want to use when calculating daily summary data.
The entry should have two values separated by a comma: a number in the range 0 to 23 (the hour of day, in local winter time) and a single word True or False to say if the day end should adjust with summer (daylight savings) time.
In the UK, the ‘meteorological day’ is usually from 09:00 to 09:00 GMT (10:00 to 10:00 BST during summer), so I use a day end hour value of 9, False.
If you prefer to use midnight, winter or summer, you should use 0, True.
After editing, your weather.ini file should look something like this:
[paths]
work = /tmp/pywws
[config]
usb activity margin = 3.0
ws type = 1080
pressure offset = 9.3
logdata sync = 1
day end hour = 9, False
You can also edit the pressure offset value to adjust how pywws calculates the relative (sea level) air pressure from the absolute value that the station measures.
If you change the pressure offset or day end hour in future, you must update all your stored data by running pywws.reprocess.
For more detail on the configuration file options, see weather.ini - configuration file format.
Changed in version 13.10_r1082: made pressure offset a config item.
Previously it was always read from the weather station.
Process the raw data¶
pywws.logdata just copies the raw data from the weather station.
To do something useful with that data you probably need hourly, daily and monthly summaries.
These are created by pywws.process. For example:
python -m pywws.process ~/weather/data
You should now have some processed files to look at:
more ~/weather/data/daily/2012/2012-12-16.txt
If you ever change your day end hour or pressure offset configuration settings, you will need to reprocess all your weather data.
You can do this by running pywws.reprocess:
python -m pywws.reprocess ~/weather/data
You are now ready to set up regular or continuous logging, as described in How to set up ‘hourly’ logging with pywws or How to set up ‘live’ logging with pywws.
Read the documentation¶
You’re looking at it right now! The User guides section is probably the most useful bit to read first, but the Python programs and modules section has a lot more detail on the various pywws modules and commands.
Comments or questions? Please subscribe to the pywws mailing list http://groups.google.com/group/pywws and let us know.
How to set up ‘hourly’ logging with pywws¶
Introduction¶
There are two quite different modes of operation with pywws.
Traditionally pywws.hourly would be run at regular intervals (usually an hour) from cron.
This is suitable for fairly static websites, but more frequent updates can be useful for sites such as Weather Underground (http://www.wunderground.com/).
The newer pywws.livelog program runs continuously and can upload data every 48 seconds.
Note that although this document (and the program name) refers to ‘hourly’ logging, you can run pywws.hourly as often or as infrequently as you like, but don’t try to run it more often than double your logging interval.
For example, if your logging interval is 10 minutes, don’t run pywws.hourly more often than every 20 minutes.
Getting started¶
First of all, you need to install pywws and make sure it can get data from your weather station. See How to get started with pywws for details.
Try running pywws.hourly from the command line, with a high level of verbosity so you can see what’s happening.
Use the pywws-hourly command to run pywws.hourly:
pywws-hourly -vvv ~/weather/data
(As usual, replace ~/weather/data with your weather data directory.)
Within five minutes (assuming you have set a 5 minute logging interval) you should see a ‘live_data new ptr’ message, followed by fetching any new data from the weather station and processing it.
Changed in version 14.04.dev1194: the pywws-hourly command replaced scripts/pywws-hourly.py.
Configuring file locations¶
Open your weather.ini file with a text editor.
You should have a [paths] section similar to the following (where xxx is your user name):
[paths]
work = /tmp/pywws
templates = /home/xxx/weather/templates/
graph_templates = /home/xxx/weather/graph_templates/
modules = /home/xxx/weather/modules/
Edit these to suit your installation and preferences.
work is a temporary directory used to store intermediate files.
If your computer uses solid state storage, such as a Raspberry Pi’s SD card, it’s a good idea to make this a “RAM disk” to reduce storage “wear”.
templates is the directory where you keep your text template files, graph_templates is the directory where you keep your graph template files, and modules is a directory for any extra modules you write.
Don’t use the pywws example directories for your templates, as they will get over-written when you upgrade pywws.
Copy your text and graph templates to the appropriate directories.
You may find some of the examples provided with pywws useful to get started.
The pywws-version -v command should show you where the examples are on your computer.
New in version 14.04.dev1194: the pywws-version command.
Configuring periodic tasks¶
In weather.ini you should have [live], [logged], [hourly], [12 hourly], and [daily] sections similar to the following:
[logged]
services = []
text = []
plot = []
[hourly]
...
These specify what pywws.hourly should do when it is run.
Tasks in the [live] and [logged] sections are done when there is new logged data, tasks in the [hourly] section are done every hour, tasks in the [12 hourly] section are done twice daily and tasks in the [daily] section are done once per day.
The plot and text entries are lists of template files for plots and text files to be processed.
The services entry is a list of online weather services to upload data and files to, e.g. 'underground' or ('ftp', '24hrs.txt').
Add the names of your template files and weather services to the appropriate entries, for example:
[logged]
services = ['underground', 'metoffice']
plot = []
text = []
[hourly]
services = [('twitter', 'tweet.txt'),
('ftp', '7days.png', '24hrs.png', 'rose_24hrs.png',
'24hrs.txt', '6hrs.txt', '7days.txt')]
plot = ['7days.png.xml', '24hrs.png.xml', 'rose_24hrs.png.xml']
text = ['tweet.txt', '24hrs.txt', '6hrs.txt', '7days.txt']
[12 hourly]
services = []
plot = []
text = []
[daily]
services = [('twitter', 'forecast.txt'), ('ftp', '28days.png', 'allmonths.txt')]
plot = ['28days.png.xml']
text = ['forecast.txt', 'allmonths.txt']
Note that the twitter and ftp “services” use files generated by the plot and text items.
It’s probably best not to add all of these at once.
You could start by uploading one file to your web site, then when that’s working add the remaining web site files.
You can add Twitter and other services later on.
You can test that things are working by removing the [last update] section from status.ini, then running pywws.hourly again:
pywws-hourly -v ~/weather/data
New in version 14.05.dev1211: [cron name] sections.
If you need more flexibility in when tasks are done you can use [cron name] sections.
See weather.ini - configuration file format for more detail.
Run as a cron job¶
Most UNIX/Linux systems have a ‘cron’ daemon that can run programs at certain times, even if you are not logged in to the computer.
You edit a ‘crontab’ file to specify what to run and when to run it.
For example, to run pywws.hourly every hour, at zero minutes past the hour:
0 * * * * pywws-hourly /home/xxx/weather/data
This might work, but if it didn’t you probably won’t get any error messages to tell you what went wrong.
It’s much better to run a script that runs pywws.hourly and then emails you any output it produces.
Here’s a script I’ve used:
#!/bin/sh
#
# weather station logger calling script
export PATH=$PATH:/usr/local/bin
if [ ! -d /home/jim/weather/data/ ]; then
exit
fi
log=/var/log/log-weather
pywws-hourly -v /home/jim/weather/data >$log 2>&1
# mail the log file
/home/jim/scripts/email-log.sh $log "weather log"
You’ll need to edit this quite a lot to suit your file locations and so on, but it gives some idea of what to do.
Comments or questions? Please subscribe to the pywws mailing list http://groups.google.com/group/pywws and let us know.
How to set up ‘live’ logging with pywws¶
Introduction¶
There are two quite different modes of operation with pywws.
Traditionally pywws.hourly would be run at regular intervals (usually an hour) from cron.
This is suitable for fairly static websites, but more frequent updates can be useful for sites such as Weather Underground (http://www.wunderground.com/).
The newer pywws.livelog program runs continuously and can upload data every 48 seconds.
Getting started¶
First of all, you need to install pywws and make sure it can get data from your weather station. See How to get started with pywws for details.
If you have previously been using pywws.hourly then disable your ‘cron’ job (or whatever else you use to run it) so it no longer runs.
You should never run pywws.hourly and pywws.livelog at the same time.
Try running pywws.livelog from the command line, with a high level of verbosity so you can see what’s happening.
Use the pywws-livelog command to run pywws.livelog:
pywws-livelog -vvv ~/weather/data
Within five minutes (assuming you have set a 5 minute logging interval) you should see a ‘live_data new ptr’ message, followed by fetching any new data from the weather station and processing it.
Let pywws.livelog run for a minute or two longer, then kill the process by typing ‘<Ctrl>C’.
Changed in version 14.04.dev1194: the pywws-livelog command replaced scripts/pywws-livelog.py.
Configuring file locations¶
Open your weather.ini file with a text editor.
You should have a [paths] section similar to the following (where xxx is your user name):
[paths]
work = /tmp/pywws
templates = /home/xxx/weather/templates/
graph_templates = /home/xxx/weather/graph_templates/
modules = /home/xxx/weather/modules/
Edit these to suit your installation and preferences.
work is a temporary directory used to store intermediate files.
If your computer uses solid state storage, such as a Raspberry Pi’s SD card, it’s a good idea to make this a “RAM disk” to reduce storage “wear”.
templates is the directory where you keep your text template files, graph_templates is the directory where you keep your graph template files, and modules is a directory for any extra modules you write.
Don’t use the pywws example directories for your templates, as they will get over-written when you upgrade pywws.
Copy your text and graph templates to the appropriate directories.
You may find some of the examples provided with pywws useful to get started.
The pywws-version -v command should show you where the examples are on your computer.
New in version 14.04.dev1194: the pywws-version command.
Configuring periodic tasks¶
In weather.ini you should have a [live] section similar to the following:
[live]
services = []
text = []
plot = []
This section specifies what pywws should do every time it gets a new reading from the weather station, i.e. every 48 seconds.
The services entry is a list of online weather services to upload data to, e.g. 'underground' or ('ftp', '24hrs.txt').
The plot and text entries are lists of template files for plots and text files to be processed.
You should probably leave all of these blank except for services.
If you use YoWindow (http://yowindow.com/) you can add an entry to the [live] section to specify your YoWindow file, e.g.:
[live]
services = ['underground']
text = ['yowindow.xml']
plot = []
If you don’t already have them, create four more sections in your weather.ini file: [logged], [hourly], [12 hourly] and [daily].
These sections should have similar entries to the [live] section, and specify what to do every time data is logged (5 to 30 minutes, depending on your logging interval), every hour, twice daily and once per day.
Add the names of your template files to the appropriate entries, for example:
[logged]
services = ['underground', 'metoffice']
plot = []
text = []
[hourly]
services = [('twitter', 'tweet.txt'),
('ftp', '7days.png', '24hrs.png', 'rose_24hrs.png',
'24hrs.txt', '6hrs.txt', '7days.txt')]
plot = ['7days.png.xml', '24hrs.png.xml', 'rose_24hrs.png.xml']
text = ['tweet.txt', '24hrs.txt', '6hrs.txt', '7days.txt']
[12 hourly]
services = []
plot = []
text = []
[daily]
services = [('twitter', 'forecast.txt'), ('ftp', '28days.png', 'allmonths.txt')]
plot = ['28days.png.xml']
text = ['forecast.txt', 'allmonths.txt']
Note that the twitter and ftp “services” use files generated by the plot and text items.
It’s probably best not to add all of these at once.
You could start by uploading one file to your web site, then when that’s working add the remaining web site files.
You can add Twitter and other services later on.
New in version 14.05.dev1211: [cron name] sections.
If you need more flexibility in when tasks are done you can use [cron name] sections.
See weather.ini - configuration file format for more detail.
Create a dedicated user (optional)¶
As pywws will be running continuously, and contacting various computers on the internet, there is a very remote risk that one of its dependencies has a security flaw that might allow someone to gain unauthorised to your computer. Running pywws as a user with minimal privileges adds a little extra protection.
You can create a user with the adduser command:
sudo adduser --system --disabled-login --shell=/bin/false pywws
The exact syntax may vary according to your operating system.
The important thing is to create a user that can’t login, and can’t run sudo, but does have a home directory.
Run in the background¶
In order to have pywws.livelog carry on running after you finish using your computer it needs to be run as a “background job”.
On most Linux / UNIX systems you can do this by putting an ampersand (‘&’) at the end of the command line.
Running a job in the background like this doesn’t always work as expected: the job may suspend when you log out.
It’s much better to run as a proper UNIX ‘daemon’ process.
On recent versions of Linux the systemd service manager makes it easy to create a daemon process.
The service is defined in a file /etc/systemd/system/pywws.service:
[Unit]
Description=pywws weather station live logging
After=time-sync.target
[Service]
Type=simple
User=pywws
Restart=on-failure
ExecStart=/usr/local/bin/pywws-livelog -v -l systemd /home/pywws/data/
The [Unit] section says pywws shouldn’t start until the computer has set its clock correctly. This is important on computers without a battery-backed real time clock, such as the Raspberry Pi.
The [Service] section specifies which user should run pywws and gives the command to run it.
The -l systemd option sends log messages to systemd instead of using a normal pywws log file.
You can use sudo service pywws start to test the pywws.service file.
After starting sudo service pywws status shows if it’s running OK, and the last few log messages:
jim@gordon:~ $ sudo service pywws status
● pywws.service - pywws weather station live logging
Loaded: loaded (/etc/systemd/system/pywws.service; static; vendor preset: enabled)
Active: active (running) since Thu 2018-08-23 17:49:01 BST; 12min ago
Main PID: 30946 (pywws-livelog)
CGroup: /system.slice/pywws.service
└─30946 /usr/bin/python3 /usr/local/bin/pywws-livelog -v -l systemd /home/pywws/data/
Aug 23 17:49:44 gordon pywws-livelog[30946]: pywws.service.wetterarchivde:server response "{'version': '6.0', 'status': 'SUCCESS'}"
Aug 23 17:49:44 gordon pywws-livelog[30946]: pywws.service.metoffice:OK
Aug 23 17:49:45 gordon pywws-livelog[30946]: pywws.service.openweathermap:OK
Aug 23 17:49:46 gordon pywws-livelog[30946]: pywws.service.cwop:OK
Aug 23 17:49:46 gordon pywws-livelog[30946]: pywws.service.underground:server response "success"
Aug 23 17:57:41 gordon pywws-livelog[30946]: pywws.weatherstation:setting sensor clock 5.33264
Aug 23 17:57:41 gordon pywws-livelog[30946]: pywws.weatherstation:sensor clock drift 1.4672 1.08387
Aug 23 18:00:25 gordon pywws-livelog[30946]: pywws.service.mastodon:OK
Aug 23 18:00:26 gordon pywws-livelog[30946]: pywws.service.twitter:OK
Aug 23 18:00:26 gordon pywws-livelog[30946]: pywws.service.sftp:OK
jim@gordon:~ $
If you’d prefer to use a normal pywws log file the pywws.service file might look like this:
[Unit]
Description=pywws weather station live logging
After=time-sync.target
[Service]
Type=simple
User=pywws
Restart=on-failure
PermissionsStartOnly=true
ExecStartPre=/bin/mkdir -p /var/log/pywws
ExecStartPre=/bin/chown -R pywws:nogroup /var/log/pywws/
ExecStart=/usr/local/bin/pywws-livelog -v -l /var/log/pywws/pywws.log /home/pywws/data/
In this example the log file is /var/log/pywws/pywws.log.
The directory /var/log/pywws/ might not exist after a reboot (e.g. if /var/log/ has been moved to RAM disk to reduce SD card wear) so ExecStartPre is used to create it and transfer its ownership to the pywws user.
PermissionsStartOnly=true ensures the ExecStartPre commands are run as root.
The udev system can be used to start the pywws service when the computer boots or the weather station is plugged into the USB port.
(It also stops pywws if the weather station is unplugged.)
Create a file /etc/udev/rules.d/39-weather-station.rules as follows:
SUBSYSTEM=="usb" \
, ATTRS{idVendor}=="1941" \
, ATTRS{idProduct}=="8021" \
, OWNER="pywws" \
, TAG+="systemd" \
, ENV{SYSTEMD_WANTS}="pywws.service"
This sets the owner of the weather station’s USB port to pywws, then adds pywws.service to the things systemd should be running.
If you can’t use systemd for some reason then the pywws.livelogdaemon program can be used to run pywws as a daemon, if you have the python-daemon library installed:
pywws-livelog-daemon -v ~/weather/data ~/weather/data/pywws.log start
(Note that the log file is a required parameter, not an option.) Unfortunately the python-daemon package appears not to be maintained, and I’ve had problems with it on some Linux versions. You’ll also need to setup something to start pywws automatically.
There are various ways of configuring a Linux system to start a program when the machine boots up.
Typically these involve putting a file in /etc/init.d/.
A slightly harder problem is ensuring a program restarts if it crashes.
My solution to both problems is to run the following script from cron, several times an hour.
#!/bin/sh
export PATH=$PATH:/usr/local/bin
# exit if NTP hasn't set computer clock
[ `ntpdc -c sysinfo | awk '/stratum:/ {print $2}'` -ge 10 ] && exit
pidfile=/var/run/pywws.pid
datadir=/home/jim/weather/data
logfile=$datadir/live_logger.log
# exit if process is running
[ -f $pidfile ] && kill -0 `cat $pidfile` && exit
# email last few lines of the logfile to see why it died
if [ -f $logfile ]; then
log=/tmp/log-weather
tail -40 $logfile >$log
/home/jim/scripts/email-log.sh $log "weather log"
rm $log
fi
# restart process
pywws-livelog-daemon -v -p $pidfile $datadir $logfile start
The process id of the daemon is stored in pidfile.
If the process is running, the script does nothing.
If the process has crashed, it emails the last 40 lines of the log file to me (using a script that creates a message and passes it to sendmail) and then restarts pywws.livelogdaemon.
You’ll need to edit this quite a lot to suit your file locations and so on, but it gives some idea of what to do.
Comments or questions? Please subscribe to the pywws mailing list http://groups.google.com/group/pywws and let us know.
How to integrate pywws with various weather services¶
This guide gives brief instructions on how to use pywws with some other weather services and software. It is not comprehensive, and most services are covered in more detail elsewhere.
YoWindow¶
YoWindow is a weather display widget that can display data from an internet source, or from your weather station. To display data from your station pywws needs to write to a local file, typically every 48 seconds when new data is received. This is easy to do:
Stop all pywws software
Copy the
yowindow.xmlexample template to your text template directory.Add the yowindow template to the
[live]tasks inweather.ini:[live] text = ['yowindow.xml']
Restart pywws live logging.
This will write the file to the output subdirectory of the work directory set in weather.ini.
If you prefer to store the file somewhere else you can use the pywws.service.copy service to copy it there. For example:
[copy]
directory = /home/heavyweather/
[live]
text = ['yowindow.xml']
services = [('copy', 'yowindow.xml')]
You can check the file is being updated every 48 seconds by using more or cat to dump it to the screen.
Finally configure yowindow to use this file. See http://yowindow.com/pws_setup.php for instructions on how to do this.
Other “services”¶
The remaining weather service uploads are handled by modules in the pywws.service sub-package.
pywws.service.ftp |
Upload files to a web server by FTP. |
pywws.service.sftp |
Upload files to a web server by SFTP. |
pywws.service.copy |
Copy files to another directory. |
pywws.service.cwop |
Upload weather data to Citizen Weather Observer Program. |
pywws.service.metoffice |
Upload weather data to UK Met Office “WOW”. |
pywws.service.mqtt |
Upload weather data to MQTT message broker. |
pywws.service.openweathermap |
Upload weather data to Open Weather Map. |
pywws.service.pwsweather |
Upload weather data to PWS Weather. |
pywws.service.temperaturnu |
Upload current temperature to temperatur.nu. |
pywws.service.underground |
Upload data to Weather Underground. |
pywws.service.weathercloud |
Upload data to WeatherCloud. |
pywws.service.wetterarchivde |
Upload weather data to wetter.com. |
pywws.service.windy |
Upload data to Windy. |
pywws.service.twitter |
Post messages to Twitter. |
pywws.service.mastodon |
Post messages to Mastodon. |
These each use a separate thread to upload the data so that a slow or not responding service doesn’t delay other processing or uploads.
The service uploaders are all used in a similar fashion:
Create an account with the service.
Stop all pywws software.
Run the service module directly to initialise its entry in
weather.ini. For example:python -m pywws.service.underground /home/jim/weather/data
Edit
weather.iniand add your account details to the appropriate section (e.g.[underground]).Run the service module directly (with high verbosity) to make sure your account details are correct:
python -m pywws.service.underground -vvv /home/jim/weather/data
Each service’s server software responds differently to correct or incorrect uploads. You should be able to tell from the response if it was successful or not.
Edit
weather.iniand add the service to the[logged](and optionally[live]) sections, e.g.:[logged] services = ['underground'] [live] services = ['underground']
Note that some services, such as
pywws.service.copy, need one or more parameters. Instead of a single word entry, such asunderground, these use a bracketed list, for example('copy', 'yowindow.xml').Restart pywws live logging.
Some of the services are more complicated to configure. More detailed instructions are given in the module’s documentation. Follow the links in the table above.
Many of the services will upload the last seven days of data (referred to as “catchup” mode) when first run. This may take an hour or more, but the use of separate threads means this doesn’t adversely affect the rest of pywws.
Writing your own uploader¶
If you’d like to send data to a service which is not (yet) included in pywws you can write your own uploader module and put it in your modules directory.
You should start by copying one of the existing modules from pywws.service.
Choose one with an API most like the service you want to upload to.
Give the module a one word lowercase name that will be used as the uploader service name.
Testing the module is a little different from before:
python ~/weather/modules/myservice.py -vvv ~/weather/data
where ~/weather/modules/myservice.py is the full path of your new module.
Note what sort of response you get from the server.
Some servers, such as Weather Underground, send a single word 'success' response to indicate success, and a longer string indicating the cause of any failure.
Other servers use HTTP response codes to indicate failure.
Your module’s upload_data method must return a (bool, str) tuple where the bool value indicates success (if True) and the str value contains any message from the server.
(If the server returns no message this string should be set to 'OK'.)
Under normal operation pywws will log this message whenever it changes.
Once your uploader is working you could contribute it to pywws if it’s likely to be useful to other people. Don’t forget to document it fully, then either send it to Jim or create a GitHub pull request. See Contributing to pywws for instructions on doing this.
Comments or questions? Please subscribe to the pywws mailing list http://groups.google.com/group/pywws and let us know.
How to use pywws in another language¶
Introduction¶
Some parts of pywws can be configured to use your local language instead of British English. This requires an appropriate language file which contains translations of the various strings used in pywws. The pywws project relies on users to provide these translations.
The pywws documentation can also be translated into other languages. This is a lot more work, but could be very useful to potential users who do not read English very well.
Using existing language files¶
There may already be a pywws translation for your preferred language.
First you need to choose the appropriate two-letter code from the list at http://www.w3schools.com/tags/ref_language_codes.asp.
For example, fr is the code for French.
Now use the pywws.localisation module to do a quick test:
python -m pywws.localisation -t fr
This should produce output something like this:
Locale changed from (None, None) to ('fr_FR', 'UTF-8')
Translation set OK
Locale
decimal point: 23,2
date & time: lundi, 17 décembre (17/12/2012 16:00:48)
Translations
'NNW' => 'NNO'
'rising very rapidly' => 'en hausse très rapide'
'Rain at times, very unsettled' => 'Quelques précipitations, très perturbé'
This shows that pywws is already able to generate French output, and that your installation is correctly configured.
Now you can edit the language entry in your weather.ini file to use your language code.
If the above test shows no translations into your language then you need to create a new language file, as described below.
The pywws default text encoding is ISO-8859-1, also known as Latin-1.
This is suitable for several western European languages but not for some others.
If you encounter problems you may need to use a different encoding.
See the documentation of pywws.template and pywws.plot for more details.
If you have downloaded the pywws source files, or cloned the GitHub repository (see how to get started with pywws), you can compile a non-English copy of the documentation. This requires the Sphinx package, see dependencies.
First delete the old documentation (if it exists) and then recompile using your language:
cd ~/weather/pywws
rm -rf doc
LANG=fr python -B setup.py build_sphinx
Note that the build_sphinx command doesn’t have a --locale (or -l) option, so the language is set by a temporary environment variable.
You can view the translated documentation by using a web browser to read the file ~/weather/pywws/doc/html/index.html.
Writing new language files¶
There are two ways to write new language files (or update existing ones) – use the Transifex online system or use local files. Transifex is preferred as it allows several people to work on the same language, and makes your work instantly available to others.
To test your translation you will need to have downloaded the pywws source files, or cloned the GitHub repository (see how to get started with pywws).
You will also need to install the Babel package, see dependencies.
If you’d like to use Transifex, please go to the pywws Transifex project, click on “help translate pywws” and create a free account.
Visit the pywws project page on Transifex and click on your language, then click on the “resource” you want to translate.
(pywws contains the program strings used when running pywws, the others contain strings from the pywws documentation.)
This opens a dialog where you can choose to translate the strings online.
Please read Notes for translators before you start.
When you have finished translating you should use the transifex-client program (see dependencies) to download files for testing.
For example, this command downloads any updated files for the French language:
cd ~/weather/pywws
tx pull -l fr
Now you are ready to Test the pywws translations.
If you prefer not to use the Transifex web site you can edit language files on your own computer. This is done in two stages, as follows.
Program messages are extracted using the Babel package:
cd ~/weather/pywws
mkdir -p build/gettext
python -B setup.py extract_messages
This creates the file build/gettext/pywws.pot.
This is a “portable object template” file that contains the English language strings to be translated.
The documentation strings are extracted using the Sphinx package:
cd ~/weather/pywws
python -B setup.py extract_messages_doc
This creates several .pot files in the build/gettext/ directory.
The sphinx-intl command is used to convert the .pot files to language specific .po files:
cd ~/weather/pywws
sphinx-intl update --locale-dir src/pywws/lang -p build/gettext -l fr
Now you can open the .po files in src/pywws/lang/fr/LC_MESSAGES/ with your favourite text editor and start filling in the empty msgstr strings with your translation of the corresponding msgid string.
Please read Notes for translators before you start.
Test the pywws translations¶
The Babel package is used to compile program strings:
python -B setup.py compile_catalog --locale fr
(Replace fr with the code for the language you are testing.)
After compilation you can test the translation:
python setup.py build
sudo python setup.py install
python -m pywws.Localisation -t fr
Sphinx is used to build the translated documentation:
cd ~/weather/pywws
rm -rf doc
LANG=fr python -B setup.py build_sphinx
You can view the translated documentation by using a web browser to read the file ~/weather/pywws/doc/html/index.html.
Notes for translators¶
The pywws program strings (pywws.po) are quite simple.
They comprise simple weather forecasts (“Fine weather”), air pressure changes (“rising quickly”) and the 16 points of the compass (“NNE”).
Leave the “(%Z)” in “Time (%Z)” unchanged and make sure your translation’s punctuation matches the original.
The other files contain strings from the pywws documentation. These are in reStructuredText. This is mostly plain text, but uses characters such as backquotes (`), colons (:) and asterisks (*) for special purposes. You need to take care to preserve this special punctuation. Do not translate program source, computer instructions and cross-references like these:
`pip <http://www.pip-installer.org/>`_
:py:class:`datetime.datetime`
:obj:`ParamStore <pywws.DataStore.ParamStore>`\\ (root_dir, file_name)
pywws.Forecast
``pywws-livelog``
Translating all of the pywws documentation is a lot of work.
However, when the documentation is “compiled” any untranslated strings revert to their English original.
This means that a partial translation could still be useful – I suggest starting with the documentation front page, index.po.
Send Jim the translation¶
I’m sure you would like others to benefit from the work you’ve done in translating pywws.
If you’ve been using Transifex then please send me an email (jim@jim-easterbrook.me.uk) to let me know there’s a new translation available.
Otherwise, please email me any .po files you create.
Comments or questions? Please subscribe to the pywws mailing list http://groups.google.com/group/pywws and let us know.
weather.ini - configuration file format¶
Nearly all configuration of pywws is via a single file in the data directory: weather.ini. This file has a structure similar to that of Microsoft Windows INI files. It is divided into “sections”, each of which has a number of “name = value” entries. The order in which sections appear is not important.
Any plain text editor can be used to do edit the file. (Don’t try to edit it while any other pywws software is running.) In many cases pywws will initialise the entries to sensible values.
Another file, status.ini, is used to store some information that pywws uses internally. It is described at the end of this document. In normal use you should not need to edit it.
The following sections are currently in use:
- config: miscellaneous system configuration.
- paths: directories in which templates etc. are stored.
- live: tasks to be done every 48 seconds.
- logged: tasks to be done every time the station logs a data record.
- cron: tasks to be done at a particular time or date.
- hourly: tasks to be done every hour.
- 12 hourly: tasks to be done every 12 hours.
- daily: tasks to be done every day.
- ftp: configuration of uploading to a website.
- underground, metoffice, temperaturnu etc: configuration of posting to ‘services’.
config: miscellaneous system configuration¶
[config]
ws type = 1080
day end hour = 9, False
pressure offset = 9.4
gnuplot encoding = utf8
template encoding = iso-8859-1
language = en_GB
logdata sync = 1
rain day threshold = 0.2
usb activity margin = 3.0
gnuplot version = 4.6
frequent writes = False
ws type is the “class” of weather station. It should be set to 1080 for most weather stations, or 3080 if your station console displays solar illuminance.
day end hour is the end of the “meteorological day”, in local time, and whether daylight savings time should be applied.
Typical values are 9, False (UK Met Office standard) or 0, True (midnight, summer and winter).
Note that using daylight savings time will mean that one day a year is 25 hours long and one is 23 hours long.
You must update all your stored data by running pywws.reprocess after you change this value.
pressure offset is added to the absolute air pressure to get the relative (sea level) air pressure, unit: hPa.
The initial value is copied from the weather station, assuming you have set it up to display the correct relative pressure, but you can adjust the value in weather.ini to calibrate your station. You can calculate this value or use the difference to a trusted weatherstation or weather service nearby.
You must update all your stored data by running pywws.reprocess after you change this value.
Changed in version 13.10_r1082: made pressure offset a config item.
Previously it was always read from the weather station.
gnuplot encoding is the text encoding used when plotting graphs. The default value of iso_8859_1 allows the degree symbol, which is useful in a weather application! Other values might be needed if your language includes accented characters. The possible values depend on your gnuplot installation so some experimentation may be needed.
template encoding is the text encoding used for templates.
The default value is iso-8859-1, which is the encoding used in most of the example templates.
If you create templates with a different character set, you should change this value to match your templates.
language is used to localise pywws. It’s optional, as pywws usually uses the computer’s default language as set by the LANG environment variable. The available languages are those in the translations subdirectory of your pywws installation. If you set any other language, pywws will fall back to using English.
logdata sync sets the quality of synchronisation used by pywws.logdata. Set it to 0 for fast & inaccurate or 1 for slower but precise.
rain day threshold is the amount of rain (in mm) that has to fall in one day for it to qualify as a rainy day in the monthly summary data.
You must update all your stored data by running pywws.reprocess after you change this value.
New in version 13.10_r1094: usb activity margin controls the algorithm that avoids the “USB lockup” problem that affects some stations.
It sets the number of seconds either side of expected station activity (receiving a reading from outside or logging a reading) that pywws does not get data from the station.
If your station is not affected by the USB lockup problem you can set usb activity margin to 0.0.
New in version 13.11_r1102: gnuplot version tells pywws.plot and pywws.windrose what version of gnuplot is installed on your computer.
This allows them to use version-specific features to give improved plot quality.
New in version 14.01_r1133: frequent writes tells pywws.regulartasks to save weather data and status to file every time there is new logged data.
The default is to save the files every hour, to reduce “wear” on solid state memory such as the SD cards used with Raspberry Pi computers.
If your weather data directory is stored on a conventional disc drive you can set frequent writes to True.
paths: directories in which templates etc. are stored¶
[paths]
templates = /home/$USER/weather/templates/
graph_templates = /home/$USER/weather/graph_templates/
work = /tmp/weather
user_calib = /home/$USER/weather/modules/usercalib
modules = /home/$USER/weather/modules/
These entries specify where your text templates and graph templates are stored, where temporary files should be created, where template output (that is not uploaded) should be put, the location of your calibration module (if you have one), and where any other modules you create are stored.
live: tasks to be done every 48 seconds¶
[live]
services = ['underground', ('copy', 'yowindow.xml')]
text = ['yowindow.xml']
plot = []
This section specifies tasks that are to be carried out for every data sample during ‘live logging’, i.e. every 48 seconds.
services is a list of ‘services’ to upload data to.
Some are just a single word, others have one or more parameters and need to be enclosed in brackets.
Each one listed must have a module in pywws.service or your modules directory.
See integration - other services for more detail.
pywws will automatically limit the frequency of service uploads according to each service’s requirements.
text and plot are lists of text and plot templates to be processed.
logged: tasks to be done every time the station logs a data record¶
[logged]
services = ['underground', 'metoffice']
text = []
plot = []
This section specifies tasks that are to be carried out every time a data record is logged when ‘live logging’ or every time an hourly cron job is run.
services is a list of ‘services’ to upload data to.
Some are just a single word, others have one or more parameters and need to be enclosed in brackets.
Each one listed must have a module in pywws.service or your modules directory.
See integration - other services for more detail.
text and plot are lists of text and plot templates to be processed.
cron: tasks to be done at a particular time or date¶
New in version 14.05.dev1211.
[cron daily 9]
format = 0 9 * * *
plot = ['28days.png.xml']
text = ['forecast.txt', 'forecast_9am.txt', 'forecast_week.txt']
services = [('twitter', 'forecast.txt'), ('ftp', 'forecast_9am.txt', 'forecast_week.txt')]
[cron daily 21]
format = 0 21 * * *
text = ['forecast_9am.txt']
services = [('ftp', 'forecast_9am.txt')]
plot = []
[cron weekly]
format = 0 9 * * 6
plot = ['2008.png.xml', '2009.png.xml', '2010.png.xml', '2011.png.xml',
'2012.png.xml', '2013.png.xml']
text = ['2008.txt', '2009.txt', '2010.txt', '2011.txt', '2012.txt', '2013.txt']
services = [('ftp', '2008.png', '2009.png', '2010.png', '2011.png',
'2012.png', '2013.png', '2008.txt', '2009.txt',
'2010.txt', '2011.txt', '2012.txt', '2013.txt')]
[cron name] sections provide a very flexible way to specify tasks to be done at a particular time and/or date.
name can be anything you like, but each [cron name] section must have a unique name.
To use [cron name] sections you need to install the “croniter” package.
See Dependencies for more detail.
format specifies when the tasks should be done (in local time), in the usual crontab format.
(See man 5 crontab on any Linux computer.)
Processing is not done exactly on the minute, but when the next live or logged data arrives.
hourly: tasks to be done every hour¶
[hourly]
services = [('twitter', 'tweet.txt'),
('ftp', '7days.png', '24hrs.png', 'rose_12hrs.png',
'24hrs.txt', '6hrs.txt', '7days.txt', 'feed_hourly.xml')]
text = ['tweet.txt', '24hrs.txt', '6hrs.txt', '7days.txt', 'feed_hourly.xml']
plot = ['7days.png.xml', '24hrs.png.xml', 'rose_12hrs.png.xml']
This section specifies tasks that are to be carried out every hour when ‘live logging’ or running an hourly cron job.
services is a list of ‘services’ to upload data to.
Some are just a single word, others have one or more parameters and need to be enclosed in brackets.
Each one listed must have a module in pywws.service or your modules directory.
See integration - other services for more detail.
text and plot are lists of text and plot templates to be processed.
12 hourly: tasks to be done every 12 hours¶
[12 hourly]
services = []
text = []
plot = []
This section specifies tasks that are to be carried out every 12 hours when ‘live logging’ or running an hourly cron job. Use it for things that don’t change very often, such as monthly graphs. The tasks are done at your day end hour, and 12 hours later.
services is a list of ‘services’ to upload data to.
Some are just a single word, others have one or more parameters and need to be enclosed in brackets.
Each one listed must have a module in pywws.service or your modules directory.
See integration - other services for more detail.
text and plot are lists of text and plot templates to be processed.
daily: tasks to be done every 24 hours¶
[daily]
services = [('ftp', '2008.png', '2009.png', '2010.png', '28days.png', 'feed_daily.xml')]
text = ['feed_daily.xml']
plot = ['2008.png.xml', '2009.png.xml', '2010.png.xml', '28days.png.xml']
This section specifies tasks that are to be carried out every day when ‘live logging’ or running an hourly cron job. Use it for things that don’t change very often, such as monthly or yearly graphs. The tasks are done at your day end hour.
services is a list of ‘services’ to upload data to.
Some are just a single word, others have one or more parameters and need to be enclosed in brackets.
Each one listed must have a module in pywws.service or your modules directory.
See integration - other services for more detail.
text and plot are lists of text and plot templates to be processed.
underground, metoffice, temperaturnu etc: configuration of posting to ‘services’¶
[underground]
station = IXYZABA5
password = secret
These sections contain information such as passwords and station IDs needed to upload data to weather services. The names of the data entries depend on the service. The example shown is for Weather Underground.
station is the PWS ID allocated to your weather station by Weather Underground.
password is your Weather Underground password.
status.ini - status file format¶
This file is written by pywws and should not (usually) be edited. The following sections are currently in use:
- fixed: values copied from the weather station’s “fixed block”.
- clock: synchronisation information.
- last update: date and time of most recent task completions.
fixed: values copied from the weather station’s “fixed block”¶
[fixed]
fixed block = {...}
fixed block is all the data stored in the first 256 bytes of the station’s memory.
This includes maximum and minimum values, alarm threshold settings, display units and so on.
clock: synchronisation information¶
[clock]
station = 1360322930.02
sensor = 1360322743.69
These values record the measured times when the station’s clock logged some data and when the outside sensors transmitted a new set of data. They are used to try and prevent the USB interface crashing if the computer accesses the weather station at the same time as either of these events, a common problem with many EasyWeather compatible stations. The times are measured every 24 hours to allow for drift in the clocks.
last update: date and time of most recent task completions¶
[last update]
hourly = 2013-05-30 19:04:15
logged = 2013-05-30 19:04:15
daily = 2013-05-30 09:04:15
openweathermap = 2013-05-30 18:59:15
underground = 2013-05-30 18:58:34
metoffice = 2013-05-30 18:59:15
12 hourly = 2013-05-30 09:04:15
These record date & time of the last successful completion of various tasks. They are used to allow unsuccessful tasks (e.g. network failure preventing uploads) to be retried after a few minutes.
Comments or questions? Please subscribe to the pywws mailing list http://groups.google.com/group/pywws and let us know.
Understanding pywws log files¶
The pywws software uses Python’s logging system to report errors, warnings and information that may be useful in diagnosing problems. When you run the software interactively these messages are sent to your terminal window, when running a daemon process or cron job they should be written to a log file.
This document explains some of the pywws logging messages. It’s not a complete guide, and the messages you see will depend on your weather station and configuration, but it should help you understand some of the more common ones.
Many pywws commands have a -v or --verbose option to increase the verbosity of the logging messages.
This option can be repeated for even more verbosity, which may be useful when trying to diagnose a particular fault.
Here are some typical logging outputs. The first shows pywws being run interactively:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 | jim@gordon:~ $ pywws-livelog -v ~/weather/data/
09:25:11:pywws.logger:pywws version 18.5.0, build 1541 (d505b50)
09:25:11:pywws.logger:Python version 3.5.3 (default, Jan 19 2017, 14:11:04) [GCC 6.3.0 20170124]
09:25:12:pywws.weatherstation:using pywws.device_libusb1
09:25:17:pywws.calib:Using user calibration
09:25:46:pywws.weatherstation:status {'rain_overflow': False, 'lost_connection': False}
09:25:53:pywws.regulartasks:doing task sections ['live']
09:25:54:pywws.service.openweathermap:OK
09:25:54:pywws.service.openweathermap:1 record sent
09:25:55:pywws.service.underground:server response "success"
09:25:55:pywws.service.underground:1 record sent
09:26:41:pywws.regulartasks:doing task sections ['live']
09:26:42:pywws.service.openweathermap:1 record sent
09:26:44:pywws.service.underground:1 record sent
09:27:29:pywws.regulartasks:doing task sections ['live']
09:27:30:pywws.service.openweathermap:1 record sent
09:27:33:pywws.service.underground:1 record sent
09:28:17:pywws.regulartasks:doing task sections ['live']
09:28:19:pywws.service.openweathermap:1 record sent
09:28:22:pywws.service.underground:1 record sent
09:29:05:pywws.regulartasks:doing task sections ['live']
09:29:07:pywws.service.openweathermap:1 record sent
09:29:07:pywws.service.underground:1 record sent
09:29:16:pywws.weatherstation:live_data new ptr: 0053a0
09:29:16:pywws.process:Generating summary data
09:29:17:pywws.regulartasks:doing task sections ['logged']
09:29:17:pywws.service.wetterarchivde:server response "{'version': '6.0', 'status': 'SUCCESS'}"
09:29:17:pywws.service.wetterarchivde:1 record sent
09:29:17:pywws.service.cwop:OK
09:29:17:pywws.service.cwop:1 record sent
09:29:20:pywws.service.metoffice:OK
09:29:20:pywws.service.metoffice:1 record sent
|
Note that each line begins with a time stamp, in local time.
Line 1 is the command used to start pywws.
Line 2 shows the pywws version.
Line 3 shows the Python version.
Line 4 shows which Python USB library is being used.
Line 5 shows that a "user calibration" routine is being used.
Line 6 shows the current value of the weather station “status” bits.
This will be shown again if the status changes.
Lines 7, 12, 15, 18 & 21 show that tasks in the [live] section of weather.ini are being executed.
You can see from the time stamps that they happen at 48 second intervals.
Line 24 shows that the station has “logged” data and moved on to the next memory address.
pywws then generates summary data and executes tasks in the [logged] section of weather.ini.
The remaining lines show uploads to various weather “services” (see How to integrate pywws with various weather services).
Uploads to openweathermap and underground are done every 48 seconds.
Uploads to wetterarchivde, cwop, and metoffice are less frequent.
The first upload to a service logs OK or a message from the server.
This is not shown again unless the response changes.
When running pywws as a daemon process the logging is less verbose:
2018-05-27 10:50:40:pywws.logger:pywws version 18.5.0, build 1541 (d505b50)
2018-05-27 10:51:19:pywws.weatherstation:status {'lost_connection': False, 'rain_overflow': False}
2018-05-27 10:54:19:pywws.service.cwop:1 record(s) dropped
2018-05-27 10:54:19:pywws.service.cwop:OK
2018-05-27 10:54:19:pywws.service.wetterarchivde:server response "{'version': '6.0', 'status': 'SUCCESS'}"
2018-05-27 10:54:19:pywws.service.wetterarchivde:2 records sent
2018-05-27 10:54:20:pywws.service.openweathermap:OK
2018-05-27 10:54:20:pywws.service.openweathermap:2 records sent
2018-05-27 10:54:20:pywws.service.underground:server response "success"
2018-05-27 10:54:20:pywws.service.metoffice:OK
2018-05-27 10:54:20:pywws.service.underground:2 records sent
2018-05-27 10:54:21:pywws.service.metoffice:2 records sent
2018-05-27 11:00:31:pywws.towebsite:OK
2018-05-27 11:00:33:pywws.totwitter:OK
2018-05-27 11:54:14:pywws.weatherstation:setting station clock 14.371
2018-05-27 11:54:14:pywws.weatherstation:station clock drift -0.499934 -0.401405
2018-05-27 14:00:36:pywws.towebsite:[Errno 111] Connection refused
2018-05-27 14:01:16:pywws.towebsite:OK
2018-05-27 15:00:34:pywws.towebsite:[Errno 111] Connection refused
2018-05-27 15:17:25:pywws.towebsite:[Errno 110] Connection timed out
2018-05-27 15:18:05:pywws.towebsite:OK
2018-05-28 01:05:47:pywws.weatherstation:setting sensor clock 11.1295
2018-05-28 01:05:47:pywws.weatherstation:sensor clock drift 1.02042 0.987513
2018-05-28 01:10:29:pywws.service.metoffice:HTTPConnectionPool(host='wow.metoffice.gov.uk', port=80): Read timed out. (read timeout=60)
2018-05-28 01:11:09:pywws.service.metoffice:repeated data 2018-05-28 00:09:14
2018-05-28 01:14:25:pywws.service.metoffice:OK
2018-05-28 01:50:31:pywws.service.metoffice:HTTPConnectionPool(host='wow.metoffice.gov.uk', port=80): Read timed out. (read timeout=60)
2018-05-28 01:52:52:pywws.service.metoffice:repeated data 2018-05-28 00:49:14
2018-05-28 01:55:22:pywws.service.metoffice:HTTPConnectionPool(host='wow.metoffice.gov.uk', port=80): Read timed out. (read timeout=60)
2018-05-28 01:57:42:pywws.service.metoffice:OK
2018-05-28 09:00:38:pywws.totwitter:2 records sent
2018-05-28 10:50:07:pywws.service.metoffice:HTTPConnectionPool(host='wow.metoffice.gov.uk', port=80): Read timed out. (read timeout=60)
2018-05-28 10:50:47:pywws.service.metoffice:OK
2018-05-28 11:59:14:pywws.weatherstation:setting station clock 14.0721
2018-05-28 11:59:14:pywws.weatherstation:station clock drift -0.297843 -0.375514
2018-05-28 19:44:20:pywws.weatherstation:live_data log extended
Each line begins with a date and time stamp, in local time.
Because pywws hadn’t been run for 10 minutes each service starts by uploading 2 “catchup” records, except cwop which only accepts live data so one record is dropped.
At 11:00 am the first [hourly] tasks are run, uploading to a website and sending to Twitter.
The 14:00 and 15:00 website uploads failed, but were successful later on.
The metoffice upload at 01:10:29 appeared to fail, but when it was retried at 01:11:09 the server said it already had the data.
(Note the data timestamp is in UTC, the log message time stamp is in BST, one hour ahead.)
The remaining lines show status messages that are described in more detail below.
Clock drift¶
2018-05-28 01:05:47:pywws.weatherstation:setting sensor clock 11.1295
2018-05-28 01:05:47:pywws.weatherstation:sensor clock drift 1.02042 0.987513
2018-05-28 11:59:14:pywws.weatherstation:setting station clock 14.0721
2018-05-28 11:59:14:pywws.weatherstation:station clock drift -0.297843 -0.375514
These lines report how the weather station’s internal (“station”) and external (“sensor”) clocks are drifting with respect to the computer’s clock.
(The 3080 class stations also have a “solar” clock as the sunlight data is sent at 60 second intervals.)
These measurements are used to avoid accessing the station’s USB port at the same time as it is receiving data or logging data, as this is known to cause some stations’ USB ports to become inaccessible.
The two “drift” figures are the current value (only accurate to about 1 second) and the long term average.
You should ensure that the usb activity margin value in your weather.ini file is at least 0.5 seconds greater than the absolute value of the long term drift of each clock.
Note that these drift values change with temperature.
The clock drifts are measured at approximately 24 hour intervals. If pywws loses synchronisation with your station it will measure them again. Doing this measurement increases the risk of causing a USB lockup, so if pywws often loses synchronisation you should try and find out why it’s happening.
Network problems¶
Occasionally one or more of the services and web sites you upload data to may become unavailable. This leads to error messages like these:
2018-05-28 21:03:02:pywws.service.underground:HTTPSConnectionPool(host='rtupdate.wunderground.com', port=443): Max retries exceeded with url: /weatherstation/updateweatherstation.php?rtfreq=48&winddir=5&softwaretype=pywws&windspeedmph=0.67&tempf=68.7&dateutc=2018-05-28+20%3A02%3A35&dewptf=65.3&action=updateraw&ID=ISURREYE4&windgustmph=2.24&PASSWORD=xxxxxxxx&baromin=30.0910&humidity=89&realtime=1&dailyrainin=0.122173&rainin=0 (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0xb25a3f30>: Failed to establish a new connection: [Errno -2] Name or service not known',))
2018-05-28 21:03:15:pywws.service.openweathermap:HTTPConnectionPool(host='api.openweathermap.org', port=80): Max retries exceeded with url: /data/3.0/measurements?appid=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0xb25a3db0>: Failed to establish a new connection: [Errno 101] Network is unreachable',))
2018-05-29 20:38:42:pywws.service.underground:http status: 500
2018-05-29 20:39:23:pywws.service.underground:server response "success"
2018-05-30 18:19:59:pywws.service.wetterarchivde:http status: 504
2018-05-30 18:20:39:pywws.service.wetterarchivde:server response "{'info': 'Report exists at 2018-05-30T17:19:00.000Z', 'version': '6.0', 'status': 'SUCCESS', 'log': 'dbea2445-d930-4053-89ea-20c04a5030c1'}"
2018-05-30 18:24:17:pywws.service.wetterarchivde:server response "{'version': '6.0', 'status': 'SUCCESS'}"
2018-05-31 01:50:04:pywws.service.openweathermap:('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
2018-05-31 01:50:44:pywws.service.openweathermap:OK
To avoid swamping the log files duplicate messages are not logged. When the response changes it is logged again.
Status¶
2015-09-01 21:50:21:pywws.weather_station:status {'lost_connection': True, 'rain_overflow': False}
The raw weather station data includes some “status” bits.
If any of these bits changes value when pywws is running, the status value is logged.
The most common problem is lost_connection: the weather station console is not receiving data from the outside sensors.
Contact is often restored a few minutes later, but if not you may need to reset your weather station console by taking its batteries out.
The rain_overflow bit is set when the rain gauge counter has reached its maximum value and gone back to zero.
There are 6 bits of data in the status byte whose function is not yet known. If any of these bits is set the value will be added to the reported status. Do let me know if this happens as it might enable us to find the meaning of the unused bits.
Log extended¶
2018-05-29 22:16:19:pywws.weatherstation:live_data log extended
2018-05-29 22:32:19:pywws.weatherstation:live_data log extended
2018-05-29 22:48:19:pywws.weatherstation:live_data log extended
2018-05-29 23:04:19:pywws.weatherstation:live_data log extended
This shows a curiosity in the weather station’s internal processing. As the internal and external sensors drift there comes a time when an external reading is expected at the same time as the station is due to log some data. To avoid a clash the station delays logging by one minute. As the external readings are at 48 second intervals this avoids the problem until 16 minutes later (with the normal 5 minute logging interval) when another one minute delay is needed. Eventually the clocks drift apart and normal operation is resumed.
The 3080 class stations also receive solar data at 60 second intervals.
If this clashes with the logging time the station delays logging by one minute.
Unfortunately this doesn’t help, so the station effectively stops logging data until the clocks drift apart again.
If you are running pywws “live logging” then it will cover the gap by saving live readings at five minute intervals (if your logging interval is set to five minutes) until the station resumes normal operation.
If you are running “hourly” logging then you may get a large gap in your data.
Rain reset¶
2015-08-25 13:30:51:pywws.process:2015-08-25 12:30:48 rain reset 1048.4 -> 1047.1
2015-08-25 13:35:51:pywws.process:2015-08-25 12:30:48 rain reset 1048.4 -> 1047.1
2015-08-25 13:40:51:pywws.process:2015-08-25 12:30:48 rain reset 1048.4 -> 1047.1
The raw rainfall data from the outside sensors is the total number of times the “see saw” has tipped since the external sensors were last reset (by a battery change, unless you do it quickly).
This number should only ever increase, so the pywws.process module warns of any decrease in the value as it may indicate corrupted data that needs manually correcting.
The logging message includes the UTC time stamp of the problem data to help you find it.
Live data missed¶
2015-10-30 04:49:56:pywws.weatherstation:live_data missed
Sometimes pywws fails to capture live data. This happens if a new data record is identical to the previous one so pywws doesn’t detect a change. This is unlikely to happen if you are receiving wind data properly.
Note that this is just an occasional missing “live” record though, so if it does not happen often you shouldn’t worry too much about it.
“Live log” synchronisation¶
If you run pywws at a high verbosity you may see messages like the following:
jim@gordon:~ $ pywws-livelog -vv ~/weather/data/
10:32:46:pywws.logger:pywws version 18.5.0, build 1541 (d505b50)
10:32:46:pywws.logger:Python version 3.5.3 (default, Jan 19 2017, 14:11:04) [GCC 6.3.0 20170124]
10:32:46:pywws.weatherstation:using pywws.device_libusb1
10:32:48:pywws.calib:Using user calibration
10:32:51:pywws.weatherstation:read period 5
10:32:51:pywws.weatherstation:delay 3, pause 0.5
10:32:52:pywws.weatherstation:status {'rain_overflow': False, 'lost_connection': False}
10:32:52:pywws.weatherstation:delay 3, pause 0.5
10:32:53:pywws.weatherstation:delay 3, pause 0.5
10:32:54:pywws.weatherstation:delay 3, pause 0.5
10:32:54:pywws.weatherstation:delay 3, pause 0.5
10:32:55:pywws.weatherstation:delay 3, pause 0.5
10:32:56:pywws.weatherstation:delay 3, pause 0.5
10:32:56:pywws.weatherstation:delay 3, pause 0.5
10:32:57:pywws.weatherstation:delay 3, pause 0.5
10:32:58:pywws.weatherstation:delay 3, pause 0.5
10:32:58:pywws.weatherstation:delay 3, pause 0.5
10:32:59:pywws.weatherstation:delay 3, pause 0.5
10:33:00:pywws.weatherstation:delay 3, pause 0.5
10:33:00:pywws.weatherstation:delay 3, pause 0.5
10:33:01:pywws.weatherstation:delay 3, pause 0.5
10:33:02:pywws.weatherstation:delay 3, pause 0.5
10:33:02:pywws.weatherstation:live_data new data
10:33:02:pywws.weatherstation:setting sensor clock 14.7283
10:33:02:pywws.regulartasks:doing task sections ['live']
10:33:03:pywws.service.underground:thread started Thread-2
10:33:03:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): rtupdate.wunderground.com
10:33:03:pywws.service.openweathermap:thread started Thread-4
10:33:03:pywws.weatherstation:delay 3, pause 43.4936
10:33:03:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): api.openweathermap.org
10:33:03:requests.packages.urllib3.connectionpool:http://api.openweathermap.org:80 "POST /data/3.0/measurements?appid=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx HTTP/1.1" 204 0
10:33:03:pywws.service.openweathermap:OK
10:33:03:pywws.service.openweathermap:1 record sent
10:33:04:requests.packages.urllib3.connectionpool:https://rtupdate.wunderground.com:443 "GET /weatherstation/updateweatherstation.php?baromin=29.9877&dewptf=63.2&ID=ISURREYE4&action=updateraw&dailyrainin=0&windgustmph=3.13&softwaretype=pywws&winddir=48&tempf=68.5&dateutc=2018-05-31+09%3A33%3A02&realtime=1&windspeedmph=1.57&humidity=83&rtfreq=48&PASSWORD=xxxxxxxxxxxx&rainin=0 HTTP/1.1" 200 8
10:33:04:pywws.service.underground:server response "success"
10:33:04:pywws.service.underground:1 record sent
10:33:47:pywws.weatherstation:delay 3, pause 0.5
10:33:47:pywws.weatherstation:avoid 5.795123949846001
10:33:53:pywws.weatherstation:live_data new data
10:33:53:pywws.regulartasks:doing task sections ['live']
10:33:54:pywws.weatherstation:delay 4, pause 15.0615
10:33:56:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): api.openweathermap.org
10:33:56:requests.packages.urllib3.connectionpool:http://api.openweathermap.org:80 "POST /data/3.0/measurements?appid=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx HTTP/1.1" 204 0
10:33:56:pywws.service.openweathermap:OK
10:33:56:pywws.service.openweathermap:1 record sent
10:33:56:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): rtupdate.wunderground.com
10:33:57:requests.packages.urllib3.connectionpool:https://rtupdate.wunderground.com:443 "GET /weatherstation/updateweatherstation.php?baromin=29.9877&dewptf=63.2&ID=ISURREYE4&action=updateraw&dailyrainin=0&windgustmph=2.24&softwaretype=pywws&winddir=38&tempf=68.5&dateutc=2018-05-31+09%3A33%3A50&realtime=1&windspeedmph=0.67&humidity=83&rtfreq=48&PASSWORD=xxxxxxxxxxxx&rainin=0 HTTP/1.1" 200 8
10:33:57:pywws.service.underground:server response "success"
10:33:57:pywws.service.underground:1 record sent
10:34:09:pywws.weatherstation:delay 4, pause 0.5
10:34:10:pywws.weatherstation:avoid 5.808650196657673
10:34:16:pywws.weatherstation:live_data new ptr: 005470
10:34:16:pywws.process:Generating summary data
10:34:16:pywws.process:daily: 2018-05-31 09:00:00
10:34:16:pywws.process:monthly: 2018-05-01 09:00:00
10:34:16:pywws.regulartasks:doing task sections ['logged']
10:34:16:pywws.service.cwop:thread started Thread-1
10:34:17:pywws.service.wetterarchivde:thread started Thread-3
10:34:17:pywws.service.metoffice:thread started Thread-5
10:34:17:pywws.weatherstation:delay 0, pause 18.132
10:34:17:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): interface.wetterarchiv.de
10:34:17:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): wow.metoffice.gov.uk
10:34:17:requests.packages.urllib3.connectionpool:http://interface.wetterarchiv.de:80 "POST /weather/ HTTP/1.1" 200 36
10:34:17:pywws.service.wetterarchivde:server response "{'status': 'SUCCESS', 'version': '6.0'}"
10:34:17:pywws.service.wetterarchivde:1 record sent
10:34:17:pywws.service.cwop:server software: b'# javAPRSSrvr 4.3.0b17'
10:34:17:pywws.service.cwop:login: "user EW4610 pass -1 vers pywws 18.5.0"
10:34:17:pywws.service.cwop:packet: "EW4610>APRS,TCPIP*:@310934z5121.90N/00015.07W_038/001g002t069r000p000b10155h83.pywws-18.5.0"
10:34:17:pywws.service.cwop:server login ack: b'# logresp EW4610 unverified, server CWOP-2'
10:34:17:pywws.service.cwop:OK
10:34:17:pywws.service.cwop:1 record sent
10:34:17:requests.packages.urllib3.connectionpool:http://wow.metoffice.gov.uk:80 "GET /automaticreading?baromin=29.9877&dewptf=63.2&softwaretype=pywws-18.5.0&dailyrainin=0.0000&windgustmph=2.24&siteid=18837259&winddir=38&tempf=68.5&dateutc=2018-05-31+09%3A34%3A13&windspeedmph=0.67&humidity=83&siteAuthenticationKey=xxxxxx&rainin=0.0000 HTTP/1.1" 200 2
10:34:17:pywws.service.metoffice:OK
10:34:17:pywws.service.metoffice:1 record sent
^C10:34:21:pywws.storage:waiting for thread Thread-2
10:34:21:pywws.storage:waiting for thread Thread-4
10:34:21:pywws.storage:waiting for thread Thread-3
10:34:21:pywws.storage:waiting for thread Thread-1
10:34:21:pywws.storage:waiting for thread Thread-5
10:34:21:pywws.storage:flushing
The “read period” message at 10:32:51 shows that the weather station has the usual 5 minute logging interval.
The “delay 3, pause 0.5” messages show pywws waiting for the station to receive data from the outside sensors.
The delay value is the number of minutes since the station last logged some data.
The pause value is how many seconds pywws will wait before fetching data from the station again.
At 10:33:02 new data is received and the “sensor” clock is set.
After initiating uploads to underground and openweathermap the live logging loop sleeps for 43 seconds.
The uploads happen in their own threads while the main loop is paused.
At 10:33:47 the main loop resumes polling the station.
Almost immediately it pauses for 5.8 seconds to avoid USB activity around the time the station should receive external data.
At 10:34:10 USB activity is avoided when the station is expected to “log” data.
At 10:34:16 the new memory pointer is detected and the logged data is processed.
At 10:34:21 I pressed Ctrl-C to terminate the program.
After shutting down the upload threads any unsaved data is flushed to file and the program finishes.
Crash with traceback¶
Sometimes pywws software crashes. When it does, the log file will often contain a traceback like this:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 | 2018-05-31 11:13:47:pywws.livelog:LIBUSB_ERROR_IO [-1]
Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/pywws-18.5.0-py3.5.egg/pywws/livelog.py", line 70, in live_log
logged_only=(not tasks.has_live_tasks())):
File "/usr/local/lib/python3.5/dist-packages/pywws-18.5.0-py3.5.egg/pywws/logdata.py", line 245, in live_data
for data, ptr, logged in self.ws.live_data(logged_only=logged_only):
File "/usr/local/lib/python3.5/dist-packages/pywws-18.5.0-py3.5.egg/pywws/weatherstation.py", line 546, in live_data
new_ptr = self.current_pos()
File "/usr/local/lib/python3.5/dist-packages/pywws-18.5.0-py3.5.egg/pywws/weatherstation.py", line 704, in current_pos
self._read_fixed_block(0x0020), self.lo_fix_format['current_pos'])
File "/usr/local/lib/python3.5/dist-packages/pywws-18.5.0-py3.5.egg/pywws/weatherstation.py", line 760, in _read_fixed_block
result += self._read_block(mempos)
File "/usr/local/lib/python3.5/dist-packages/pywws-18.5.0-py3.5.egg/pywws/weatherstation.py", line 748, in _read_block
new_block = self.cusb.read_block(ptr)
File "/usr/local/lib/python3.5/dist-packages/pywws-18.5.0-py3.5.egg/pywws/weatherstation.py", line 342, in read_block
if not self.dev.write_data(buf):
File "/usr/local/lib/python3.5/dist-packages/pywws-18.5.0-py3.5.egg/pywws/device_libusb1.py", line 131, in write_data
0x200, 0, str_buf, timeout=50)
File "/usr/lib/python3/dist-packages/usb1/__init__.py", line 1390, in controlWrite
sizeof(data), timeout)
File "/usr/lib/python3/dist-packages/usb1/__init__.py", line 1366, in _controlTransfer
mayRaiseUSBError(result)
File "/usr/lib/python3/dist-packages/usb1/__init__.py", line 133, in mayRaiseUSBError
__raiseUSBError(value)
File "/usr/lib/python3/dist-packages/usb1/__init__.py", line 125, in raiseUSBError
raise __STATUS_TO_EXCEPTION_DICT.get(value, __USBError)(value)
usb1.USBErrorIO: LIBUSB_ERROR_IO [-1]
|
Line 1 shows the exception that caused the crash. Lines 3 to 26 show where in the program the problem happened. Usually the last one is of interest, but the other function calls show how we got there. Line 27 shows the full exception. In this case it’s a USBError raised by the libusb1 library.
Comments or questions? Please subscribe to the pywws mailing list http://groups.google.com/group/pywws and let us know.
Humidity Index (Humidex)¶
Section author: Rodney Persky
Background¶
Using your weather station can be fun, and reporting daily to various weather data sites can be very useful for your neighbours to check out the weather. However, at some point you may want to know how the weather effects your body, and if there is a way to tell when it’s good or not to work outdoors.
Here enters a whole realm of calculations based on energy transferring though walls, and the resistance offered by them. It can be a great learning adventure, and can save you a great deal of money, finding out how energy moves around.
Introduction¶
Humidex is a tool to determine how an individuals body will react to the combination of Wind, Humidity and Temperature. The background of which is a heat balance across from your midriff to your skin, and is complimentary to ISO 7243 “Hot Environments - Estimation of the heat stress on working man”. A few important notes,
- These indices are based off a number of assumptions which may result in over or under-estimation of your bodies internal state
- A personal weather station may not show the correct conditions, and may have an over or under estimation of the humidity, wind or temperature
- Clothing choices effect the personal fatigue and the bodies ability to reject heat, a low Humidity Index doesn’t mean you can wear anything
- An individuals fitness will effect their bodies response to changing temperature, and experience will aid in knowing when to stop working
- The duration of activities that can be performed requires knowledge on the intensity, which cannot be represented though this index
Assumptions¶
There are a number of assumptions that have been made to make this work which will directly affect its useability. These assumptions however have not been made available from Environment Canada, who are the original developers of the Humidex used in the PYWWS function cadhumidex. It is safe enough however to say that the following would have been some assumptions:
- Clothing type, thickness
- Skin area exposed to free air
- Sun exposure
However, there are a number of assumptions pywws needs to make in calculating the Humidex:
- The humidity, wind and temperature readings are correct
There are also assumptions about the individuals body type and ‘acclimatisation’
- An individuals fitness will effect their bodies response to changing temperature
- Experience will aid in knowing when to stop working
Important References¶
Being Prepared for Summer - http://www.ec.gc.ca/meteo-weather/default.asp?lang=En&n=86C0425B-1
How to use¶
The function is descriptively named cadhumidex and has the parameters temperature and humidity, essentially the function operates as a conversion and can be used in a straightforward manner:
<ycalc>cadhumidex(data['temp_out'],data['hum_out'])</ycalc>
Putting it together, I have added colours that follow basic warning colors and the different brackets to produce a decent graph:
<?xml version="1.0" encoding="ISO-8859-1"?>
<graph>
<title>Humidity Index, Bands indicate apparent discomfort in standard on-site working conditions</title>
<size>1820, 1024</size>
<duration>hours=48</duration>
<xtics>2</xtics>
<xformat>%H%M</xformat>
<dateformat></dateformat>
<plot>
<yrange>29, 55</yrange>
<y2range>29, 55</y2range>
<ylabel></ylabel>
<y2label>Humidex</y2label>
<source>raw</source>
<subplot>
<title>Humidex</title>
<ycalc>cadhumidex(data['temp_out'],data['hum_out'])</ycalc>
<colour>4</colour>
<axes>x1y2</axes>
</subplot>
<subplot>
<title>HI > 54, Heat Stroke Probable</title>
<ycalc>54</ycalc>
<axes>x1y2</axes>
<colour>1</colour>
</subplot>
<subplot>
<title>HI > 45, Dangerous</title>
<ycalc>45</ycalc>
<axes>x1y2</axes>
<colour>8</colour>
</subplot>
<subplot>
<title>HI > 40, Intense</title>
<ycalc>40</ycalc>
<axes>x1y2</axes>
<colour>6</colour>
</subplot>
<subplot>
<title>HI > 35, Evident</title>
<ycalc>35</ycalc>
<axes>x1y2</axes>
<colour>2</colour>
</subplot>
<subplot>
<title>HI > 30, Noticeable</title>
<ycalc>30</ycalc>
<axes>x1y2</axes>
<colour>3</colour>
</subplot>
</plot>
</graph>
Not running the latest update?¶
If you are not running the latest update / do not want to, then this can be implemented using a longer <ycalc> as follows:
<ycalc>data['temp_out']+0.555*(6.112*10**(7.5*data['temp_out']/(237.7+data['temp_out']))*data['hum_out']/100-10)</ycalc>
Comments or questions? Please subscribe to the pywws mailing list http://groups.google.com/group/pywws and let us know.
Comments or questions? Please subscribe to the pywws mailing list http://groups.google.com/group/pywws and let us know.
Python programs and modules¶
Set up and configure pywws¶
pywws.testweatherstation |
Test connection to weather station. |
pywws.setweatherstation |
Set some weather station parameters. |
pywws.version |
Display pywws version information. |
pywws.reprocess |
Regenerate hourly and daily summary data. |
pywws.usbtest |
Test quality of USB connection to weather station |
pywws.mergeewdata |
Convert EasyWeather.dat data to pywws format |
Get data and process it¶
pywws.hourly |
Get weather data, process it, prepare graphs & text files and upload to a web site. |
pywws.livelog |
Get weather data, store it, and process it. |
pywws.livelogdaemon |
Run ‘live logging’ as a UNIX daemon. |
Upload data to online “services”¶
pywws.service |
Base classes for “service” uploaders. |
pywws.service.ftp |
Upload files to a web server by FTP. |
pywws.service.sftp |
Upload files to a web server by SFTP. |
pywws.service.copy |
Copy files to another directory. |
pywws.service.cwop |
Upload weather data to Citizen Weather Observer Program. |
pywws.service.metoffice |
Upload weather data to UK Met Office “WOW”. |
pywws.service.mqtt |
Upload weather data to MQTT message broker. |
pywws.service.openweathermap |
Upload weather data to Open Weather Map. |
pywws.service.pwsweather |
Upload weather data to PWS Weather. |
pywws.service.temperaturnu |
Upload current temperature to temperatur.nu. |
pywws.service.underground |
Upload data to Weather Underground. |
pywws.service.weathercloud |
Upload data to WeatherCloud. |
pywws.service.wetterarchivde |
Upload weather data to wetter.com. |
pywws.service.windy |
Upload data to Windy. |
pywws.service.twitter |
Post messages to Twitter. |
pywws.service.mastodon |
Post messages to Mastodon. |
“Internal” modules¶
pywws.regulartasks |
Routines to perform common tasks such as plotting gaphs or uploading files. |
pywws.logdata |
Save weather station history to file |
pywws.process |
Generate hourly, daily & monthly summaries of raw weather station data . |
pywws.calib |
Calibrate raw weather station data |
pywws.plot |
Plot graphs of weather data according to an XML recipe |
pywws.windrose |
Plot a “wind rose” |
pywws.template |
Create text data file based on a template |
pywws.forecast |
Predict future weather using recent data |
pywws.weatherstation |
Get data from WH1080/WH3080 compatible weather stations. |
pywws.device_libusb1 |
Low level USB interface to weather station, using python-libusb1. |
pywws.device_pyusb1 |
Low level USB interface to weather station, using PyUSB v1.0. |
pywws.device_pyusb |
Low level USB interface to weather station, using PyUSB v0.4. |
pywws.device_ctypes_hidapi |
Low level USB interface to weather station, using ctypes to access hidapi. |
pywws.device_cython_hidapi |
Low level USB interface to weather station, using cython-hidapi. |
pywws.storage |
Store parameters in easy to access files, and access backend data |
pywws.filedata |
Store weather data in easy to access files |
pywws.sqlite3data |
Store weather data in an SQLite3 database file. |
pywws.timezone |
Provide a couple of datetime.tzinfo compatible objects representing local time and UTC. |
pywws.localisation |
Provide translations of strings into local language |
pywws.conversions |
conversions.py - a set of functions to convert pywws native units (Centigrade, mm, m/s, hPa) to other popular units |
pywws.logger |
Configure Python logging system |
pywws.constants |
Bits of data used in several places. |
Comments or questions? Please subscribe to the pywws mailing list http://groups.google.com/group/pywws and let us know.
GNU General Public License¶
GNU GENERAL PUBLIC LICENSE
Version 2, June 1991
Copyright (C) 1989, 1991 Free Software Foundation, Inc.
51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
Everyone is permitted to copy and distribute verbatim copies
of this license document, but changing it is not allowed.
Preamble
The licenses for most software are designed to take away your
freedom to share and change it. By contrast, the GNU General Public
License is intended to guarantee your freedom to share and change free
software--to make sure the software is free for all its users. This
General Public License applies to most of the Free Software
Foundation's software and to any other program whose authors commit to
using it. (Some other Free Software Foundation software is covered by
the GNU Library General Public License instead.) You can apply it to
your programs, too.
When we speak of free software, we are referring to freedom, not
price. Our General Public Licenses are designed to make sure that you
have the freedom to distribute copies of free software (and charge for
this service if you wish), that you receive source code or can get it
if you want it, that you can change the software or use pieces of it
in new free programs; and that you know you can do these things.
To protect your rights, we need to make restrictions that forbid
anyone to deny you these rights or to ask you to surrender the rights.
These restrictions translate to certain responsibilities for you if you
distribute copies of the software, or if you modify it.
For example, if you distribute copies of such a program, whether
gratis or for a fee, you must give the recipients all the rights that
you have. You must make sure that they, too, receive or can get the
source code. And you must show them these terms so they know their
rights.
We protect your rights with two steps: (1) copyright the software, and
(2) offer you this license which gives you legal permission to copy,
distribute and/or modify the software.
Also, for each author's protection and ours, we want to make certain
that everyone understands that there is no warranty for this free
software. If the software is modified by someone else and passed on, we
want its recipients to know that what they have is not the original, so
that any problems introduced by others will not reflect on the original
authors' reputations.
Finally, any free program is threatened constantly by software
patents. We wish to avoid the danger that redistributors of a free
program will individually obtain patent licenses, in effect making the
program proprietary. To prevent this, we have made it clear that any
patent must be licensed for everyone's free use or not licensed at all.
The precise terms and conditions for copying, distribution and
modification follow.
GNU GENERAL PUBLIC LICENSE
TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION
0. This License applies to any program or other work which contains
a notice placed by the copyright holder saying it may be distributed
under the terms of this General Public License. The "Program", below,
refers to any such program or work, and a "work based on the Program"
means either the Program or any derivative work under copyright law:
that is to say, a work containing the Program or a portion of it,
either verbatim or with modifications and/or translated into another
language. (Hereinafter, translation is included without limitation in
the term "modification".) Each licensee is addressed as "you".
Activities other than copying, distribution and modification are not
covered by this License; they are outside its scope. The act of
running the Program is not restricted, and the output from the Program
is covered only if its contents constitute a work based on the
Program (independent of having been made by running the Program).
Whether that is true depends on what the Program does.
1. You may copy and distribute verbatim copies of the Program's
source code as you receive it, in any medium, provided that you
conspicuously and appropriately publish on each copy an appropriate
copyright notice and disclaimer of warranty; keep intact all the
notices that refer to this License and to the absence of any warranty;
and give any other recipients of the Program a copy of this License
along with the Program.
You may charge a fee for the physical act of transferring a copy, and
you may at your option offer warranty protection in exchange for a fee.
2. You may modify your copy or copies of the Program or any portion
of it, thus forming a work based on the Program, and copy and
distribute such modifications or work under the terms of Section 1
above, provided that you also meet all of these conditions:
a) You must cause the modified files to carry prominent notices
stating that you changed the files and the date of any change.
b) You must cause any work that you distribute or publish, that in
whole or in part contains or is derived from the Program or any
part thereof, to be licensed as a whole at no charge to all third
parties under the terms of this License.
c) If the modified program normally reads commands interactively
when run, you must cause it, when started running for such
interactive use in the most ordinary way, to print or display an
announcement including an appropriate copyright notice and a
notice that there is no warranty (or else, saying that you provide
a warranty) and that users may redistribute the program under
these conditions, and telling the user how to view a copy of this
License. (Exception: if the Program itself is interactive but
does not normally print such an announcement, your work based on
the Program is not required to print an announcement.)
These requirements apply to the modified work as a whole. If
identifiable sections of that work are not derived from the Program,
and can be reasonably considered independent and separate works in
themselves, then this License, and its terms, do not apply to those
sections when you distribute them as separate works. But when you
distribute the same sections as part of a whole which is a work based
on the Program, the distribution of the whole must be on the terms of
this License, whose permissions for other licensees extend to the
entire whole, and thus to each and every part regardless of who wrote it.
Thus, it is not the intent of this section to claim rights or contest
your rights to work written entirely by you; rather, the intent is to
exercise the right to control the distribution of derivative or
collective works based on the Program.
In addition, mere aggregation of another work not based on the Program
with the Program (or with a work based on the Program) on a volume of
a storage or distribution medium does not bring the other work under
the scope of this License.
3. You may copy and distribute the Program (or a work based on it,
under Section 2) in object code or executable form under the terms of
Sections 1 and 2 above provided that you also do one of the following:
a) Accompany it with the complete corresponding machine-readable
source code, which must be distributed under the terms of Sections
1 and 2 above on a medium customarily used for software interchange; or,
b) Accompany it with a written offer, valid for at least three
years, to give any third party, for a charge no more than your
cost of physically performing source distribution, a complete
machine-readable copy of the corresponding source code, to be
distributed under the terms of Sections 1 and 2 above on a medium
customarily used for software interchange; or,
c) Accompany it with the information you received as to the offer
to distribute corresponding source code. (This alternative is
allowed only for noncommercial distribution and only if you
received the program in object code or executable form with such
an offer, in accord with Subsection b above.)
The source code for a work means the preferred form of the work for
making modifications to it. For an executable work, complete source
code means all the source code for all modules it contains, plus any
associated interface definition files, plus the scripts used to
control compilation and installation of the executable. However, as a
special exception, the source code distributed need not include
anything that is normally distributed (in either source or binary
form) with the major components (compiler, kernel, and so on) of the
operating system on which the executable runs, unless that component
itself accompanies the executable.
If distribution of executable or object code is made by offering
access to copy from a designated place, then offering equivalent
access to copy the source code from the same place counts as
distribution of the source code, even though third parties are not
compelled to copy the source along with the object code.
4. You may not copy, modify, sublicense, or distribute the Program
except as expressly provided under this License. Any attempt
otherwise to copy, modify, sublicense or distribute the Program is
void, and will automatically terminate your rights under this License.
However, parties who have received copies, or rights, from you under
this License will not have their licenses terminated so long as such
parties remain in full compliance.
5. You are not required to accept this License, since you have not
signed it. However, nothing else grants you permission to modify or
distribute the Program or its derivative works. These actions are
prohibited by law if you do not accept this License. Therefore, by
modifying or distributing the Program (or any work based on the
Program), you indicate your acceptance of this License to do so, and
all its terms and conditions for copying, distributing or modifying
the Program or works based on it.
6. Each time you redistribute the Program (or any work based on the
Program), the recipient automatically receives a license from the
original licensor to copy, distribute or modify the Program subject to
these terms and conditions. You may not impose any further
restrictions on the recipients' exercise of the rights granted herein.
You are not responsible for enforcing compliance by third parties to
this License.
7. If, as a consequence of a court judgment or allegation of patent
infringement or for any other reason (not limited to patent issues),
conditions are imposed on you (whether by court order, agreement or
otherwise) that contradict the conditions of this License, they do not
excuse you from the conditions of this License. If you cannot
distribute so as to satisfy simultaneously your obligations under this
License and any other pertinent obligations, then as a consequence you
may not distribute the Program at all. For example, if a patent
license would not permit royalty-free redistribution of the Program by
all those who receive copies directly or indirectly through you, then
the only way you could satisfy both it and this License would be to
refrain entirely from distribution of the Program.
If any portion of this section is held invalid or unenforceable under
any particular circumstance, the balance of the section is intended to
apply and the section as a whole is intended to apply in other
circumstances.
It is not the purpose of this section to induce you to infringe any
patents or other property right claims or to contest validity of any
such claims; this section has the sole purpose of protecting the
integrity of the free software distribution system, which is
implemented by public license practices. Many people have made
generous contributions to the wide range of software distributed
through that system in reliance on consistent application of that
system; it is up to the author/donor to decide if he or she is willing
to distribute software through any other system and a licensee cannot
impose that choice.
This section is intended to make thoroughly clear what is believed to
be a consequence of the rest of this License.
8. If the distribution and/or use of the Program is restricted in
certain countries either by patents or by copyrighted interfaces, the
original copyright holder who places the Program under this License
may add an explicit geographical distribution limitation excluding
those countries, so that distribution is permitted only in or among
countries not thus excluded. In such case, this License incorporates
the limitation as if written in the body of this License.
9. The Free Software Foundation may publish revised and/or new versions
of the General Public License from time to time. Such new versions will
be similar in spirit to the present version, but may differ in detail to
address new problems or concerns.
Each version is given a distinguishing version number. If the Program
specifies a version number of this License which applies to it and "any
later version", you have the option of following the terms and conditions
either of that version or of any later version published by the Free
Software Foundation. If the Program does not specify a version number of
this License, you may choose any version ever published by the Free Software
Foundation.
10. If you wish to incorporate parts of the Program into other free
programs whose distribution conditions are different, write to the author
to ask for permission. For software which is copyrighted by the Free
Software Foundation, write to the Free Software Foundation; we sometimes
make exceptions for this. Our decision will be guided by the two goals
of preserving the free status of all derivatives of our free software and
of promoting the sharing and reuse of software generally.
NO WARRANTY
11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY
FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN
OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES
PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED
OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS
TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE
PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING,
REPAIR OR CORRECTION.
12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR
REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES,
INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING
OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED
TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY
YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER
PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE
POSSIBILITY OF SUCH DAMAGES.
END OF TERMS AND CONDITIONS
How to Apply These Terms to Your New Programs
If you develop a new program, and you want it to be of the greatest
possible use to the public, the best way to achieve this is to make it
free software which everyone can redistribute and change under these terms.
To do so, attach the following notices to the program. It is safest
to attach them to the start of each source file to most effectively
convey the exclusion of warranty; and each file should have at least
the "copyright" line and a pointer to where the full notice is found.
<one line to give the program's name and a brief idea of what it does.>
Copyright (C) <year> <name of author>
This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 2 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program; if not, write to the Free Software
Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
Also add information on how to contact you by electronic and paper mail.
If the program is interactive, make it output a short notice like this
when it starts in an interactive mode:
Gnomovision version 69, Copyright (C) year name of author
Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'.
This is free software, and you are welcome to redistribute it
under certain conditions; type `show c' for details.
The hypothetical commands `show w' and `show c' should show the appropriate
parts of the General Public License. Of course, the commands you use may
be called something other than `show w' and `show c'; they could even be
mouse-clicks or menu items--whatever suits your program.
You should also get your employer (if you work as a programmer) or your
school, if any, to sign a "copyright disclaimer" for the program, if
necessary. Here is a sample; alter the names:
Yoyodyne, Inc., hereby disclaims all copyright interest in the program
`Gnomovision' (which makes passes at compilers) written by James Hacker.
<signature of Ty Coon>, 1 April 1989
Ty Coon, President of Vice
This General Public License does not permit incorporating your program into
proprietary programs. If your program is a subroutine library, you may
consider it more useful to permit linking proprietary applications with the
library. If this is what you want to do, use the GNU Library General
Public License instead of this License.
Comments or questions? Please subscribe to the pywws mailing list http://groups.google.com/group/pywws and let us know.
pywws Contributors¶
The copywright to pywws and its documentation is jointly held by the following contributors.
Developers
----------
Jim Easterbrook jim@jim-easterbrook.me.uk
x2q
3v1n0
Robin Kearney robin@kearney.co.uk
Rod Persky
Morten Høybye Frederiksen morten@mfd-consult.dk
Simon Josefsson simon@josefsson.org
Matthew Hilton matthilton2005@gmail.com
Sabine Tobolka oe1yvw@gmail.com
Markus Birth markus@birth-online.de
Chris Ramsay chris@ramsay-family.net
Christian Benke benkokakao@gmail.com
Ian Wilkinson null@sgtwilko.f9.co.uk
Tim Richardson tim@potton.me.uk
Richard Truran ashenshugar@outlook.com
Mark Jarvis jarvism@thisaddressdoesnotexist.co.uk
Vladimir Michl vladimir.michl.vt@gmail.com
Translators
-----------
Edoardo edoardo69@hotmail.it
Jacques Desroches metelsto@gmail.com
Sunshades joacim@ahlstrand.info
Johabu johabu96@yahoo.de
karte2@gmail.com
Kyle Gordon kyle@lodge.glasgownet.com>
Πέτρος nouvakis@sch.gr
Ramiro ramiro.sanchez@telefonica.net
Rick Sulman rick@sulman.org
Pyttsen weather@spacelab.se
Tech2304 tech2304@gmail.com
Pablo Vera pablo.vera82@gmail.com
Contributing to pywws¶
If you would like to add a feature to pywws (or fix a problem with it) then please do. Open source software thrives when its users become active contributors. The process is quite simple:
Join GitHub - it’s free.
Fork the pywws repo - see Fork a Repo for help.
Clone your fork to a computer you can use to develop your new feature.
Use git to commit changes as you make them and push the changes to your fork of pywws.
Please add a signed-off-by line to your commits which certify your developer certificate of origin (see below). For example, if your name is “John Smith”, and your email address is “jsmith@example.com”, just include the following line at the bottom of your commit messages:
Signed-off-by: John Smith <jsmith@example.com>
You should be able to do this automatically by using the
-soption on yourgit commitcommands.Add your name and email to the
src/contributors/contributors.txtfile. Don’t forget the-soption when you commit this change.Test your changes!
When everything’s working as you expect, submit a Pull Request.
Developer Certificate of Origin¶
Including a signed-off-by line in your commits indicates that you certify the following:
Developer Certificate of Origin
Version 1.1
Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
660 York Street, Suite 102,
San Francisco, CA 94110 USA
Everyone is permitted to copy and distribute verbatim copies of this
license document, but changing it is not allowed.
Developer's Certificate of Origin 1.1
By making a contribution to this project, I certify that:
(a) The contribution was created in whole or in part by me and I
have the right to submit it under the open source license
indicated in the file; or
(b) The contribution is based upon previous work that, to the best
of my knowledge, is covered under an appropriate open source
license and I have the right under that license to submit that
work with modifications, whether created in whole or in part
by me, under the same open source license (unless I am
permitted to submit under a different license), as indicated
in the file; or
(c) The contribution was provided directly to me by some other
person who certified (a), (b) or (c) and I have not modified
it.
(d) I understand and agree that this project and the contribution
are public and that a record of the contribution (including all
personal information I submit with it, including my sign-off) is
maintained indefinitely and may be redistributed consistent with
this project or the open source license(s) involved.
Clauses (a), (b) and (c) reassure pywws users that the project will remain open source well in to the future. Clause (d) reminds you that your contributions will be publicly available, and you do not have the right to withdraw them in future.
Comments or questions? Please subscribe to the pywws mailing list http://groups.google.com/group/pywws and let us know.
Change Log¶
pywws - Python software for USB Wireless Weather Stations
http://github.com/jim-easterbrook/pywws
Copyright (C) 2008-21 pywws contributors
This program is free software; you can redistribute it and/or
modify it under the terms of the GNU General Public License
as published by the Free Software Foundation; either version 2
of the License, or (at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program; if not, write to the Free Software
Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
Changes in v21.4.0:
1/ Fixed problems with timezones ready for next release of tzlocal.
2/ Fixed rare problem with 'saved_date' being 'None'.
Changes in v21.3.0:
1/ Added -d option to pywws-setweatherstation. This is useful if your
station sets loads of alarms after a reset.
2/ Updated pywws.timezone to cope with changes in Python 3.6.
3/ Various bug fixes and minor improvements.
Changes in v20.1.0:
1/ Added a "Windy" service uploader module.
2/ Various bug fixes and minor improvements.
Changes in v19.7.1:
1/ Fix file data iteration bug introduced in v19.7.0.
Changes in v19.7.0:
1/ Update default template_txt for service.mqtt to provide wider range of
values out of the box. The published format is not backwards compatible,
but this only affects new users.
2/ Improved reliability with "solar" stations when lost contact with solar
sensors.
3/ Some file data errors should now be detected and a log message sent.
Changes in v19.4.0:
1/ Can optionally use SQLite3 data store instead of CSV files.
2/ Improve weathercloud service uploader.
3/ Cope slightly better with stations that aren't logging data.
Changes in v18.10.1:
1/ Fix bug when running a service directly (in Python 2 only) introduced
in v18.9.0.
Changes in v18.10.0:
1/ Fix template loop bug (in Python 2 only) introduced in v18.9.0.
Changes in v18.9.0:
1/ Major changes to ftp uploads. New ftp/sftp/copy services do the
uploading. pywws should make the required changes to weather.ini
automatically, but users should check after running v18.9.0 for the
first time.
2/ New services to post to Mastodon and WeatherCloud.
3/ MQTT service can now use TLS.
4/ Various bug fixes and minor improvements.
Changes in v18.6.3:
1/ Restore Python source text encoding comments needed by some Python 2
versions.
2/ Add example AWS uploader module from Chris Ramsay.
Changes in v18.6.2:
1/ Fix duplicate data bug in live logging.
Changes in v18.6.1:
1/ Fix bug in "fixed block" date parsing.
Changes in v18.6.0:
1/ Substantial rewrite of much of pywws. It is now more "standard" in many
ways, such as names of modules, and is compatible with Python 2.7 and 3.
2/ New "service" uploaders. "underground_rf" and "cwop_ham" have been
merged in to "underground" and "cwop".
3/ "Day end hour" can now vary with daylight savings time.
4/ Improved reliability with "3080" class stations.
5/ Various other bug fixes and minor improvements.
Changes in v18.04.1:
1/ Fix bug in utc <-> local time conversions.
Changes in v18.04.0:
1/ Now works with broken weather stations that have stopped logging data.
(Although there may be more gaps in the data.)
2/ Cope better with missing wind direction data.
3/ Use HTTPS for Weather Underground uploads.
4/ Other minor bug fixes.
Changes in v17.11.0:
1/ Allow MQTT service without password.
2/ Allow SFTP uploads with public key.
3/ Increase Twitter character limit from 140 to 280.
4/ Various other bug fixes and minor improvements.
Changes in v16.12.0:
1/ Added "candlestick" plot type.
2/ Added cloud base calculation function.
3/ Various other bug fixes and minor improvements.
Changes in v16.08.0:
1/ Fix Python 2.5 incompatibilities.
2/ Fix python-twitter v3 tweet length problem.
Changes in v16.07.1:
1/ Further changes to handle UK Met Office server quirks.
Changes in v16.07.0:
1/ Fix bug with UK Met Office uploads server change.
2/ Allow user commands in wind roses.
3/ Various other bug fixes and minor improvements.
Changes in v15.12.0:
1/ Fix bug with Twitter messages being excessively truncated.
2/ Improve handling of utf-8 encoded templates.
3/ Improved plots and wind roses with 'pngcairo' "terminal".
4/ Various bug fixes and minor improvements.
Changes in v15.11.0:
1/ Add Russian translation of program text.
2/ Improved documentation.
3/ Various bug fixes and minor improvements.
Changes in v15.07.0:
1/ Can include multiple media in Twitter messages.
2/ Attempt to fix bug in wind rose axes labels.
3/ Enable inclusion of time & date in wind rose title.
4/ Various bug fixes and minor improvements.
Changes in v15.01.0:
1/ Added 'MQTT' service.
2/ Added another USB library option.
3/ Improved Python 3 compatibility.
4/ Various bug fixes and minor improvements.
Changes in v14.12.0:
1/ Updated temperatur.nu and wetterarchiv.de service details to suit new
APIs.
Changes in v14.06.1:
1/ Revised version numbering scheme.
2/ Compiled documentation no longer included in releases.
3/ Can partially specify start & stop date/time in graphs, e.g. to start a
plot at midnight, no matter when it is plotted.
Changes in v14.06:
1/ Can now send images to Twitter.
2/ Periodic tasks can be specified with a cron style syntax.
3/ Added wind direction filter for use in graphs or user calibration
modules.
4/ Wind direction is now stored as a float. Old templates that use the
wind_dir_text array will need updating, probably to use the
winddir_text() function.
5/ Started using "Transifex" to host translations. Changed tools and
procedures to create new translations.
6/ Improved USB hangup avoidance strategy for stations with large clock
drift figures.
7/ Various bug fixes and minor improvements.
Changes in v14.05:
1/ Rearranged package layout, moving examples and documentation.
2/ Added 'entry point' auto-generated commands for some modules.
3/ Added verbose output option to pywws-version command.
4/ Various bug fixes and minor improvements.
Changes in v14.03:
1/ Extracts additional status from 'wind_dir' byte. You must run
pywws-reprocess.py with the -u option after upgrading from any previous
version.
2/ Added Citizen Weather Observer Program to available 'services'.
3/ Improved asynchronous upload task queuing.
4/ Various bug fixes and minor improvements.
Changes in v14.02:
1/ Improved time zone handling, including non whole hour time zones.
2/ New 'frequent writes' config option.
3/ Improved 'live log' sync, particularly with 3080 type stations.
4/ Record recent memory pointer to improve detection of gaps in data.
5/ Various bug fixes and minor improvements.
Changes in v13.12:
1/ Changed API of user calibration module.
2/ Can use python-twitter *or* tweepy library.
3/ Added a script to run live logging as a UNIX daemon process.
4/ Changed data store to use separate read and write caches.
5/ Various bug fixes and minor improvements.
Changes in v13.10:
1/ Changed Twitter library from tweepy to python-twitter.
2/ Added ability to do uploads asynchronously.
3/ Rearranged and improved documentation.
4/ Various bug fixes and minor improvements.
Changes in v13.06:
1/ Substantially rearranged directories, getting rid of 'code' and 'code3'.
2/ Removed 'makefile' - everything is now done via 'setup.py'.
3/ Removed 'RunModule.py' - use 'python -m pywws.module' now.
4/ Separated storage of config (weather.ini) and status (status.ini).
5/ Replaced toservice.py "rapid fire" mode with a separate config file for
Weather Underground rapid fire.
6/ Added 2 more low-level USB access modules.
7/ Various bug fixes and minor improvements.
Changes in v13.03:
1/ Added 'rain days' to monthly data. (Reprocess required when upgrading.)
2/ Extended template syntax to include comments.
3/ Added 'humidity index' function.
4/ Added French translation of documentation.
5/ Reduced frequency of saving data files.
6/ Various bug fixes.
Changes in v12.12:
1/ Added support for Python 3.
2/ Added French documentation translation.
3/ Used 'binary search' to speed up data access.
4/ Various bug fixes.
Changes in v12.11:
1/ Moved development from Google code to GitHub.
2/ Made software attempt to avoid USB activity at times when it is assumed
the weather station might be writing to its memory. This might solve
the USB lockup problem, but it's too early to tell.
Changes in v12.10:
1/ Added a 'winddir_text' function for use in templates.
2/ Added <ytics> and <y2tics> options to graph plots.
3/ Various bug fixes.
Changes in v12.07:
1/ Added Open Weather Map to the services.
2/ Fixed problem with Weather Underground uploads that started on 1st June.
3/ Various bug fixes and software structure improvements.
Changes in v12.05:
1/ Made 'fixed block' data available to template calculations.
2/ Fixed buggy auto-detection of 3080 weather stations.
3/ Added a function to generate the Zambretti forecast code letter.
4/ Added a program to test USB communication reliablility.
5/ Various bug fixes and software structure improvements.
Changes in v12.02:
1/ Separated out low level USB communications to enable use of different
libraries. Now works on recent versions of Mac OS.
2/ Added humidity, pressure & wind data to summary data.
3/ Merged Weather Underground and UK Met Office uploaders into one combined
module. Added more 'service' uploaders.
4/ Various bug fixes and software structure improvements.
Changes in v11.10:
1/ Complete restructuring of documentation.
2/ Added a user defined 'calibration' process.
3/ Sets 'locale' according to language setting.
4/ Added ability to upload to UK Met Office 'WOW'.
5/ Various bug fixes and software structure improvements.
6/ New language files: French, Danish.
Changes in v11.05:
1/ Added support for '3080' family stations that have illuminance and
UV sensors.
2/ Broadened the range of tasks that can be done with 'live' data.
3/ Various bug fixes and software structure improvements.
Changes in v11.02:
1/ Various bug fixes and software structure improvements.
2/ Improved wind direction averaging.
3/ Added conversion functions for common things such as C to F.
4/ Added a YoWindow module.
5/ Improved Zambretti forecaster.
Changes in v10.12:
1/ Various bug fixes and software structure improvements.
2/ Added a 'goto' instruction to Template.py.
3/ Added a 'Zambretti' forecast function to Template.py. This should
be treated as an experiment, and not relied upon for accuracy.
Changes in v10.10:
1/ Added 'catchup' mode to ToUnderground.py.
2/ Created 'Tasks.py' to handle common tasks.
3/ Made better use of Python's logger for info and error
messages.
4/ Changed over from 'python-twitter' to 'tweepy' for Twitter
access. Twitter authorisation using OAuth now works.
5/ Added 'LiveLog.py' live logging program.
6/ Added 'SetWeatherStation.py' to do some configuration of weather
station. No longer need EasyWeather to set logging interval!
7/ Added 'Rapid Fire' ability to ToUnderground.py.
8/ Added plain text versions of HTML documentation.
9/ Many bug fixes and minor improvements.
Changes in v10.08:
1/ Added internal temperature to daily and monthly summaries.
Run Reprocess.py when upgrading from earlier versions.
2/ Added 'prevdata' to Template.py. Allows calculations that
compare values from different times.
3/ Made 'pressure_offset' available to calculations in Plot.py
and Template.py. This is only useful when using 'raw' data.
4/ Improved synchronisation to weather station's clock when
fetching stored data.
Changes in v10.06:
1/ Improved localisation code.
2/ Minor bug fixes.
3/ Added Y axis label angle control to plots.
Changes in v10.04:
1/ Changed version numbering to year.month.
2/ Allowed "upload" to a local directory instead of ftp site.
3/ Added "calc" option to text templates (Template.py).
4/ Added -v / --verbose option to Hourly.py to allow silent operation.
5/ Added internationalisation / localisation of some strings.
6/ Made 'raw' data available to text templates.
7/ Added ability to upload to Weather Underground.
8/ Added dual axis and cumulative graph capability.
Changes in v0.9:
1/ Added lowest daytime max and highest nighttime min temperatures
to monthly data.
2/ Added average temperature to daily and monthly data.
3/ Added 'terminal' element to Plot.py templates for greater control
over output appearance.
4/ Added 'command' element to Plot.py templates for even more
control, for advanced users.
5/ Added secure upload option.
6/ Minor speed improvements.
Changes in v0.8:
1/ Added meteorological day end hour user preference
2/ Attempts at Windows compatibility
3/ Corrected decoding of wind data at speeds over 25.5 m/s
4/ Improved speed with new data caching strategy
Changes in v0.7:
1/ Several bug fixes, mostly around new weather stations with not
much data
2/ Added min & max temperature extremes to monthly data
3/ Added template and workspace directory locations to weather.ini
4/ Increased versatility of Plot.py with layout and title elements
Changes in v0.6:
1/ Added monthly data
2/ Changed 'pressure' to 'abs_pressure' or 'rel_pressure'
Changes in v0.5:
1/ Small bug fixes.
2/ Added start time to daily data
3/ Replaced individual plot programs with XML "recipe" system
Changes in v0.4:
1/ Can post brief messages to Twitter.
2/ Now time zone aware. Uses UTC for data indexing and local time
for graphs and text data files.
Changes in v0.3:
1/ Now uses templates to generate text data
2/ Added 28 day plot
3/ Minor efficiency improvements
4/ Improved documentation
Changes in v0.2:
1/ Now uses Python csv library to read and write data
2/ Creates hourly and daily summary files
3/ Includes rain data in graphs
Comments or questions? Please subscribe to the pywws mailing list http://groups.google.com/group/pywws and let us know.
Indices and tables¶
Credits¶
I would not have been able to get any information from the weather station without access to the source of Michael Pendec’s “wwsr” program. I am also indebted to Dave Wells for decoding the weather station’s “fixed block data”.
Last of all, a big thank you to all the pywws users who have helped with questions and suggestions, and especially to those who have translated pywws and its documentation into other languages.
Legalese¶
This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.
This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
Comments or questions? Please subscribe to the pywws mailing list http://groups.google.com/group/pywws and let us know.