How to set up ‘hourly’ logging with pywws

Introduction

There are two quite different modes of operation with pywws. Traditionally pywws.hourly would be run at regular intervals (usually an hour) from cron. This is suitable for fairly static websites, but more frequent updates can be useful for sites such as Weather Underground (http://www.wunderground.com/). The newer pywws.livelog program runs continuously and can upload data every 48 seconds.

Note that although this document (and the program name) refers to ‘hourly’ logging, you can run pywws.hourly as often or as infrequently as you like, but don’t try to run it more often than double your logging interval. For example, if your logging interval is 10 minutes, don’t run pywws.hourly more often than every 20 minutes.

Getting started

First of all, you need to install pywws and make sure it can get data from your weather station. See How to get started with pywws for details.

Try running pywws.hourly from the command line, with a high level of verbosity so you can see what’s happening. Use the pywws-hourly command to run pywws.hourly:

pywws-hourly -vvv ~/weather/data

(As usual, replace ~/weather/data with your weather data directory.) Within five minutes (assuming you have set a 5 minute logging interval) you should see a ‘live_data new ptr’ message, followed by fetching any new data from the weather station and processing it.

Changed in version 14.04.dev1194: the pywws-hourly command replaced scripts/pywws-hourly.py.

Configuring file locations

Open your weather.ini file with a text editor. You should have a [paths] section similar to the following (where xxx is your user name):

[paths]
work = /tmp/pywws
templates = /home/xxx/weather/templates/
graph_templates = /home/xxx/weather/graph_templates/
modules = /home/xxx/weather/modules/

Edit these to suit your installation and preferences. work is a temporary directory used to store intermediate files. If your computer uses solid state storage, such as a Raspberry Pi’s SD card, it’s a good idea to make this a “RAM disk” to reduce storage “wear”. templates is the directory where you keep your text template files, graph_templates is the directory where you keep your graph template files, and modules is a directory for any extra modules you write. Don’t use the pywws example directories for your templates, as they will get over-written when you upgrade pywws.

Copy your text and graph templates to the appropriate directories. You may find some of the examples provided with pywws useful to get started. The pywws-version -v command should show you where the examples are on your computer.

New in version 14.04.dev1194: the pywws-version command.

Configuring periodic tasks

In weather.ini you should have [live], [logged], [hourly], [12 hourly], and [daily] sections similar to the following:

[logged]
services = []
text = []
plot = []

[hourly]
...

These specify what pywws.hourly should do when it is run. Tasks in the [live] and [logged] sections are done when there is new logged data, tasks in the [hourly] section are done every hour, tasks in the [12 hourly] section are done twice daily and tasks in the [daily] section are done once per day.

The plot and text entries are lists of template files for plots and text files to be processed. The services entry is a list of online weather services to upload data and files to, e.g. 'underground' or ('ftp', '24hrs.txt'). Add the names of your template files and weather services to the appropriate entries, for example:

[logged]
services = ['underground', 'metoffice']
plot = []
text = []

[hourly]
services = [('twitter', 'tweet.txt'),
            ('ftp', '7days.png', '24hrs.png', 'rose_24hrs.png',
                    '24hrs.txt', '6hrs.txt', '7days.txt')]
plot = ['7days.png.xml', '24hrs.png.xml', 'rose_24hrs.png.xml']
text = ['tweet.txt', '24hrs.txt', '6hrs.txt', '7days.txt']

[12 hourly]
services = []
plot = []
text = []

[daily]
services = [('twitter', 'forecast.txt'), ('ftp', '28days.png', 'allmonths.txt')]
plot = ['28days.png.xml']
text = ['forecast.txt', 'allmonths.txt']

Note that the twitter and ftp “services” use files generated by the plot and text items. It’s probably best not to add all of these at once. You could start by uploading one file to your web site, then when that’s working add the remaining web site files. You can add Twitter and other services later on. You can test that things are working by removing the [last update] section from status.ini, then running pywws.hourly again:

pywws-hourly -v ~/weather/data

New in version 14.05.dev1211: [cron name] sections. If you need more flexibility in when tasks are done you can use [cron name] sections. See weather.ini - configuration file format for more detail.

Run as a cron job

Most UNIX/Linux systems have a ‘cron’ daemon that can run programs at certain times, even if you are not logged in to the computer. You edit a ‘crontab’ file to specify what to run and when to run it. For example, to run pywws.hourly every hour, at zero minutes past the hour:

0 * * * *       pywws-hourly /home/xxx/weather/data

This might work, but if it didn’t you probably won’t get any error messages to tell you what went wrong. It’s much better to run a script that runs pywws.hourly and then emails you any output it produces. Here’s a script I’ve used:

#!/bin/sh
#
# weather station logger calling script

export PATH=$PATH:/usr/local/bin

if [ ! -d /home/jim/weather/data/ ]; then
  exit
  fi

log=/var/log/log-weather

pywws-hourly -v /home/jim/weather/data >$log 2>&1

# mail the log file
/home/jim/scripts/email-log.sh $log "weather log"

You’ll need to edit this quite a lot to suit your file locations and so on, but it gives some idea of what to do.

Comments or questions? Please subscribe to the pywws mailing list http://groups.google.com/group/pywws and let us know.