pywws.filedata

Store weather data in easy to access files

Introduction

This module is at the core of pywws file based storage. It stores data on disc, but without the overhead of a full scale database system. I have designed it to run on a small memory machine such as a Raspberry Pi or even a router. To minimise memory usage it only loads one day’s worth of raw data at a time into memory.

From a “user” point of view, the data is accessed as a cross between a list and a dictionary. Each data record is indexed by a datetime.datetime object (dictionary behaviour), but records are stored in order and can be accessed as slices (list behaviour).

For example, to access the hourly data for Christmas day 2009, one might do the following:

from datetime import datetime
import pywws.filedata
hourly = pywws.filedata.HourlyStore('weather_data')
for data in hourly[datetime(2009, 12, 25):datetime(2009, 12, 26)]:
    print(data['idx'], data['temp_out'])

Some more examples of data access:

# get value nearest 9:30 on Christmas day 2008
data[data.nearest(datetime(2008, 12, 25, 9, 30))]
# get entire array, equivalent to data[:]
data[datetime.min:datetime.max]
# get last 12 hours worth of data
data[datetime.utcnow() - timedelta(hours=12):]

Note that the datetime.datetime index is in UTC. You may need to apply an offset to convert to local time.

The module provides five classes to store different data. RawStore takes “raw” data from the weather station; CalibStore, HourlyStore, DailyStore and MonthlyStore store processed data (see pywws.process). All are derived from the same CoreStore class, they only differ in the keys and types of data stored in each record.

Detailed API

Classes

CalibStore(root_dir)	Stores ‘calibrated’ weather station data.
CoreStore(root_dir)
DailyStore(root_dir)	Stores daily summary weather station data.
HourlyStore(root_dir)	Stores hourly summary weather station data.
MonthlyStore(root_dir)	Stores monthly summary weather station data.
RawStore(root_dir)	Stores raw weather station data.

class pywws.filedata.CoreStore(root_dir)

Bases: object

before(idx)

Return datetime of newest existing data record whose datetime is < idx.

Might not even be in the same year! If no such record exists, return None.

after(idx)

Return datetime of oldest existing data record whose datetime is >= idx.

Might not even be in the same year! If no such record exists, return None.

nearest(idx)

Return datetime of record whose datetime is nearest idx.

flush()

update(E) → None. Equivelent to: for k in E: D[ k['idx'] ] = k"

Update D from list-like iterable E containing dicts. Pre-existing items being overwritten. Dicts are assumed to contain all appropriate keys and values.

clear()

Clears all data from the data store permanently

class pywws.filedata.RawStore(root_dir)

Bases: pywws.filedata.CoreStore

Stores raw weather station data.

dir_name = 'raw'

key_list = ['idx', 'delay', 'hum_in', 'temp_in', 'hum_out', 'temp_out', 'abs_pressure', 'wind_ave', 'wind_gust', 'wind_dir', 'rain', 'status', 'illuminance', 'uv']

solar_items = 2

conv = {'abs_pressure': <class 'float'>, 'delay': <class 'int'>, 'hum_in': <class 'int'>, 'hum_out': <class 'int'>, 'idx': <function WSDateTime.from_csv>, 'illuminance': <class 'float'>, 'rain': <class 'float'>, 'status': <bound method WSStatus.from_csv of <class 'pywws.weatherstation.WSStatus'>>, 'temp_in': <class 'float'>, 'temp_out': <class 'float'>, 'uv': <class 'int'>, 'wind_ave': <class 'float'>, 'wind_dir': <class 'int'>, 'wind_gust': <class 'float'>}

class pywws.filedata.CalibStore(root_dir)

Bases: pywws.filedata.CoreStore

Stores ‘calibrated’ weather station data.

dir_name = 'calib'

key_list = ['idx', 'delay', 'hum_in', 'temp_in', 'hum_out', 'temp_out', 'abs_pressure', 'rel_pressure', 'wind_ave', 'wind_gust', 'wind_dir', 'rain', 'status', 'illuminance', 'uv']

solar_items = 2

conv = {'abs_pressure': <class 'float'>, 'delay': <class 'int'>, 'hum_in': <class 'int'>, 'hum_out': <class 'int'>, 'idx': <function WSDateTime.from_csv>, 'illuminance': <class 'float'>, 'rain': <class 'float'>, 'rel_pressure': <class 'float'>, 'status': <bound method WSStatus.from_csv of <class 'pywws.weatherstation.WSStatus'>>, 'temp_in': <class 'float'>, 'temp_out': <class 'float'>, 'uv': <class 'int'>, 'wind_ave': <class 'float'>, 'wind_dir': <class 'float'>, 'wind_gust': <class 'float'>}

class pywws.filedata.HourlyStore(root_dir)

Bases: pywws.filedata.CoreStore

Stores hourly summary weather station data.

dir_name = 'hourly'

key_list = ['idx', 'hum_in', 'temp_in', 'hum_out', 'temp_out', 'abs_pressure', 'rel_pressure', 'pressure_trend', 'wind_ave', 'wind_gust', 'wind_dir', 'rain', 'illuminance', 'uv']

solar_items = 2

conv = {'abs_pressure': <class 'float'>, 'hum_in': <class 'int'>, 'hum_out': <class 'int'>, 'idx': <function WSDateTime.from_csv>, 'illuminance': <class 'float'>, 'pressure_trend': <class 'float'>, 'rain': <class 'float'>, 'rel_pressure': <class 'float'>, 'temp_in': <class 'float'>, 'temp_out': <class 'float'>, 'uv': <class 'int'>, 'wind_ave': <class 'float'>, 'wind_dir': <class 'float'>, 'wind_gust': <class 'float'>}

class pywws.filedata.DailyStore(root_dir)

Bases: pywws.filedata.CoreStore

Stores daily summary weather station data.

dir_name = 'daily'

key_list = ['idx', 'start', 'hum_out_ave', 'hum_out_min', 'hum_out_min_t', 'hum_out_max', 'hum_out_max_t', 'temp_out_ave', 'temp_out_min', 'temp_out_min_t', 'temp_out_max', 'temp_out_max_t', 'hum_in_ave', 'hum_in_min', 'hum_in_min_t', 'hum_in_max', 'hum_in_max_t', 'temp_in_ave', 'temp_in_min', 'temp_in_min_t', 'temp_in_max', 'temp_in_max_t', 'abs_pressure_ave', 'abs_pressure_min', 'abs_pressure_min_t', 'abs_pressure_max', 'abs_pressure_max_t', 'rel_pressure_ave', 'rel_pressure_min', 'rel_pressure_min_t', 'rel_pressure_max', 'rel_pressure_max_t', 'wind_ave', 'wind_gust', 'wind_gust_t', 'wind_dir', 'rain', 'illuminance_ave', 'illuminance_max', 'illuminance_max_t', 'uv_ave', 'uv_max', 'uv_max_t']

solar_items = 6

conv = {'abs_pressure_ave': <class 'float'>, 'abs_pressure_max': <class 'float'>, 'abs_pressure_max_t': <function WSDateTime.from_csv>, 'abs_pressure_min': <class 'float'>, 'abs_pressure_min_t': <function WSDateTime.from_csv>, 'hum_in_ave': <class 'float'>, 'hum_in_max': <class 'int'>, 'hum_in_max_t': <function WSDateTime.from_csv>, 'hum_in_min': <class 'int'>, 'hum_in_min_t': <function WSDateTime.from_csv>, 'hum_out_ave': <class 'float'>, 'hum_out_max': <class 'int'>, 'hum_out_max_t': <function WSDateTime.from_csv>, 'hum_out_min': <class 'int'>, 'hum_out_min_t': <function WSDateTime.from_csv>, 'idx': <function WSDateTime.from_csv>, 'illuminance_ave': <class 'float'>, 'illuminance_max': <class 'float'>, 'illuminance_max_t': <function WSDateTime.from_csv>, 'rain': <class 'float'>, 'rel_pressure_ave': <class 'float'>, 'rel_pressure_max': <class 'float'>, 'rel_pressure_max_t': <function WSDateTime.from_csv>, 'rel_pressure_min': <class 'float'>, 'rel_pressure_min_t': <function WSDateTime.from_csv>, 'start': <function WSDateTime.from_csv>, 'temp_in_ave': <class 'float'>, 'temp_in_max': <class 'float'>, 'temp_in_max_t': <function WSDateTime.from_csv>, 'temp_in_min': <class 'float'>, 'temp_in_min_t': <function WSDateTime.from_csv>, 'temp_out_ave': <class 'float'>, 'temp_out_max': <class 'float'>, 'temp_out_max_t': <function WSDateTime.from_csv>, 'temp_out_min': <class 'float'>, 'temp_out_min_t': <function WSDateTime.from_csv>, 'uv_ave': <class 'float'>, 'uv_max': <class 'int'>, 'uv_max_t': <function WSDateTime.from_csv>, 'wind_ave': <class 'float'>, 'wind_dir': <class 'float'>, 'wind_gust': <class 'float'>, 'wind_gust_t': <function WSDateTime.from_csv>}

class pywws.filedata.MonthlyStore(root_dir)

Bases: pywws.filedata.CoreStore

Stores monthly summary weather station data.

dir_name = 'monthly'

key_list = ['idx', 'start', 'hum_out_ave', 'hum_out_min', 'hum_out_min_t', 'hum_out_max', 'hum_out_max_t', 'temp_out_ave', 'temp_out_min_lo', 'temp_out_min_lo_t', 'temp_out_min_hi', 'temp_out_min_hi_t', 'temp_out_min_ave', 'temp_out_max_lo', 'temp_out_max_lo_t', 'temp_out_max_hi', 'temp_out_max_hi_t', 'temp_out_max_ave', 'hum_in_ave', 'hum_in_min', 'hum_in_min_t', 'hum_in_max', 'hum_in_max_t', 'temp_in_ave', 'temp_in_min_lo', 'temp_in_min_lo_t', 'temp_in_min_hi', 'temp_in_min_hi_t', 'temp_in_min_ave', 'temp_in_max_lo', 'temp_in_max_lo_t', 'temp_in_max_hi', 'temp_in_max_hi_t', 'temp_in_max_ave', 'abs_pressure_ave', 'abs_pressure_min', 'abs_pressure_min_t', 'abs_pressure_max', 'abs_pressure_max_t', 'rel_pressure_ave', 'rel_pressure_min', 'rel_pressure_min_t', 'rel_pressure_max', 'rel_pressure_max_t', 'wind_ave', 'wind_gust', 'wind_gust_t', 'wind_dir', 'rain', 'rain_days', 'illuminance_ave', 'illuminance_max_lo', 'illuminance_max_lo_t', 'illuminance_max_hi', 'illuminance_max_hi_t', 'illuminance_max_ave', 'uv_ave', 'uv_max_lo', 'uv_max_lo_t', 'uv_max_hi', 'uv_max_hi_t', 'uv_max_ave']

solar_items = 12

conv = {'abs_pressure_ave': <class 'float'>, 'abs_pressure_max': <class 'float'>, 'abs_pressure_max_t': <function WSDateTime.from_csv>, 'abs_pressure_min': <class 'float'>, 'abs_pressure_min_t': <function WSDateTime.from_csv>, 'hum_in_ave': <class 'float'>, 'hum_in_max': <class 'int'>, 'hum_in_max_t': <function WSDateTime.from_csv>, 'hum_in_min': <class 'int'>, 'hum_in_min_t': <function WSDateTime.from_csv>, 'hum_out_ave': <class 'float'>, 'hum_out_max': <class 'int'>, 'hum_out_max_t': <function WSDateTime.from_csv>, 'hum_out_min': <class 'int'>, 'hum_out_min_t': <function WSDateTime.from_csv>, 'idx': <function WSDateTime.from_csv>, 'illuminance_ave': <class 'float'>, 'illuminance_max_ave': <class 'float'>, 'illuminance_max_hi': <class 'float'>, 'illuminance_max_hi_t': <function WSDateTime.from_csv>, 'illuminance_max_lo': <class 'float'>, 'illuminance_max_lo_t': <function WSDateTime.from_csv>, 'rain': <class 'float'>, 'rain_days': <class 'int'>, 'rel_pressure_ave': <class 'float'>, 'rel_pressure_max': <class 'float'>, 'rel_pressure_max_t': <function WSDateTime.from_csv>, 'rel_pressure_min': <class 'float'>, 'rel_pressure_min_t': <function WSDateTime.from_csv>, 'start': <function WSDateTime.from_csv>, 'temp_in_ave': <class 'float'>, 'temp_in_max_ave': <class 'float'>, 'temp_in_max_hi': <class 'float'>, 'temp_in_max_hi_t': <function WSDateTime.from_csv>, 'temp_in_max_lo': <class 'float'>, 'temp_in_max_lo_t': <function WSDateTime.from_csv>, 'temp_in_min_ave': <class 'float'>, 'temp_in_min_hi': <class 'float'>, 'temp_in_min_hi_t': <function WSDateTime.from_csv>, 'temp_in_min_lo': <class 'float'>, 'temp_in_min_lo_t': <function WSDateTime.from_csv>, 'temp_out_ave': <class 'float'>, 'temp_out_max_ave': <class 'float'>, 'temp_out_max_hi': <class 'float'>, 'temp_out_max_hi_t': <function WSDateTime.from_csv>, 'temp_out_max_lo': <class 'float'>, 'temp_out_max_lo_t': <function WSDateTime.from_csv>, 'temp_out_min_ave': <class 'float'>, 'temp_out_min_hi': <class 'float'>, 'temp_out_min_hi_t': <function WSDateTime.from_csv>, 'temp_out_min_lo': <class 'float'>, 'temp_out_min_lo_t': <function WSDateTime.from_csv>, 'uv_ave': <class 'float'>, 'uv_max_ave': <class 'float'>, 'uv_max_hi': <class 'int'>, 'uv_max_hi_t': <function WSDateTime.from_csv>, 'uv_max_lo': <class 'int'>, 'uv_max_lo_t': <function WSDateTime.from_csv>, 'wind_ave': <class 'float'>, 'wind_dir': <class 'float'>, 'wind_gust': <class 'float'>, 'wind_gust_t': <function WSDateTime.from_csv>}

---

Comments or questions? Please subscribe to the pywws mailing list http://groups.google.com/group/pywws and let us know.