Latest revision as of 14:41, 2 February 2016

Database for the Lightmeter Network

The lightmeter data is centrally collected at a Zentrum für Astronomie Heidelberg in the context of the German Astrophysical Virtual Observatory (GAVO).

   The Lightmeter database: Lightweather at GAVO
   Stations in the datase: List with metadata as coordinates and calibrations
   Recent data: Query form and graph of recent uploads

Data are submitted by HTTP uploads to that location. To make this work, some rules must be followed.

Station Metadata

To make your data useful, it must be accompanied by some metadata. If you use skysensor (in a version recent enough), you can submit its ini-file that is located in the same directory as the executable. Otherwise, fill out the following template:

fullname: (Short free text, not more than, say, 70 characters)
stationId: (The station id)
long: (longitude in decimal degrees, east is positive)
lat: (latitude in decimal degrees)
height: (in meters above sea level)

Mail your file to gavo@ari.uni-heidelberg.de. We will reply by providing you an access key that you will need to upload the data.

The station id consists of the ISO country code, a designation of the location and a number; e.g., DE_HEIDELBERG_1; only letters, digits and underscores are allowed. On metadata submission, choose your own identifier. We may have to change your station id to avoid name clashes, though.

Important: If you move your sensor, it must receive a new station id, and you must submit a new metadata set. Analoguosly, if you change your sensor, you must obtain a new station id. A "station" is the combination of a specific location and a specific sensor.

Data Upload

After you have received your access key (and possibly a revised station id), you can begin uploading. This is done by HTTP uploads. You should try uploading manually first. To do this, go to http://vo.uni-hd.de/lightmeter/q/upload/custom/STATIONID/ACCESSKEY (where you must replace STATIONID and ACCESSKEY with what was sent to you, of course).

Upload Formats

You can upload both CSV and "text" format. Text format consists of lines

date time(utc) temperature count status

Such a file could look like this:

2009-05-29 00:00:01  11,3  235160 1
2009-05-29 00:00:02  11,3  235240 1
2009-05-29 00:00:03  11,3  235320 1
2009-05-29 00:00:04  11,3  235320 1
2009-05-29 00:00:05  11,3  235264 1
2009-05-29 00:00:06  11,3  234864 1
2009-05-29 00:00:07  11,3  235360 1

CSV format has semicolons as separators and must not have headers. It should look like this:

27.04.2009;12:31:14;36,9;°C;1942560;ok;
27.04.2009;12:31:15;37,0;°C;1947960;ok;
27.04.2009;12:31:16;37,0;°C;1951800;ok;
27.04.2009;12:31:17;36,9;°C;1943880;ok;
27.04.2009;12:31:18;37,0;°C;1947960;ok;
27.04.2009;12:31:19;36,9;°C;1956960;ok;

We will accept floating point numbers with both commas and decimal points as decimal separators.

You can gzip your submissions before transfer.

The receiving software is somewhat naive and infers the content from the file name extension. Legal extensions are:

• .txt -- text format
• .txt.gz -- gzip compressed text format
• .csv -- csv format
• .csv.gz -- gzip compressed csv format

Make sure you follow these conventions.

You can also upload ZIP archives of such files. The only legal extension here is zip (lower case!).

Automatic Uploads

Of course, manual uploads will become quite tedious. Therefore, we provide an automatic upload facility. While you can use anything that can do HTTP uploads (your the file goes into the inFile key, and you must give a __nevow_form__ key with the value upload), this python script does what you want (including automatic transfer compression). The program is written for python 2.x; unless you have a very good reason, you should not use python 3.x just yet, and if you have one, please feed back the (minor) patches to make the uploader work with 3.x (while keeping it ok for 2.x).

To use it, put a file named "stationinfo" into the directory from which you will upload. It must contain the station id, a blank, and the access key.

Then, in a shell, say (adapt for the location of your python interpreter and the script as necessary)

python uploadLM.py FILE1 FILE2...

We expect uploading will be included in some of the readout software.

Calibration

Your data will not immediately appear in the database since we first have to determine the calibration of your sensor. To let us do that, please wait for clear skies in the morning or evening (while taking data, of course). After you have uploaded the data obtained during such a clear dusk or dawn, notfiy TBD who will then derive a calibration from the uploaded data. Once the calibration is done, your data will show up in the database.

A step by step description of the calibration process is described at Calibration example Linz.

Topics for possible preprocessing on upload

1. time-zones --> UTC with tzdata
2. homogenize decimal separator to "." (issue for Temperature)
3. correct temperature-readings T-512 for T>256.

--GuentherWuchterl 11:02, 24. Feb. 2010 (UTC)

1. homogenize date format (Y-m-d vs. d.m.Y)
2. Where? Who? [from my view: preferably at the user's; I wouldn't want to keep corrected and uncorrected files on my side]

-- Markus 2010-02-24

1. Are there top 3 (T-512) has a solution? The problem is known from the manufacturer.

-- Hagen

Test queries

Stations in DB

Test query at GVO - to plot data scroll down and select VOplot as format

Station uploads

Examples for how to use GAVO from other fields

  http://wwwas.oats.inaf.it/aidawp5/index_eng.html?fsize=medium

  e.g.: Asteroids in the Solar System 
        http://wwwas.oats.inaf.it/aidawp5/download/eng_es9_asteroids.pdf