xAP Weather Schema Schema add change request

[pete_c]

Funny cuz my direction is to do an xAP Flash configuration on an Arm based 800Mhz touchscreen device (already based in custom flash) and in addition using the Jogglers for a similiar purpose (Intel based CPU). Response times are very quick. I did a chroot on the Arm Based 800Mhz devices; but they are still slow. Even faster was the Seagate dockstar with a Mimo USB monitor attached to it.

I think given the approach is to enhance the existing schema we shouldn't deprecate existing tags so AirPressure needs to stay (implied hPa), inconsistent as it is.

PressureTrend=falling|steady|rising ?
or maybe even a numeric
PressureTrend=-3.2 based on hourly or whatever is the key period or something ?

Historically mostly just paid attention to whether the barometric pressure was steady, rising or falling; not so much the rate though. Thats me though. I agree that we shouldn't depreciate the existing tags Kevin.

Yeah, I've had to create maps/documentation/drawings of all of my 1-wire sensors as I am using them for many things lately. IE: also added new water meters to track water usage via counters. Yes I use BSC today for all of the 1-wire sensors. After 10 years of using the AAG weather instrument I have now noticed that it over reports wind speeds something maybe related to debounce even after modification of said device. Found this out comparing it to the Fine Offset / Davis weather instruments.

The oldest 1-wire weather station stuff I have in place is made up of:

1 - AAG / Dallas Design anemometer (flawed design anyways)
2 - Pagoda (with a combo humidity, temperature and light sensor) - which quit reporting humidity and light after about 6 months or so.
3 - Lightning sensor - with a battery still running just fine after 6 years or so.
4 - Rain tipping bucket (Dallas design) - still works fine.
5 - a couple more outside 1-wire combo sensors (temp, humidity and light).
6 - not related to 1-wire I also put a GPS antenna on the same mast for time sync / geographic location stuff.

The 1-5 above I would consider one entity of a weather station and it is on one 1-wire network and doing so so after about 10 years.

I have recently enabled both the Fine Offset and Dallas weather instruments for comparisons' sake and see a difference.

You are right Kevin; there is a huge variety of 1-wire sensors which would make it difficult to standardize in the weather dot report schema; but many folks today still utilize 1-wire weather stations as an introduction to "weather watching".

That's said I agree with you relating to the 1-wire "stuff" and will continue to utilize the BSC schema for 1-wire instruments reporting.

[kev_t]

Thinking about the 'extra' keys, they don't really make a great deal of sense in this context and I agree would be better handled by BSC or TSC schema messages. Several of the rerences I found to them in any case indicated that they were 'non-standard'.

On Air pressure

AirPressureI=(Air Pressure Indication Rising/Steady/Falling)
-- optional

Is 'Dropping' the same as "Falling Quickly" ? ie Rising/Steady/Falling/Dropping

I don't agree about retaining the original AirPressure, it would make the enhanced (version 2) schema ambiguous, and perpertuate an existing error.

If a vendor was to adopt the enhanced schema with/because of the extra features, it would be trivial to modify the message to add a P or an H to the key.

Similarly if a user was to use the enhanced schema when scripting/configure their application this would be a small change for them to make particularly taking into account the extra functionality that could be provided

There are parallels in this discussion of AirPressure, with the use of rain= key in the existing implementations which I thought we had resolved.

Is a Light= key appropriate to this schema? If so what are the engineering units and/or range?

I understand your concerns about message/packet size particularly on smaller devices. I have tried sending the full message on my network here with no adverse effects so far. The News.report messages from James mi4 News app can be somewhat larger. I will reinstall a Netiom and monitor an Arduino over the next week or so.

I wont be able to do much on this until next weekend now, but I propose collating and finalising the updated schema then so the wiki can be updated. Obviously any other contributions welcome.

kevint

[Kevin]

I don't agree about retaining the original AirPressure, it would make the enhanced (version 2) schema ambiguous, and perpertuate an existing error.

If a vendor was to adopt the enhanced schema with/because of the extra features, it would be trivial to modify the message to add a P or an H to the key.

Similarly if a user was to use the enhanced schema when scripting/configure their application this would be a small change for them to make particularly taking into account the extra functionality that could be provided

There are parallels in this discussion of AirPressure, with the use of rain= key in the existing implementations which I thought we had resolved.

Ahh - perhaps I misunderstood your intent here. I thought you were going to be supplementing the existing 'weather.report' schema and not creating a new schema of a different class. If you are changing the class to weather.report.2 or whatever then you can do what you want although I would of thought that far more elegant changes were possible than for example defining units in parameter names.

If however you are supplementing the existing schema and retaining the class name then I really feel that AirPressure must remain - as otherwise you break existing users which is a no-no when they have absolutely adhered to a published schema specification. You mustn't do that as it alienates developers. You can't really backwards deprecate with the same schema name. Also it is by no means 'trivial' as you say for some users to change their parsing.

By all means add AirPressureP (optional) should you feel it's important (it is more consistent) but if it's was included in a message then AirPressure would be a mandatory key too. Which is kinda redundant.

We really should have had a way of including a revision number within an existing schema I guess.

K

[kev_t]

I dont propose creating a weather.report.2, but replacing the existing weather.report with an enhanced one, correcting the mistakes of the published schema, and of the defacto schema that is actualy in use.

Clearly existing vendors can/will continue to use the depracated version of the schema they use now until they update

I can see no point in perpetuating the mistakes once the vendors move to the new version of the schema

Perhaps the simplist solution is to replace the weather.report schema with a Weather.Display schema and at the same time depracate the original schema

kevint

[pete_c]

Is 'Dropping' the same as "Falling Quickly" ? ie Rising/Steady/Falling/Dropping

I believe that "falling" and "dropping" are synonymous relating to atmospheric pressure; but that's my impression/understanding.

For "light="; I use a formula from the value which works to convert it to UV from the national weather service here. So there is no real engineering number nomenclature coming from the read other than some number.

IE: right now the sun is setting here and the value I see is 38. Via the formula I get at 2.714 value measured in W/m squared which comes from nm of wavelengths. There are so many devices out there and means of derivation of these values it might be just a moot point.

http://en.wikipedia.org/wiki/Ultraviolet_index



When all is said and done; and the "dust" has settled then I think the vendors using the ancillary schema will update their stuff.

I know I wasn't the first to ask about new adds to the existing weather dot report broadcasts xAP schemas; and I know that the answer I received relating to the said vendors waiting or using the documented schema was probably pretty common.

It does create a bit more uniformity with one source versus a hodgepodge of unique roll your own schemas; after all Kev; you are the xAP bona fide guru.

[kev_t]

I was looking at the METAR weather reports associated with aviation and airports and I can see that it reports visibility in meters even in the UK

So assuming we report in both SI and imperial this would be

VisibiltyM=(visibility in meters)
-- optional
VisibiltyY=(visibility in yards)
-- optional

Also since the source might not be local it could be identified by the station code

Location=(Station code eg EGNJ or location)
-- optional

What do you think?

kevint

[Kevin]

Perhaps the simplist solution is to replace the weather.report schema with a Weather.Display schema and at the same time depracate the original schema

kevint

I think you should either supplement the existing schema in a similar style with the extra values you need which will be backwards compatible and NOT deprecate (at least for some considerable time) any existing key values so you don't break anything that way either ... which is by far the easiest thing to do... (KISS).

or you should spend some time creating a more capable new schema and canvas the existing developers and client software authors to adopt it which will require some effort.

If you create a new schema then I would suggest that following the existing schema is a poor design , and that some other way should be found of representing values and the units they are measured in. Perhaps allowing the data to be switched between a metric and imperial template(s) . Supplying additional parameter data when one is simply a simple conversion of another seems wasteful - but OTOH if it fits within one datagram it's not too much of a concern.

The cloud base height/type representation is also very confusing in the existing schema, and parameters should be pair optional based on sensor availability e.g. if TempC is present then TempF should be mandatory.

You should also consider support for periods of measurement like hourly, daily, monthly, annual as well as min, average and max values - the latter potentially creating a plethora of new parameters should the existing model be expanded.

TSC would have been one possible solution but IMHO that is an incomplete and flawed schema in it's current form. It is also probably overkill what is essentially always reporting data. We absolutely need the TSC schema being revisited and made fit for purpose though. Any offers ?

Keeping existing developers onside is so important to me , as it is me that takes most of the grief and support so I want to mimimise that. You don't change the rules unless it's absolutely essential. You instead offer new enticements and backwards compatibility wherever possible, which it is in this weather example.

I have personally directly supported by email / phone maybe two dozen developers with various implementions of xAP and it occupies considerable time. The current developer I am working with for example, I must have exchanged several hundred emails with in the last two months and received 64 progressive builds of the application to test. Well over a hundred hours of my time put into this one developer. However this is an exceptional case as there were major application enhancements involved and we took it step by step . Hopefully i should be announcing something in the next couple of weeks. There's a little clue in the previous sentence

K

[kev_t]

The intent was simply to make the existing schema work properly by formalising the existing schema in actual use. Through this useful discussion we have added a number of useful keys and corrected some anomalies as well.

The original schema was formulated to support the data available from Metar reports and from the BBC observations and forecasts. In these reports, there is only one engineering unit for Air Pressure and Rain is only mentioned as being light, heavy etc. The Cloud base etc key is a direct derivation of the decoded text from the metar reports

Two Weather station software vendors were convinced to support xAP, which is great, and modified the schema to suit their systems and the users who requested the support. While not ideal, to me this modification of the schema is quite legitimate and mirrors other schemas which have been similarly amended to suit the equipment being supported by the software authors, for instance like that for Caller ID/Telephony.

In the past developers have created schema to support the equipment or service they were attempting to introduce into the xAP world and have based that primarily on the data their sources were capable of providing. The xAPBSC schema has helped to reduce the need for specialist schema, but it is often a square peg in a round hole sort of fit just to use xAPBSC.

Since this schema is not a formal xAP schema like xAPBSC, and the draft xAPTSC, and is not supported by the formality of a specification or any sort of semi-formal or formal review or revision process, then vendors should continue be free to develop new and existing schema like this following the spirit of what has gone before and the basic xAP protocol definition.

In my view, not to do this will limit the ongoing viability of xAP and will remove the flexibility that developers have enjoyed up until now.

In the case of the weather schema we are discussing here, we have the weather.report class which is for current data. I would have hoped that we could have formalised the weather.forecast, Time and request classes that James used on his weather app and created a new class for history as well.

Speaking for myself, I don't have the time or the inclination to develop a brand new schema for weather from scratch, but feel that the developed schema we have here meets the requirements. I am not in any case convinced that such a brand new schema would get any take up from vendors.

In the schema I will summarise in the next post and in an attempt to remove traffic which is not required or going to be used by a user, then keys for which there is no data available should not be included in the message and the vendor should consider giving the user the option to select which key should be used where there are alternates, eg TempC and TempF. The user will then be able to configure his xAP display and data collection systems etc to his needs and the data sources he has installed.

I have specifically not included in the draft schema keys for Rain or Airpressure, ie without the conventional (for this schema) suffix. I am sure vendors might want to include them as an interim measure for compatibility but this should be a short term measure only and I see no reason to over-legitimise this by including them

I think KevinH is the last of the original xAP team and I am sure that all xAP developers and users appreciate the effort and commitment that KevinH continues puts into this.

kevint

[kev_t]

Draft Weather Schema

As descibed in the protocol definition, the message body will be preceded by a header, eg
{
v=12
hop=1
uid=FF400200
class=weather.data
source=vendor.weather
}

In the message body below, most keys are optional.

Keys for which data is not available from the data source in use should not be included.

To reduce the message complexity, message size, and to remove data that the end user will not use, the vendor may consider allowing the user to disable keys or key pairs, even if data is available

For example :
if WindDirD was not required because, for instance, it was non-numeric it would not be included
if TempF was not required because temperature in fahrenheit was not appropriate to the localisation it would not be included
If the Indoor Humidity and temperature were not required then all 3 could be omitted

Previous schema included keys for AirPressure and actual implementations included Rain, but this was not in keeping with the overall approach of the schema. Vendors may wish to keep these keys in the short term for compatibility, but both users and vendors are encouraged to use the format below for these keys.

weather.report
{
UTC=(Time of report in hh:mm format using utc time zone)
-- mandatory
DATE=(Date of report in YYYYMMDD format)
-- mandatory
Station=(Station code or identifier eg EGNJ)
-- optional
Location=(location, city, town, village)
-- optional
WindM=(Value of wind in mph or "Gusty" if gusty)
-- optional
WindK=(Value of wind in kph or "Gusty" if gusty)
-- optional
WindGustsM=(Value of wind gusts in mph)
-- optional
WindGustsK=(Value of wind gusts in kph)
-- optional
WindDirC=(Compass heading of wind N|NE|E|SE|S|SW|W|NW)
-- optional
WindDirD=(Compass heading of wind in degrees)
-- optional
TempC=(Temperature in centigrade)
-- optional
TempF=(Temperature in fahrenheit)
-- optional
DewC=(Dew point in centigrade)
-- optional
DewF=(Dew point in fahrenheit)
-- optional
RainM=(Rainfall since midnight in mm)
-- optional
RainI=(Rainfall since midnight in inches)
-- optional
Humidity=(Relative Humidity as %)
-- optional
AirPressureP=(Air pressure in hPa)
-- optional
AirPressureH=(Air pressure in inHg)
-- optional
AirPressureI=(Air Pressure Indication Rising/Steady/Falling)
-- optional
VisibiltyM=(visibility in meters)
-- optional
VisibiltyY=(visibility in yards)
-- optional
Cloud=(Overall cloud cover eg "Clear Skies")
-- optional
CloudM.X=(Cloud type and height in miles ie "Overcast at 5.4M. X increments on each cloud layer)
-- optional
CloudK.X=(Cloud type and heightin kilometers ie "Overcast at 5.4Km. X increments on each cloud layer)
-- optional
WindchillC=(Windchill Temperature in centigrade)
-- optional
WindchillF=(Windchill Temperature in fahrenheit)
-- optional
InTempC=(Indoor Temperature in centigrade)
-- optional
InTempF=(Indoor Temperature in fahrenheit)
-- optional
InHumidity=(Indoor Relative Humidity as %)
-- optional
Picture=(url to weather/webcam picture)
--optional
Icon=(The name of the picture to use, icon names from kweather www.kde.org. File names below)
-- optional
Error=(Download errors either "NoData" if last connection failed or "StationNotFound" if ICAO code isn't valid)
-- optional
}

Possible icon names, higher number signifies more clouds

cloudy1
cloudy2
cloudy3
cloudy4
cloudy5
dunno (usually with "error=nodata")
fog
hail
light_rain
mist
overcast
shower1
shower2
shower3
sleet
snow1
snow2
snow3
snow4
snow5
sunny
tstorm1
tstorm2
tstorm3

[kev_t]

For anyone interested, at the moment (feb 2012) Clas Ohlson have the WH1080 USB Wireless Weather Station on special at £59.00, in store only purchase and when I was in there in passing this week they were running a £10 off coupon for purchases over £50

http://www.clasohlson.co.uk/Product/...x?id=167399209

The WH1080 is supported by both Weather Display and Cumulus which support the existing xAP Weather Schema

Please note that although it looks like you can buy on line, you are only filling in a shopping list you can print out.

kevint