xAP Weather Schema Schema add change request

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts
  • pete_c
    Automated Home Jr Member
    • Feb 2011
    • 39

    xAP Weather Schema Schema add change request

    I am curious what I need to do to request a schema change in the weather report class?

    Today its:

    weather.report
    {
    UTC=(Time of report in hh:mm format using utc time zone)
    -- mandatory
    DATE=(Date of report in YYYYMMDD format)
    -- mandatory
    WindM=(Value of wind in mph or "Gusty" if gusty)
    -- optional
    WindK=(Value of wind in kph or "Gusty" if gusty)
    -- optional
    WindGustsM=(Value of wind gusts in mph)
    -- optional
    WindGustsK=(Value of wind gusts in kph)
    -- optional
    WindDirC=(Compas heading of wind N|NE|E|SE|S|SW|W|NW)
    -- optional
    WindDirD=(Compas heading of wind in degrees)
    -- optional
    TempC=(Temperature in centigrade)
    -- optional
    TempF=(Temperature in fahrenheit)
    -- optional
    DewC=(Dew point in fahrenheit)
    -- optional
    DewF=(Dew point in fahrenheit)
    -- optional
    AirPressure=(Air pressure in hPa)
    -- optional
    Cloud=(Overal cloud cover currently only "Clear Skies")
    -- optional
    CloudM.X=(Cloud type and height in miles ie "Overcast at 5.4M. X increments on each cloud layer)
    -- optional
    CloudK.X=(Cloud type and heightin kilometers ie "Overcast at 5.4Km. X increments on each cloud layer)
    -- optional
    Icon=(The name of the picture to use, icon names from kweather www.kde.org. File names below)
    -- optional
    Error=(Download errors either "NoData" if last connection failed or "StationNotFound" if ICAO code isn't valid)
    -- optional
    }
    Steve; the author of Cumulus added Rain (in mm I believe) which I use. It would be better though to change the Rain part to:

    RainM=x.yy
    RainI=x.yy

    and another ideal add would be relative humidity:

    Humidity=xxx.yy

    I am currently playing with the weather.report schema with Cumulus testing the Fine Offset and Davis Vantage Vue weather stations.

    I've asked the author of a Linux based similiar to Cumulus application to add XaP / weather.report schema or if I could help add XaP to his application.

    I'm looking to move the connection(s) over to a sheevaplug currently testing the Linux application on a Seagate Dockstar.
    Last edited by pete_c; 7 February 2012, 05:14 PM.
    - Pete C
    HW-HAI OPII - X10,UPB,Z-Wave,Russound,1-Wire(X4 networks) OmniTouch screens, Intel & Arm Touchscreens (Android, Wintel & Linux)and OmniStat2
    SW-HSPro - Atom D525/4Gb -W2003SP2 X86 - 18 serial ports via USB (Digi Edgeports & Digi Hub)
    Weather Cumulus - Fine Offset & Davis Vantage Vue
  • kev_t
    Automated Home Sr Member
    • Jun 2004
    • 58

    #2
    Hi Pete

    A couple of things to note about the Weather Schema:-

    It is not a xAPWeather Schema, ie it is not a formal core xAP Schema with a specification like for instance xAPBSC

    If you look at the schema, most of the keys are optional. Also you will see that most data 'types' eg wind, are available in both imperial and metric forms

    This schema originated many years ago and I think was based on the data either from a one wire weather station or the data aquired from a page scrape, or perhaps both.

    I can think of 3 readily availbale sources of Weather Schema data

    mi4's weather connector, which grabs metar data like
    2012/02/08 10:20
    EGNJ 081020Z 00000KT 9999 BKN026 00/M02 Q1044

    The WeatherDisplay program which interfaces with a variety of weather stations

    Cumulus which similarly interfaces with a variety of weather stations

    The implementation is based on the schema, but which options are selected depends on the inclinations of the implementor and the data available. In addition other keys may also be added. So the Weather Display program produces messages like

    {
    v=12
    hop=1
    uid=FF400200
    class=Weather.Report
    source=windy.weather.1
    }
    Weather.Report
    {
    tempf=34.2
    utc=10:52:25
    tempc=1.2
    icon=Sunny
    date=20120208
    windm= 0
    windk= 0
    airpressure=1041.1
    winddirc=N
    humidity=88
    rain=0.0
    extratemp1=-100.0
    extratemp2=-100.0
    extratemp3=-100.0
    extrahum1=-100
    extrahum2=-100
    extrahum3=-100
    }

    So you can see here that he has added rain (in inches) and humidity, but left out for instance wind direction in degrees (regrettably).

    The 'extra...' keys come from someone elses extention to the schema, I think for one-wire, which have been slavishly copied here but are not used.

    James' (mi4) xAPWeather connector produces messages like

    xap-header
    {
    v=12
    hop=1
    uid=FF400100
    class=Weather.Report
    source=mi4.weather.egnj
    }
    Weather.Report
    {
    UTC=10:20
    Date=20120208
    WindM=Gusty
    WindK=Gusty
    TempC=0.0
    TempF=32.0
    DewC=-2.0
    DewF=28.4
    AirPressure=1044
    Icon=cloudy3
    CloudK.0=Broken cloud at 0.8km
    CloudM.0=Broken cloud at 0.5M
    }
    Time
    {
    TOD=Day
    Time=11:06:55
    SunRise=07:29
    SunSet=16:58
    }

    Which just decodes the METAR data and adds an extra section for Sunrise and Sunset based on Latitude and Longditude. Regrettably since the BBC has changed their Weather site and / or the location indexes the non metar data sources no longer work and so the Weather Forecast messages are not available

    I don't have examples of the Cumulus message to hand, perhaps you could post an example here?

    For completeness, there is another, very different, version of the schema on Patrick Lidstones site, but this does not seem to be the common usage, or adopted as the schema on the official xAP site and so probaly should be avoided.

    So to answer your main question, from the above, I would advise that you base your extension to the schema on what has gone before

    I have a spare Dockstar here so I would be very interested to hear if you make any progress with that

    I am sure there may be other comments, but you can see above what has gone before

    Hope this helps

    Kevint
    (The other kevin)

    Comment

    • pete_c
      Automated Home Jr Member
      • Feb 2011
      • 39

      #3
      Hello Kevint,

      Understood.

      Here too now many of the online weather services are looking/changed to charge a monthly service fee to access their data and starting not to provide weather data for "free" anymore. I've gone now to doing screen scrapes and snapshots of HTML pages for weather maps and such.

      Steve has mentioned that he added rain to the Cumulus XaP broadcast; but it wasn't the standard and is hestitate to add "undocumented" XaP features to his application.

      Its like the "which came first" scenario; the chicken or the egg.

      I've used the AAG (Dallas designed anemometer) for a bit over 10 years or so along with a Dallas 1-wire Rain guage. Great stuff; 1-wire; whole home is covered here both inside and out with 1-wire devices.

      Just recently started to "test" the Fine Offset and Dallas Vantage Vue. Have the two of them gathering data and comparing it to the AAG. The AAG has always over reported wind speeds; think that's a debounce design flaw thing though. (I have 2 AAG wind speed devices which over report wind speeds).

      Here is the Cumulus data from the Fine Offset and the Davis Vantage Vue:

      xap-header
      {
      v=12
      hop=1
      uid=FF437500
      class=weather.report
      source=sanday.cumulus.ICS-HST
      }
      weather.report
      {
      UTC=13:47
      DATE=20120208
      WindM=8.0
      WindK=12.9
      WindGustsM=13.0
      WindGustsK=20.9
      WindDirD=293
      WindDirC=WNW
      TempC=-2.6
      TempF=27.4
      DewC=-5.7
      DewF=21.7
      AirPressure=1029.2
      Rain=0.0
      }
      xap-header
      {
      v=12
      hop=1
      uid=FF437500
      class=weather.report
      source=sanday.cumulus.ICS-STORAGE-2
      }
      weather.report
      {
      UTC=13:48
      DATE=20120208
      WindM=3.8
      WindK=6.1
      WindGustsM=11.4
      WindGustsK=18.4
      WindDirD=315
      WindDirC=NW
      TempC=-2.6
      TempF=27.3
      DewC=-6.2
      DewF=20.8
      AirPressure=1031.4
      Rain=0.0
      }
      Today I have two Seagates running. The one in "production" is running a sprinkler control program connected to two Rain8Nets via USB. It has a web server / web interface and a Touchscreen interface running on ArchLinux. Talks and receives XaP.

      The second Seagate in test mode is running "Dockstar Debian" and a linux (similiar to Cumulus) program called "weather view" plus a 1-wire read one wire application (which is broadcasting XaP). It is connected to both the Davis Vantage view (serially ) and the Midon Temp0x serially (via USB to serial devices).

      The author of Cumulus doesn't want to add anything XaP which is not documented; for fear he is not standardized I guess.

      Thank you KevinT.

      Attached are pics of my Seagate setup.
      Attached Files
      Last edited by pete_c; 8 February 2012, 03:12 PM.
      - Pete C
      HW-HAI OPII - X10,UPB,Z-Wave,Russound,1-Wire(X4 networks) OmniTouch screens, Intel & Arm Touchscreens (Android, Wintel & Linux)and OmniStat2
      SW-HSPro - Atom D525/4Gb -W2003SP2 X86 - 18 serial ports via USB (Digi Edgeports & Digi Hub)
      Weather Cumulus - Fine Offset & Davis Vantage Vue

      Comment

      • kev_t
        Automated Home Sr Member
        • Jun 2004
        • 58

        #4
        To be clear,

        There is no specification to adhere to for the Weather.Report schema

        There is a description of the schema, much of which is optional

        The data in the schema should be based on the data available from the data source

        Keys which are optional need not be included

        And, most importantly, where additional data is available then it should be added, since applications which parse the message should gracefully ignore anything they don't understand.

        Steve of Cumulus/Sanday fame should therefore not be concerned about adding extra keys, The most obvious of which is humidity, which is already available in the message from WeatherDisplay.

        I know there have been comments on the ommision of Humidty from the Cumulus weather message in other forums, eg that for the Livebox HAH.

        Thanks for posting the sample messages and updating us on what you are doing

        (and pretty please can we call it xAP rather than any other variation of upper and lower case? )

        kevint

        Comment

        • Kevin
          Moderator
          • Jan 2004
          • 558

          #5
          Weather Schema additions

          Can I just weigh in with a view somewhere inbetween.

          The weather schema is not an 'official' xAP schema in that it doesn't have a schema name starting xAP... however it is presented as a standard template in that if everyone adheres to it then it doesn't matter which weather station or software you use they will operate the same (acknowledging that some sensors may not be available on some stations). This is a fundamental tenet in xAP.

          Therefore it is important that things are done in a unified way . It's no good if developers are adding Hum= Hum%= RelHum= RHum= HumP= if you get my drift. Where units are involved even more problematic, the RainM might imply miles if you look at CloudM for example and RainM has an implied period of measurement.

          So I would like to try and get some concensus on what needs to be added and then get those included on the website within a schema revision and everyone benefits from a standard approach.

          We did revisit the schema before, looking to make it complete but it's very easy to get bogged down with units and measurement periods for example rain in the last hour, day, month, year etc. Let's try and get the fundamanental missing bits added, and maybe take an approach of adding additional blocks for period based data.

          You are of course free to include your own paramaters into xAP message blocks* although I've never really been comfortable with this - especially in terms of using common parameter names as they can later conflict with schema revisions. It was really intended as a 'for your own purposes' thing rather than for a released application. Regardless, extra parameters should not break anything.

          So Pete and others , suggest away, I'm keen to get something better out there too. What I would ask is to give it some good thought re parameter names, units, values and completeness so that we try and make it as thorough and capable as possible.

          K

          * edit: Just wanted to add that I agree that xAP developers should add extra keys to a schema when they've deficiencies or have extra data to include - but really with a purpose of suggesting such keys become part of a schema revision so we all benefit. Humidty is a good example here. Yes , you can add your own keys for your own purposes but that may not be appropriate for a released application and if that significantly changes the whole schema then probably it should be in another block or a new schema. There's no reason you can't support two schema concurrently either.
          Last edited by Kevin; 9 February 2012, 02:07 AM.

          Comment

          • kev_t
            Automated Home Sr Member
            • Jun 2004
            • 58

            #6
            Consensus

            I had thought that by publishing some examples of the current implementations, that have been in use for some time that we had demonstrated a consensus

            What we are talking about here is adding two keys, rain and humidity which are pretty fundamental to a weather report

            And in any case any receiving application should gracefully ignore, ie not break, if it gets keys it does not recognise


            kevint
            Last edited by kev_t; 9 February 2012, 10:14 AM.

            Comment

            • Kevin
              Moderator
              • Jan 2004
              • 558

              #7
              Kevin,

              So .. for example... what does

              Rain= show ?

              K

              Comment

              • pete_c
                Automated Home Jr Member
                • Feb 2011
                • 39

                #8
                Yes; relating the the current schema; I am and have used 1-wire instruments for many years. I am "testing" today with three weather stations. The xAP data on the xAP network is a real hodgepodge of data.

                This said I am attempting to convey a message (s) of an agreed upon standard for some addicted home automation enthusists and a shift to utilizing
                xAP as a means of home automation transport. (a do all of sorts).

                That said the author of Cumulus is aware of the currently documented weather report xAP schema and has told me (written) that he is hesitant to make changes to his application as he doesn't want to step beyond the boundaries of said documentation or "step on any toes".

                My "testing" went to utilizing very common "other" weather stations; IE: Davis and the Fine Offset and attempting to utilize broadcast xAP data with the common Cumulus application. A whole different subject but currently benchmarking the three devices using just xAP data graphing numbers such that folks can see in vivo what xAP can do for them.
                Last edited by pete_c; 9 February 2012, 02:52 PM.
                - Pete C
                HW-HAI OPII - X10,UPB,Z-Wave,Russound,1-Wire(X4 networks) OmniTouch screens, Intel & Arm Touchscreens (Android, Wintel & Linux)and OmniStat2
                SW-HSPro - Atom D525/4Gb -W2003SP2 X86 - 18 serial ports via USB (Digi Edgeports & Digi Hub)
                Weather Cumulus - Fine Offset & Davis Vantage Vue

                Comment

                • kev_t
                  Automated Home Sr Member
                  • Jun 2004
                  • 58

                  #9
                  KH

                  I guess you are referring to the fact that the rain= key does not specify the units

                  We already have at least two implentations which just report rain=

                  Pete says his Cumulus is reporting in mm
                  My WD is reporting in inches

                  From a quick look at each program this is configurable, and you use the units with which you are most familiar or approprioate to your localisation.

                  I don't have a problem with the key not specifying units as I expect my rainfall to be in inches since that is how I configured WD, and I would probably notice if it misreported an inch of rain as 25.4

                  A discussion on fixing this schema took place in 2008 and was not resolved and has been raised on a number of different support forum since, we should progress this now and update the wiki so we don't have to revisit

                  kevint
                  Last edited by kev_t; 9 February 2012, 02:44 PM.

                  Comment

                  • pete_c
                    Automated Home Jr Member
                    • Feb 2011
                    • 39

                    #10
                    That would be great!

                    Personally if the numbers don't give me what I want; then I just convert them; IE: mm to in or in to mm. I do the conversions "on the fly" as the data is being entered into a database.

                    But that is me and my setup.

                    An xAP "newbee" would assume or will just utilize the numbers "as is"; plug n play like.
                    - Pete C
                    HW-HAI OPII - X10,UPB,Z-Wave,Russound,1-Wire(X4 networks) OmniTouch screens, Intel & Arm Touchscreens (Android, Wintel & Linux)and OmniStat2
                    SW-HSPro - Atom D525/4Gb -W2003SP2 X86 - 18 serial ports via USB (Digi Edgeports & Digi Hub)
                    Weather Cumulus - Fine Offset & Davis Vantage Vue

                    Comment

                    • Kevin
                      Moderator
                      • Jan 2004
                      • 558

                      #11
                      Originally posted by kev_t View Post
                      KH

                      I guess you are referring to the fact that the rain= key does not specify the units

                      We already have at least two implentations which just report rain=

                      Pete says his Cumulus is reporting in mm
                      My WD is reporting in inches
                      Yes exactly my point - allowing people to use their own parameters has introduced an inconsistency within a schema that itself defined well the measurement units.

                      Also there is no indication of teh period of measurement - per hour - per day etc..

                      It's these sort of things we must avoid or people should use another schema of their own.

                      A discussion on fixing this schema took place in 2008 and was not resolved and has been raised on a number of different support forum since, we should progress this now and update the wiki so we don't have to revisit
                      kevint
                      Agreed - but we need a champion who understands the schema and can make reasoned and complete changes. Are humidity and rain all we need ?

                      K

                      Comment

                      • pete_c
                        Automated Home Jr Member
                        • Feb 2011
                        • 39

                        #12
                        On the Cumulus side most (many) folks are utilizing the Fine Offset or Davis weather instruments. I personally utilize the humidity / rain for a functional multizoned watering device (xAP) which calculates ET (evapotranspiration) based on this data. (internal weather instruments versus internet weather stations). I am into weather though and my Home automation screens plot data from the weather stations. (three today; 1-wire, Fine Offset and Davis).

                        I also send my data to WeatherUnderground. Here are some links of my stuff:













                        So I can't speak for others using the Cumulus weather station.

                        On a concurrent endeavor though I am "pushing" to add xAP data to a Linux based application called "WeatherView" which is kind of similiar to Cumulus but a Linux program which I do have running and configured to talk to my Davis Weather Station and it is working fine. It would be beneficial here to use the documented xAP schema (weather report) which included both humidity and rain aside from the rest of the already included data.
                        Last edited by pete_c; 9 February 2012, 03:34 PM.
                        - Pete C
                        HW-HAI OPII - X10,UPB,Z-Wave,Russound,1-Wire(X4 networks) OmniTouch screens, Intel & Arm Touchscreens (Android, Wintel & Linux)and OmniStat2
                        SW-HSPro - Atom D525/4Gb -W2003SP2 X86 - 18 serial ports via USB (Digi Edgeports & Digi Hub)
                        Weather Cumulus - Fine Offset & Davis Vantage Vue

                        Comment

                        • kev_t
                          Automated Home Sr Member
                          • Jun 2004
                          • 58

                          #13
                          Rain:-

                          On the period for reporting, I have always understood that rainfall is reset at midnight

                          Clearly you are proposing that there be two 'rains'

                          one for metric say rainSI, eng units would be mm

                          one for imperial say rainI, the eng units would be inches

                          as with all other keys, these would be optional

                          Of course this makes the current implementations not meet the newly redefined schema, but if that is what it takes to make progress then so be it

                          Humidity:-

                          the key for this would be humidity

                          The eng units would be given to be %

                          this key is optional

                          I don't know what if any other optional keys we might need, these are the ones we are discussing now

                          I would reiterate:-
                          "In any case any receiving application should gracefully ignore, ie not break, if it gets keys it does not recognise"

                          Is this acceptable to all?

                          kevint

                          Comment

                          • pete_c
                            Automated Home Jr Member
                            • Feb 2011
                            • 39

                            #14
                            one for metric say rainSI, eng units would be mm

                            one for imperial say rainI, the eng units would be inches
                            What the the "S" in "rainSI" stand for?

                            Thinking maybe simplier like:

                            RainM
                            RainI

                            and globally/internationall isn't humidity usually referred to relative humidity?

                            So the definitions would say:

                            RainI=(Value of rain in inches)
                            -- optional

                            RainM=(Value of rain in millimeter)
                            -- optional

                            Humidity=(Relative Humidity Percentage)
                            -- optional
                            Last edited by pete_c; 9 February 2012, 05:32 PM.
                            - Pete C
                            HW-HAI OPII - X10,UPB,Z-Wave,Russound,1-Wire(X4 networks) OmniTouch screens, Intel & Arm Touchscreens (Android, Wintel & Linux)and OmniStat2
                            SW-HSPro - Atom D525/4Gb -W2003SP2 X86 - 18 serial ports via USB (Digi Edgeports & Digi Hub)
                            Weather Cumulus - Fine Offset & Davis Vantage Vue

                            Comment

                            • Kevin
                              Moderator
                              • Jan 2004
                              • 558

                              #15
                              Originally posted by kev_t View Post
                              Rain:-

                              On the period for reporting, I have always understood that rainfall is reset at midnight
                              Fine.. if that is the norm / what people want then so shall it be..
                              Clearly you are proposing that there be two 'rains'

                              one for metric say rainSI, eng units would be mm

                              one for imperial say rainI, the eng units would be inches

                              as with all other keys, these would be optional
                              The very first post in this thread shows the existing schema and it is self evident that it reports the same sensor data using multiple measurement units defined in the paramater name - I think any extension to this schema should be inkeeping with that. I am not sure SI is inline with other parameter names ... however right it might be.
                              Of course this makes the current implementations not meet the newly redefined schema, but if that is what it takes to make progress then so be it
                              It's no good getting grouchy about it - if those developers chose to extend an existing schema without concensus or documenting it then that is what will happen. The cause is the very issue you're advocating - adding your own parameters and why people like SandaySoft are right when they say they don't just want to extend without concensus.
                              Humidity:-

                              the key for this would be humidity

                              The eng units would be given to be %

                              this key is optional
                              Sounds fine
                              I don't know what if any other optional keys we might need, these are the ones we are discussing now
                              That was why I asked for completeness- anyone have any more whilst we're at it ?
                              I would reiterate:-
                              "In any case any receiving application should gracefully ignore, ie not break, if it gets keys it does not recognise"
                              ... yes... I did respond to this before .. However it is the reason why we now have an ambiguous rain= value as no units were defined and also we could potentially invalidate the existing rain= usage if for example we defined it as always inches or having a measurement value appended to the value e.g. rain=12mm. It doesn't break persay but it does mean that if we used that parameter in a formal definition we may present data in an incompatible form to the existing usage.

                              How long is a max size message currently - does anyone know ?

                              K

                              Comment

                              Working...
                              X