xAP Weather Schema Schema add change request

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts
  • pete_c
    Automated Home Jr Member
    • Feb 2011
    • 39

    #31
    Funny cuz my direction is to do an xAP Flash configuration on an Arm based 800Mhz touchscreen device (already based in custom flash) and in addition using the Jogglers for a similiar purpose (Intel based CPU). Response times are very quick. I did a chroot on the Arm Based 800Mhz devices; but they are still slow. Even faster was the Seagate dockstar with a Mimo USB monitor attached to it.

    I think given the approach is to enhance the existing schema we shouldn't deprecate existing tags so AirPressure needs to stay (implied hPa), inconsistent as it is.

    PressureTrend=falling|steady|rising ?
    or maybe even a numeric
    PressureTrend=-3.2 based on hourly or whatever is the key period or something ?
    Historically mostly just paid attention to whether the barometric pressure was steady, rising or falling; not so much the rate though. Thats me though. I agree that we shouldn't depreciate the existing tags Kevin.

    Yeah, I've had to create maps/documentation/drawings of all of my 1-wire sensors as I am using them for many things lately. IE: also added new water meters to track water usage via counters. Yes I use BSC today for all of the 1-wire sensors. After 10 years of using the AAG weather instrument I have now noticed that it over reports wind speeds something maybe related to debounce even after modification of said device. Found this out comparing it to the Fine Offset / Davis weather instruments.

    The oldest 1-wire weather station stuff I have in place is made up of:

    1 - AAG / Dallas Design anemometer (flawed design anyways)
    2 - Pagoda (with a combo humidity, temperature and light sensor) - which quit reporting humidity and light after about 6 months or so.
    3 - Lightning sensor - with a battery still running just fine after 6 years or so.
    4 - Rain tipping bucket (Dallas design) - still works fine.
    5 - a couple more outside 1-wire combo sensors (temp, humidity and light).
    6 - not related to 1-wire I also put a GPS antenna on the same mast for time sync / geographic location stuff.

    The 1-5 above I would consider one entity of a weather station and it is on one 1-wire network and doing so so after about 10 years.

    I have recently enabled both the Fine Offset and Dallas weather instruments for comparisons' sake and see a difference.

    You are right Kevin; there is a huge variety of 1-wire sensors which would make it difficult to standardize in the weather dot report schema; but many folks today still utilize 1-wire weather stations as an introduction to "weather watching".

    That's said I agree with you relating to the 1-wire "stuff" and will continue to utilize the BSC schema for 1-wire instruments reporting.
    Last edited by pete_c; 12 February 2012, 06:29 PM.
    - Pete C
    HW-HAI OPII - X10,UPB,Z-Wave,Russound,1-Wire(X4 networks) OmniTouch screens, Intel & Arm Touchscreens (Android, Wintel & Linux)and OmniStat2
    SW-HSPro - Atom D525/4Gb -W2003SP2 X86 - 18 serial ports via USB (Digi Edgeports & Digi Hub)
    Weather Cumulus - Fine Offset & Davis Vantage Vue

    Comment

    • kev_t
      Automated Home Sr Member
      • Jun 2004
      • 58

      #32
      Thinking about the 'extra' keys, they don't really make a great deal of sense in this context and I agree would be better handled by BSC or TSC schema messages. Several of the rerences I found to them in any case indicated that they were 'non-standard'.

      On Air pressure

      AirPressureI=(Air Pressure Indication Rising/Steady/Falling)
      -- optional

      Is 'Dropping' the same as "Falling Quickly" ? ie Rising/Steady/Falling/Dropping

      I don't agree about retaining the original AirPressure, it would make the enhanced (version 2) schema ambiguous, and perpertuate an existing error.

      If a vendor was to adopt the enhanced schema with/because of the extra features, it would be trivial to modify the message to add a P or an H to the key.

      Similarly if a user was to use the enhanced schema when scripting/configure their application this would be a small change for them to make particularly taking into account the extra functionality that could be provided

      There are parallels in this discussion of AirPressure, with the use of rain= key in the existing implementations which I thought we had resolved.

      Is a Light= key appropriate to this schema? If so what are the engineering units and/or range?

      I understand your concerns about message/packet size particularly on smaller devices. I have tried sending the full message on my network here with no adverse effects so far. The News.report messages from James mi4 News app can be somewhat larger. I will reinstall a Netiom and monitor an Arduino over the next week or so.

      I wont be able to do much on this until next weekend now, but I propose collating and finalising the updated schema then so the wiki can be updated. Obviously any other contributions welcome.

      kevint

      Comment

      • Kevin
        Moderator
        • Jan 2004
        • 558

        #33
        Originally posted by kev_t View Post
        I don't agree about retaining the original AirPressure, it would make the enhanced (version 2) schema ambiguous, and perpertuate an existing error.

        If a vendor was to adopt the enhanced schema with/because of the extra features, it would be trivial to modify the message to add a P or an H to the key.

        Similarly if a user was to use the enhanced schema when scripting/configure their application this would be a small change for them to make particularly taking into account the extra functionality that could be provided

        There are parallels in this discussion of AirPressure, with the use of rain= key in the existing implementations which I thought we had resolved.
        Ahh - perhaps I misunderstood your intent here. I thought you were going to be supplementing the existing 'weather.report' schema and not creating a new schema of a different class. If you are changing the class to weather.report.2 or whatever then you can do what you want although I would of thought that far more elegant changes were possible than for example defining units in parameter names.

        If however you are supplementing the existing schema and retaining the class name then I really feel that AirPressure must remain - as otherwise you break existing users which is a no-no when they have absolutely adhered to a published schema specification. You mustn't do that as it alienates developers. You can't really backwards deprecate with the same schema name. Also it is by no means 'trivial' as you say for some users to change their parsing.

        By all means add AirPressureP (optional) should you feel it's important (it is more consistent) but if it's was included in a message then AirPressure would be a mandatory key too. Which is kinda redundant.

        We really should have had a way of including a revision number within an existing schema I guess.

        K
        Last edited by Kevin; 12 February 2012, 10:00 PM.

        Comment

        • kev_t
          Automated Home Sr Member
          • Jun 2004
          • 58

          #34
          I dont propose creating a weather.report.2, but replacing the existing weather.report with an enhanced one, correcting the mistakes of the published schema, and of the defacto schema that is actualy in use.

          Clearly existing vendors can/will continue to use the depracated version of the schema they use now until they update

          I can see no point in perpetuating the mistakes once the vendors move to the new version of the schema

          Perhaps the simplist solution is to replace the weather.report schema with a Weather.Display schema and at the same time depracate the original schema

          kevint
          Last edited by kev_t; 12 February 2012, 10:14 PM.

          Comment

          • pete_c
            Automated Home Jr Member
            • Feb 2011
            • 39

            #35
            Is 'Dropping' the same as "Falling Quickly" ? ie Rising/Steady/Falling/Dropping
            I believe that "falling" and "dropping" are synonymous relating to atmospheric pressure; but that's my impression/understanding.

            For "light="; I use a formula from the value which works to convert it to UV from the national weather service here. So there is no real engineering number nomenclature coming from the read other than some number.

            IE: right now the sun is setting here and the value I see is 38. Via the formula I get at 2.714 value measured in W/m squared which comes from nm of wavelengths. There are so many devices out there and means of derivation of these values it might be just a moot point.





            When all is said and done; and the "dust" has settled then I think the vendors using the ancillary schema will update their stuff.

            I know I wasn't the first to ask about new adds to the existing weather dot report broadcasts xAP schemas; and I know that the answer I received relating to the said vendors waiting or using the documented schema was probably pretty common.

            It does create a bit more uniformity with one source versus a hodgepodge of unique roll your own schemas; after all Kev; you are the xAP bona fide guru.
            - Pete C
            HW-HAI OPII - X10,UPB,Z-Wave,Russound,1-Wire(X4 networks) OmniTouch screens, Intel & Arm Touchscreens (Android, Wintel & Linux)and OmniStat2
            SW-HSPro - Atom D525/4Gb -W2003SP2 X86 - 18 serial ports via USB (Digi Edgeports & Digi Hub)
            Weather Cumulus - Fine Offset & Davis Vantage Vue

            Comment

            • kev_t
              Automated Home Sr Member
              • Jun 2004
              • 58

              #36
              Metar - visibility

              I was looking at the METAR weather reports associated with aviation and airports and I can see that it reports visibility in meters even in the UK

              So assuming we report in both SI and imperial this would be

              VisibiltyM=(visibility in meters)
              -- optional
              VisibiltyY=(visibility in yards)
              -- optional

              Also since the source might not be local it could be identified by the station code

              Location=(Station code eg EGNJ or location)
              -- optional

              What do you think?

              kevint

              Comment

              • Kevin
                Moderator
                • Jan 2004
                • 558

                #37
                Originally posted by kev_t View Post
                Perhaps the simplist solution is to replace the weather.report schema with a Weather.Display schema and at the same time depracate the original schema

                kevint
                I think you should either supplement the existing schema in a similar style with the extra values you need which will be backwards compatible and NOT deprecate (at least for some considerable time) any existing key values so you don't break anything that way either ... which is by far the easiest thing to do... (KISS).

                or you should spend some time creating a more capable new schema and canvas the existing developers and client software authors to adopt it which will require some effort.

                If you create a new schema then I would suggest that following the existing schema is a poor design , and that some other way should be found of representing values and the units they are measured in. Perhaps allowing the data to be switched between a metric and imperial template(s) . Supplying additional parameter data when one is simply a simple conversion of another seems wasteful - but OTOH if it fits within one datagram it's not too much of a concern.

                The cloud base height/type representation is also very confusing in the existing schema, and parameters should be pair optional based on sensor availability e.g. if TempC is present then TempF should be mandatory.

                You should also consider support for periods of measurement like hourly, daily, monthly, annual as well as min, average and max values - the latter potentially creating a plethora of new parameters should the existing model be expanded.

                TSC would have been one possible solution but IMHO that is an incomplete and flawed schema in it's current form. It is also probably overkill what is essentially always reporting data. We absolutely need the TSC schema being revisited and made fit for purpose though. Any offers ?

                Keeping existing developers onside is so important to me , as it is me that takes most of the grief and support so I want to mimimise that. You don't change the rules unless it's absolutely essential. You instead offer new enticements and backwards compatibility wherever possible, which it is in this weather example.

                I have personally directly supported by email / phone maybe two dozen developers with various implementions of xAP and it occupies considerable time. The current developer I am working with for example, I must have exchanged several hundred emails with in the last two months and received 64 progressive builds of the application to test. Well over a hundred hours of my time put into this one developer. However this is an exceptional case as there were major application enhancements involved and we took it step by step . Hopefully i should be announcing something in the next couple of weeks. There's a little clue in the previous sentence

                K
                Last edited by Kevin; 13 February 2012, 03:41 AM.

                Comment

                • kev_t
                  Automated Home Sr Member
                  • Jun 2004
                  • 58

                  #38
                  Clearly we still have some unresloved issues here

                  The intent was simply to make the existing schema work properly by formalising the existing schema in actual use. Through this useful discussion we have added a number of useful keys and corrected some anomalies as well.

                  The original schema was formulated to support the data available from Metar reports and from the BBC observations and forecasts. In these reports, there is only one engineering unit for Air Pressure and Rain is only mentioned as being light, heavy etc. The Cloud base etc key is a direct derivation of the decoded text from the metar reports

                  Two Weather station software vendors were convinced to support xAP, which is great, and modified the schema to suit their systems and the users who requested the support. While not ideal, to me this modification of the schema is quite legitimate and mirrors other schemas which have been similarly amended to suit the equipment being supported by the software authors, for instance like that for Caller ID/Telephony.

                  In the past developers have created schema to support the equipment or service they were attempting to introduce into the xAP world and have based that primarily on the data their sources were capable of providing. The xAPBSC schema has helped to reduce the need for specialist schema, but it is often a square peg in a round hole sort of fit just to use xAPBSC.

                  Since this schema is not a formal xAP schema like xAPBSC, and the draft xAPTSC, and is not supported by the formality of a specification or any sort of semi-formal or formal review or revision process, then vendors should continue be free to develop new and existing schema like this following the spirit of what has gone before and the basic xAP protocol definition.

                  In my view, not to do this will limit the ongoing viability of xAP and will remove the flexibility that developers have enjoyed up until now.

                  In the case of the weather schema we are discussing here, we have the weather.report class which is for current data. I would have hoped that we could have formalised the weather.forecast, Time and request classes that James used on his weather app and created a new class for history as well.

                  Speaking for myself, I don't have the time or the inclination to develop a brand new schema for weather from scratch, but feel that the developed schema we have here meets the requirements. I am not in any case convinced that such a brand new schema would get any take up from vendors.

                  In the schema I will summarise in the next post and in an attempt to remove traffic which is not required or going to be used by a user, then keys for which there is no data available should not be included in the message and the vendor should consider giving the user the option to select which key should be used where there are alternates, eg TempC and TempF. The user will then be able to configure his xAP display and data collection systems etc to his needs and the data sources he has installed.

                  I have specifically not included in the draft schema keys for Rain or Airpressure, ie without the conventional (for this schema) suffix. I am sure vendors might want to include them as an interim measure for compatibility but this should be a short term measure only and I see no reason to over-legitimise this by including them

                  I think KevinH is the last of the original xAP team and I am sure that all xAP developers and users appreciate the effort and commitment that KevinH continues puts into this.

                  kevint

                  Comment

                  • kev_t
                    Automated Home Sr Member
                    • Jun 2004
                    • 58

                    #39
                    Draft Weather Schema

                    Draft Weather Schema

                    As descibed in the protocol definition, the message body will be preceded by a header, eg
                    {
                    v=12
                    hop=1
                    uid=FF400200
                    class=weather.data
                    source=vendor.weather
                    }

                    In the message body below, most keys are optional.

                    Keys for which data is not available from the data source in use should not be included.

                    To reduce the message complexity, message size, and to remove data that the end user will not use, the vendor may consider allowing the user to disable keys or key pairs, even if data is available

                    For example :
                    if WindDirD was not required because, for instance, it was non-numeric it would not be included
                    if TempF was not required because temperature in fahrenheit was not appropriate to the localisation it would not be included
                    If the Indoor Humidity and temperature were not required then all 3 could be omitted

                    Previous schema included keys for AirPressure and actual implementations included Rain, but this was not in keeping with the overall approach of the schema. Vendors may wish to keep these keys in the short term for compatibility, but both users and vendors are encouraged to use the format below for these keys.

                    weather.report
                    {
                    UTC=(Time of report in hh:mm format using utc time zone)
                    -- mandatory
                    DATE=(Date of report in YYYYMMDD format)
                    -- mandatory
                    Station=(Station code or identifier eg EGNJ)
                    -- optional
                    Location=(location, city, town, village)
                    -- optional
                    WindM=(Value of wind in mph or "Gusty" if gusty)
                    -- optional
                    WindK=(Value of wind in kph or "Gusty" if gusty)
                    -- optional
                    WindGustsM=(Value of wind gusts in mph)
                    -- optional
                    WindGustsK=(Value of wind gusts in kph)
                    -- optional
                    WindDirC=(Compass heading of wind N|NE|E|SE|S|SW|W|NW)
                    -- optional
                    WindDirD=(Compass heading of wind in degrees)
                    -- optional
                    TempC=(Temperature in centigrade)
                    -- optional
                    TempF=(Temperature in fahrenheit)
                    -- optional
                    DewC=(Dew point in centigrade)
                    -- optional
                    DewF=(Dew point in fahrenheit)
                    -- optional
                    RainM=(Rainfall since midnight in mm)
                    -- optional
                    RainI=(Rainfall since midnight in inches)
                    -- optional
                    Humidity=(Relative Humidity as %)
                    -- optional
                    AirPressureP=(Air pressure in hPa)
                    -- optional
                    AirPressureH=(Air pressure in inHg)
                    -- optional
                    AirPressureI=(Air Pressure Indication Rising/Steady/Falling)
                    -- optional
                    VisibiltyM=(visibility in meters)
                    -- optional
                    VisibiltyY=(visibility in yards)
                    -- optional
                    Cloud=(Overall cloud cover eg "Clear Skies")
                    -- optional
                    CloudM.X=(Cloud type and height in miles ie "Overcast at 5.4M. X increments on each cloud layer)
                    -- optional
                    CloudK.X=(Cloud type and heightin kilometers ie "Overcast at 5.4Km. X increments on each cloud layer)
                    -- optional
                    WindchillC=(Windchill Temperature in centigrade)
                    -- optional
                    WindchillF=(Windchill Temperature in fahrenheit)
                    -- optional
                    InTempC=(Indoor Temperature in centigrade)
                    -- optional
                    InTempF=(Indoor Temperature in fahrenheit)
                    -- optional
                    InHumidity=(Indoor Relative Humidity as %)
                    -- optional
                    Picture=(url to weather/webcam picture)
                    --optional
                    Icon=(The name of the picture to use, icon names from kweather www.kde.org. File names below)
                    -- optional
                    Error=(Download errors either "NoData" if last connection failed or "StationNotFound" if ICAO code isn't valid)
                    -- optional
                    }

                    Possible icon names, higher number signifies more clouds

                    cloudy1
                    cloudy2
                    cloudy3
                    cloudy4
                    cloudy5
                    dunno (usually with "error=nodata")
                    fog
                    hail
                    light_rain
                    mist
                    overcast
                    shower1
                    shower2
                    shower3
                    sleet
                    snow1
                    snow2
                    snow3
                    snow4
                    snow5
                    sunny
                    tstorm1
                    tstorm2
                    tstorm3
                    Last edited by kev_t; 25 February 2012, 10:09 AM. Reason: added icon names/spelling

                    Comment

                    • kev_t
                      Automated Home Sr Member
                      • Jun 2004
                      • 58

                      #40
                      A bit of a bargain

                      For anyone interested, at the moment (feb 2012) Clas Ohlson have the WH1080 USB Wireless Weather Station on special at £59.00, in store only purchase and when I was in there in passing this week they were running a £10 off coupon for purchases over £50



                      The WH1080 is supported by both Weather Display and Cumulus which support the existing xAP Weather Schema

                      Please note that although it looks like you can buy on line, you are only filling in a shopping list you can print out.

                      kevint
                      Last edited by kev_t; 25 February 2012, 10:05 AM.

                      Comment

                      • pete_c
                        Automated Home Jr Member
                        • Feb 2011
                        • 39

                        #41
                        The efforts put forth are sincerly appreciated. Thank you both for your time and effort.

                        I would like to apologize if I misstepped in my approach; I didn't really know how or where to post relating to this endeavor.

                        I've become a Joggler / weather addicted personality afflicted person lately; a bit excesive I guess.

                        The weather station above WH1080 is the same one of two that I purchased; here (in the US) known by many names; one as the Fine Offset. It is a good price. I purchased my Fine Offset here in the US for $66 USD. Very luckily I also was able to purchase a Davis Vantage Vue for a bit over $150 USD here; way discounted as the best price I could find was about $230 USD for same said Davis. Some person (weather forum administrator) is also making custom serial port devices for the Davis for about $15 such that the device can be connected to the Davis Consoles. It has no buffer; but works very well and it's better than paying $200 for the software/HW when all you need is the hardware. Both of the weather instruments were purchased on Ebay and mostly related to when they were offered; both auctions ended at around 3 AM local time.

                        Thinking of taking the Joggler/Cumulus/Fine Offset setup to Florida sometime in the next few weeks to see how it does next to the salty water.
                        Last edited by pete_c; 25 February 2012, 04:49 PM.
                        - Pete C
                        HW-HAI OPII - X10,UPB,Z-Wave,Russound,1-Wire(X4 networks) OmniTouch screens, Intel & Arm Touchscreens (Android, Wintel & Linux)and OmniStat2
                        SW-HSPro - Atom D525/4Gb -W2003SP2 X86 - 18 serial ports via USB (Digi Edgeports & Digi Hub)
                        Weather Cumulus - Fine Offset & Davis Vantage Vue

                        Comment

                        • kev_t
                          Automated Home Sr Member
                          • Jun 2004
                          • 58

                          #42
                          Wiki Access

                          It would appear that I no longer have access to the wiki, so I can progress this no further

                          kevint
                          Last edited by kev_t; 15 April 2012, 07:01 PM.

                          Comment

                          • Kevin
                            Moderator
                            • Jan 2004
                            • 558

                            #43
                            Not something I've changed Kevin - the WiKi has always been a bit of a mystery to me...

                            K

                            Comment

                            • pete_c
                              Automated Home Jr Member
                              • Feb 2011
                              • 39

                              #44
                              Something odd here. I cannot get to the protocal definition part of the wiki anymore.
                              - Pete C
                              HW-HAI OPII - X10,UPB,Z-Wave,Russound,1-Wire(X4 networks) OmniTouch screens, Intel & Arm Touchscreens (Android, Wintel & Linux)and OmniStat2
                              SW-HSPro - Atom D525/4Gb -W2003SP2 X86 - 18 serial ports via USB (Digi Edgeports & Digi Hub)
                              Weather Cumulus - Fine Offset & Davis Vantage Vue

                              Comment

                              • Kevin
                                Moderator
                                • Jan 2004
                                • 558

                                #45
                                It's OK here for me Pete...

                                K

                                Comment

                                Working...
                                X