Cut to the Idratek bus, and whole system stop

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts
  • marcuslee
    Automated Home Ninja
    • Dec 2009
    • 279

    Cut to the Idratek bus, and whole system stop

    Commissioning an Idratek system - Cortex with QLD dimmer DIN modules and some normal pattress modules which are yet to come online.

    Unfortunately someone cut a Cat5 lead, which was connected to the Idratek bux.

    System came to a halt, and at the time all the lights went out in the house.

    Cortex report a Red dialogue box error which at the time which I believe said communication could not continue until a connection could be established, without a OK or Cancel button.

    In haste to get the house (full of tradesmen) operational again, a quick check of IPD looked a bit weird not normal-ish, though I have to say I've not really paid too much attention to the IPD LEDs as it's always just worked, so I did a quick figure out of which bus the cut cat5 was on, pulled the plug, did a power button off on for the IPD, and shutdown and restarted Cortex app (I'd forgotten about dialogue boxes in Cortex doesn't necessarily mean you can't still go back to the Cortex window).

    And it was all on it's way again (except for the report these nodes as faulty standard Cortex message).

    I've yet to do the research on what exactly happened to have it go out so spectacularly, however anyone else done this (as I'm sure it's not too uncommon)?

    Cheers,

    Marcus
  • chris_j_hunter
    Automated Home Legend
    • Dec 2007
    • 1713

    #2
    all our many upsets have been due to compromised cables causing spur fuses to blow (so, plenty of messages due to unresponsive modules), or modules being plugged-in prematurely (various issues arising, including non-unique addresses), or modules being unplugged mistakenly (more unresponsiveness) ...

    never, SFAICR, have we had communication can't continue until' messages ...

    and, when lights have gone out (many times), it's always been due to due-process in starting-up the network (ie: stopping the network freezes the state of things, while starting it up invokes chosen / default start-up states - commonly lights off) ...

    so, not sure what could have caused what you experienced, exactly !

    NB: no dimmer modules in our set-up, yet ...
    Last edited by chris_j_hunter; 4 March 2012, 01:17 AM.
    Our self-build - going further with HA...

    Comment

    • Karam
      Automated Home Legend
      • Mar 2005
      • 863

      #3
      Marcus,

      I'll have to look up the exact detail of such a warning, but essentially if a cable is cut you can expect that the cutting process or even state of cable afterwards is likely to create shorts between internal cores. This means you could have caused a power bus short or databus short or both. Obviously one would hope that this is not likely to happen in a normally running system ... however if it does then it will invoke some guard actions and recovery attempts. Some of what happens will also depend on the settings in the IPS Management settings (where there are options for how you want the IPS to deal with faults e.g automatic or not - it is not always the case that you don't want obvious signs for user intervention). But for example if a bus fault is detected then an automated reaction would be for the IPS to first shut down all spurs and then power them up one at a time checking each one as it goes along and re-isolating one/s that it finds faults with during this process. The initial shutdown in such a situation will of course cause your lights to go out because dimmers are powered from the network and for relay controlled lights the usual state is that lights are off when relay de-energised.

      So the network should recover (other than faulted spur) in its own time, but as I say depends also on management settings.

      Nonetheless if you can let us know state of the settings and provide the log file around the incident we can have a look to see if there is anything further we can learn.

      Comment

      • marcuslee
        Automated Home Ninja
        • Dec 2009
        • 279

        #4
        Thanks for reply Chris

        Originally posted by Karam View Post
        Marcus,

        I'll have to look up the exact detail of such a warning, but essentially if a cable is cut you can expect that the cutting process or even state of cable afterwards is likely to create shorts between internal cores. This means you could have caused a power bus short or databus short or both.
        Looks like it was prob both (was a clean cut with a set of snips).

        Originally posted by Karam View Post
        Obviously one would hope that this is not likely to happen in a normally running system ... however if it does then it will invoke some guard actions and recovery attempts. Some of what happens will also depend on the settings in the IPS Management settings (where there are options for how you want the IPS to deal with faults e.g automatic or not - it is not always the case that you don't want obvious signs for user intervention). But for example if a bus fault is detected then an automated reaction would be for the IPS to first shut down all spurs and then power them up one at a time checking each one as it goes along and re-isolating one/s that it finds faults with during this process. The initial shutdown in such a situation will of course cause your lights to go out because dimmers are powered from the network and for relay controlled lights the usual state is that lights are off when relay de-energised.

        So the network should recover (other than faulted spur) in its own time, but as I say depends also on management settings.

        Nonetheless if you can let us know state of the settings and provide the log file around the incident we can have a look to see if there is anything further we can learn.
        Got it, and indeed since then I've had a read and checked a few of the IPD settings. For some reason I get the impression though that the IPD didn't resequence the Spurs. I say this, as it took me a little while (in the dark, with only the glow of the laptop screen), for me to figure out which cable as cut, then to lookup the diagram to figure out which spur it would be plugged into, and then to know which spur to unplug.

        Though having said that, now that I've become more familiar with how long the system takes to power up (having done it a few more times now), I would guess it's roughly the same time it would take to repower all spurs vs the time I took to take remedial actions.


        On another note though now, I should check the IPD fuse (something I didn't think of doing, as I'm used to knowing a fuse has gone by complete disintegration on the QLD!), as since then, having had time to look at it (without a house full of people) Spur 1 is definitely dead (enabled in Cortex, but showing 0mA draw).

        However I did a swap out to the spare IPD module we have, and it leads me to some other questions:
        - I was apprehensive about swapping it out, given that I wasn't sure how this would go, especially given the criticality of it, however I did it anyway
        - I was aware it should have the same Node ID assigned, having re-read the IPD-001 commissioning notes (I was finger crossing that Cortex might just not notice the swap over)
        - however upon swap in, it couldn't detect the IPD (only faulted out the old IPD). Luckily though the new IPD continued to function, and everything is back online, except there is no IPD listed.

        I've also confirmed the old IPD was in Cortex as 0500.

        Additionally no new node object wizard came up, so the new IPD is there like a ghost.

        Any advice on what best to do to have it see the IPD?

        Also even if not seen, and the fact the IPD still functions, does it mean IPD battery backup also is functioning?

        Finally Karam I remember you mentioned once about having dual IPDs in a system to cater for module load, is there any tech info on how wire this in and get it running?

        As it stands, all 4 spurs are nearly max loaded to 1A, but with this event, I think the design could be better improved, with the load halved across 2 IPDs; to mitigate a complete black out, and also in worst case scenario, it could be a simple re-patch of the dead IPD's spurs, to the equivalent spurs on the other IPD?

        Comment

        • Viv
          Automated Home Ninja
          • Dec 2004
          • 284

          #5
          With the network stopped.
          Communications tab selected.
          In 'Enter PCIF command code' box.
          Type
          FA91000C440003
          Followed by return or transmit popup.

          You should expect a response something like...

          16:11:58:296 > World
          16:11:58:296 > FA0691000C440003
          16:11:58:328 < 180022013B
          16:11:58:343 < 18000C0125
          16:11:58:390 < 900B1024880B080627000A00A1
          16:11:58:390 < House Power Supply 1024 (Version)

          The packet FA91000C440003 sends a broadcast request for all power supplies on network to respond with their version.

          The response
          16:11:58:390 < 900B1024880B080627000A00A1
          shows an address 1024

          Viv

          Comment

          • Karam
            Automated Home Legend
            • Mar 2005
            • 863

            #6
            Regarding the fuse there is a slight difference (er, well, more than slight) between blowing out on a full mains short circuit as compared to an IDRANet bus short - probably a few hundred amps difference

            So as you say looks like you at least had a power bus short but probably also a databus one.

            I think swapping the fuse might have been easier than the whole IPD . But anyway as you say you'd expect Cortex not to notice if the new unit had the same ID, so what I'm guessing is that you may have done the swapping with Cortex still launched or maybe even still running the network? Was this the case?

            What I suggest you try is stop the network (but leave DC power on) and then close down and relaunch Cortex.

            If you have set up an auto switch over to Reflex watchdog for the previous IPD then you will have to program this into the swapped in IPD. The Reflex code is retained in Cortex so its just a case of visiting the Reflex menu for the IPD object and clicking on program this node - this needs to be done with networked stopped but don't worry Cortex won't allow you to do it when the network is running anyway.

            There should be no adverse effect on the battery charging and backup switch over as this is carried out by an independent circuit.

            Yes I'll dig up the wiring document for the dual IPD.

            Comment

            • marcuslee
              Automated Home Ninja
              • Dec 2009
              • 279

              #7
              Thanks Viv, Karam for the prompt reply.

              I'll have to find the time to perform a scheduled maintenance (prob this weekend) and will give it a go.

              Cheers,

              Marcus

              Comment

              • chris_j_hunter
                Automated Home Legend
                • Dec 2007
                • 1713

                #8
                not sure I'm understanding all this ...

                OK, the message about being unable to communicate was probably due to just one spur having been active, then cut ...

                but why should the IPD go invisible, though still functioning ?

                Presumably both old & new IPDs had the address 0500 ... ?

                BTW, we're running two IPDs, to keep power demands well within bounds - works fine, even with fuses blowing causing upsets, recovery is straightforward ...

                ditto the business of modifying IDs, when have duplicates that need to be spilt - ie: easy to sort, too ...

                presumably IPDs would be no different ... ?
                Last edited by chris_j_hunter; 6 March 2012, 12:03 PM.
                Our self-build - going further with HA...

                Comment

                • Viv
                  Automated Home Ninja
                  • Dec 2004
                  • 284

                  #9
                  Originally posted by chris_j_hunter View Post
                  not sure I'm understanding all this ...
                  If you cut through the network cable and you get a message saying the PC is unable to communicate it will be because of a database short. A short on the database clamps the entire network. That is why the IPS will shutdown and then restore spurs one at a time checking for shorts. If you have a short on the always on spur which is designed to be connected to the PCA/PCU and perhaps the network reset module then their is nothing the IPS can do and requires the short to be removed.

                  The IPD units are shipped pre-programmed to 0500. Replacing one with a spare which is used as supplied will mean it will work without messages being reported to re-commision.

                  Note the reflex programming may need re-doing.

                  From experience I would not want to load a spur more than say 700ma in startup state. That is relays not energised, intercoms not on etc. This is to allow an increase when various devices are activated. But you can always view your history for the IPS to check its perfomance.

                  Viv.

                  Comment

                  • chris_j_hunter
                    Automated Home Legend
                    • Dec 2007
                    • 1713

                    #10
                    Thanks - interesting ...

                    so far, our spurs are all under 350ma, half of them under half that, and that’s with some relays on (for lights & door interlocks & pump power supplies, a few being on 24/7, most being on for good parts of the day, every day) ...

                    >network reset module ...

                    hmm, what’s that, is it something we should be including ??
                    Our self-build - going further with HA...

                    Comment

                    • Gumby
                      Moderator
                      • May 2004
                      • 437

                      #11
                      Originally posted by chris_j_hunter View Post

                      >network reset module ...

                      hmm, what’s that, is it something we should be including ??
                      DBI - can be configured to make the two buttons toggle between Cortex driven and Reflex driven, so gives a very user-friendly emergency interface to force Reflex mode on.
                      ----------------------
                      www.gumbrell.com

                      Comment

                      • Gumby
                        Moderator
                        • May 2004
                        • 437

                        #12
                        Originally posted by marcuslee View Post
                        Unfortunately someone cut a Cat5 lead, which was connected to the Idratek bux.
                        Commissioning during building is a fraught time. It's worth thinking about putting in temporary light switches instead of modules so that your modules are not exposed to damage.

                        You could also hack up a simple adaptor for cheap 'Mod-Tap' style Cat5 testers such that you could test at least continuity and wiring on all your legs before replacing cables gets very difficult. You obviously get two wires marked as either shorted or not connected, but you can learn to ignore that pattern.

                        You might also consider using these if you install modules before painting: http://safetyshieldproducts.co.uk/index.php. Cheap shower caps are a start, but these stop crap getting in the back of modules.
                        ----------------------
                        www.gumbrell.com

                        Comment

                        • chris_j_hunter
                          Automated Home Legend
                          • Dec 2007
                          • 1713

                          #13
                          we've found Idratek's resistor connector invaluable when trouble-shooting wiring faults ...

                          so far, we've had only bad connections - just a handful, all trapped insulation, and ISTR always one of the blue pairs, so signal rather than power !

                          solid-core STP might be expected to be vulnerable to breakage during initial install & after, if have to open-up a few times - but, so far, we've suffered none ...

                          those Safteyshields look a good idea, 'though we'd need rather a lot of them & we've been installing / commissioning / screwing-down as building work proceeds, so installing them ahead of painting might be a pfaff & perhaps risk cable breakages ...
                          Our self-build - going further with HA...

                          Comment

                          • marcuslee
                            Automated Home Ninja
                            • Dec 2009
                            • 279

                            #14
                            I also concur with the resistor method being a very clever and excellent way of figuring out shorts etc.

                            Regarding update on original issue:
                            - confirmed that the fuse blew on the IPD only (and I could have done a fuse swap).
                            - however to perform a fuse swap, is it advisable to uncheck the enable spur checkbox or even turn off the IPD S+/S-, or can a fuse be done immediately and in-situ?
                            - Looking at the settings for the original IPD in Cortex as it stands now, reveals that indeed:
                            - no Auto Associate was made to nodes (and no auto fault nodes on Spur trip enabled)
                            - no Auto spur trip recovery enabled
                            - which would explain why it was auto-recovered (the above not enabled yet as a few nodes still to be added).
                            - also though after Auto Associate I take it, it's possible to either delete or re-Auto Associate later if we juggle the nodes around?

                            Regarding the new IPD:
                            - I did try Stop network, Close Cortex, Restart Cortex, however it didn't detect the new IPD
                            - I should note whilst doing the above, I decided to leave one of the QLD dimmer channels on to see the effect, and it remained lit until the restart of Cortex
                            - I issued the commands to find the new IPD, and it came back with 0504 ( however I'm sure it was never configured since it was received)
                            - knowing the NodeID I then did "Design Network > Add Idratek Network Object > IPS", and then network enabled it
                            - everything looked good, and it showed current being taken across all Spurs, but I note none of the Spurs was showing with the enabled checkbox
                            - I checkbox enabled all of them, whereupon the IPD recycled each Spur and brought it all back online


                            So it leaves the question, what is the best configuration for this? I believe it may be:
                            - "Auto Associate" all Spurs
                            - "Automatically fault nodes on spur trip" enable
                            - but I don't really want "Automatic spur trip recovery" enabled
                            - also probably don't want "Spur n time before re-powering allowed" enabled either

                            or should the IPD should have re-powered all Spurs anyway? I remember distinctly at time of event, red LED being lit for Spurs 2-4 (not 1, where it was blown), no other LEDs light like they would be normally, and that was about it. And I'm sure I did a Cortex Stop, Start, and possibly even a S+/S- off, on, though i can't remember the order.

                            Also, any info on how to wire dual IPDs?

                            Cheers,

                            Marcus

                            Comment

                            • chris_j_hunter
                              Automated Home Legend
                              • Dec 2007
                              • 1713

                              #15
                              a lot of that sounds familiar (!) ...

                              wiring dual IPDs is quite straight-forward, effectively they link in-parallel, with no 12V between them, IIRC - I could dig-out instructions, but things may have changed since we did ours, so better not ...

                              BTW, we’re about two-thirds installed (*) and, apart from the IPD fuse-blowing issues & a few wiring hic-cups, and few small things with DRBs (**), we’ve really only had two problems - presence not working (because our build & installation is still too incomplete), and the knock-on effects that has, and some difficulties with analogue modules (***) ...

                              how's yours been going, otherwise ?

                              Chris

                              (*) it’s been that way for a while, because we kept adding to our list, but hopefully we’ve now come to the end of that ...

                              (**) all our DRBs came with 1AF fuses, some of which we’ve had to swap for 4AFs and, for some reason, it’s always been DRBs that have been involved when we’ve got muddled at plug-in time - meaning we’ve now a small pile of them waiting to have their non-unique IDs reset !

                              (***) adding the (we didn't understand were) required 10kΩ resistors sorted the QAIs, but the QAOs are still causing us grief ...
                              Our self-build - going further with HA...

                              Comment

                              Working...
                              X