Odd XW AGS problem

twistedtree · October 2016

I've just noticed a very odd AGS issue with my Xantrex/Schneider XW system. System consists of an XW6048, 2x MPPT60, SCP, ComBox, and AGS.

Over the past week or so, I have happened to be in the same room as my generator, and noticed that it occasionally begins an uncommanded start cycle, but only for 1 to at most 2 seconds, then stops. When I say "uncommanded", I mean that none of the start criteria have been met that would normally cause the generator to start. If I start and stop the generator via the SCP/AGS using the Manual On and Manual Off settings, it works exactly as expected, and exactly as it has for the past 7-8 years.

It's a diesel generator, so the start cycle begins with ignition on and pre-heat on, normally for about 10 seconds before the starter is engaged. When this funky behavior happens, I hear a click, the fuel pump activates, and the pre-heat activates - all normal for a start cycle - but it then terminates after maybe 1 second. Their are no errors or warnings kicked off by any of the XW components, and no apparent start criteria has been met to trigger the AGS. I've hung around a couple of times when this has happened, and it has repeated itself, sometimes after a minute or two, and sometimes right away, and sometimes not at all (or at least as long as I waited).

This system has been in service for 7-8 years and I have never noticed this before. But the generator is in a barn and I honestly don't spend a lot of time in this particular room. So I suppose it might have always been doing this and I just didn't notice, but I think that's really unlikely since over the past week I have noticed it several times when I happened to be passing through the room. So I think it most likely has started recently.

What's changed in my system coincident with the emergence of this new phenomenon, is the replacement of a Xantrex Gateway with a Conext ComBox. Because these changes are coincident, it seems the most likely cause. But I can't for the life of me understand how installing a ComBox in place of a Gateway could cause such a thing.

Has anyone ever seen a problem similar to this? Anyone have any ideas? I'm scratching my head, and there isn't much hair left.

Dave Angelini · October 2016

The only way that is easy to do would be to switch back to the gateway.
The COMBOX really can't do this as you reasoned unless it is is at the end of its 130 feet system length and corrupting data.
I think that you just did not notice before, which to me is kind of scary with an AGS.
I need to use them sometimes but really discourage them with remote equipment installations.
You might be getting a warning from the big guy to be careful. I just did an insurance claim for a guy who came home to a burned down offgrid home. Good Luck!

twistedtree · October 2016

Thanks. From what I know about canbus, corrupted data will never generate false command. Data packets are protected by a CRC, so corruption would have to both generate a valid (but incorrect) command AND generate a correct corresponding CRC. That would never happen. And to begin the start sequence and then stop it would seem to require multiple incorrect, yet logically related packets, all with correct CRCs. Usually data corruption would result it bad packet reports, but I don't know if any of the XW gear reports those. I suppose it's also possible that the XW products just ignore CRC errors, but that would be pretty careless of them. Any chance you know how they do it?

That said, I have a confession to make. My bus is longer than 130'. But it's been 100% reliable for the past 7 years - at least in all ways I can see.

The more I think about it, this has to be a new problem. For me to happen to overhear it multiple times over a week, but to have never heard it previously over the preceding 7 years seems super unlikely.

I've ordered a counter that I can wire into the start circuit to count how often it happens. I think that will aid in debugging this. One test will be to swap back in the Gateway to see what happens. That seems like the best next step.

As for having autostart in the first place, I'm afraid it's necessity at this site. It can be unattended for stretches where the solar can't keep up, and some of the occupants aren't operationally savvy enough to monitor and run it manually when they are there. The AGS has worked great, save for this issue. But I agree it's not without risk.

Dave Angelini · October 2016

I definately know how Schnieder does it but I am a hardware and offgrid enabler and too busy to care. It is all on their site as they went open source about 3 years ago. Check it out. An online education is in their support pages.

I have been beta testing for them Schneider/xantrex/trace and know that bad things happen when xanbus bus gets too long. They even use comboxs in a master/slave config for getting more devices in very large systems. They have some very nice surprises in the pipeline. I can tell you that the XW system is here for a very long time and the XW + series has room to grow even at 11 years of maturity.

Have you looked at your system on the Conext Insight? Very safe way to look in from outside the firewall anywhere in the world. I have a client that I monitor their unskilled users. They too have an AGS but not because of limited solar resource. Their issue is limited intelligence and tons of money

mike95490 · October 2016

Did you properly install the little "Terminator" plug at both ends of the data bus? That stops signal reflections in the data line.

twistedtree · October 2016

OK, thanks for the pointers. It will be a couple of weeks before I get back there and can diagnose further, but will let everyone know what I find. As for terminators, yes, they are correctly installed at both ends, with the AGS+term at one extreme end of the bus.

My Xanbus experience is limited to installing and using this system, so I welcome hearing how it behaves under adverse conditions. I just estimated my xbus length and it's probably 250', so clearly over the 130' allowed. Maybe this has become the problem with the introduction of the ComBox in place of the gateway. It's certainly cause for suspicion. But all my non-xbus CAN experience says it would result in packets that have detected errors and get dropped.

If it is a result of the long bus, does anyone make a repeater? The bus is well contained in the house with all equipment within probably 30-50' of each other. Then there is a long run out to the barn where the gen and AGS are installed. That's what stretches the bus.

Dave Angelini · October 2016

It is usually easier to bring the ethernet with a repeater out to near the 130 foot limit. That is what I have done or just bury an ethernet cable.
You may be right about the packets, detected errors, and drops. I just take exception anytime that someone says "always"
Always is a very big statement that often misses unknown exceptions.

twistedtree · October 2016

Dave Angelini said:

It is usually easier to bring the ethernet with a repeater out to near the 130 foot limit. That is what I have done or just bury an ethernet cable.

I'm not quite following what you are saying. Right now I have Cat5e cable running from the ComBox (last device inside the house, and well within the 130' limit), out to the barn and to the AGS which is at the extreme end of the Xanbus. This run to the barn and AGS is what stretches the bus length beyond the 130' limit.

Can you say more about what you are suggesting?

Short of adding some sort of xanbus repeater into the system, the only solution I see to get the bus length down to 130' or less would be to move the AGS into the house and terminate the xbus there, then use the Cat5 to the barn to carry the various relay circuits out to the generator. It might also require some remote-relays to limit the power draw over the long Cat5 run since it's light gauge wire.

And BTW, I agree about "always". It's a dangerous word. Something clearly isn't working right with my system, so one of those unknown exceptions seems to have crept up.

Dave Angelini · October 2016

If you think about the set-up you have and the much greater length limit of Ethernet you will figure it out.
My advice is of a general nature. When it becomes specific I am billing and there are plenty of others here who this is a hobby for.
Think about it and you will figure it out! Good Luck. I do plenty of pro bono, but not much on this continent...

twistedtree · November 2016

I'm back working on this AGS problem. As a recap, I get periodic "abbreviated phantom start attempts" at the AGS/generator. It's abbreviated in that the start sequence begins (ignition on, pre-heat on), but lasts for no more than a second and stops. It's a phantom start attempt because there is no start criteria that has been satisfied that would cause a start. When there is a valid start criteria, the generator starts and runs fine. And it responds to manual start and stop commands as issued via the SCP. This system has been working fine for the past 8 years, but I recently swapped out an XW Gateway for a Combox. The Combox appears to be working correctly, but this problem appears to have started coincident with installation of the Combox, so it seems to be a piece of the puzzle.

Because these phantom starts only last a second, and seem to be randomly spaced, the only way I notice them is when I happen to be in the same room in the barn where the generator is housed. To get a better handle on how often this is happening, I got a little electronic counter and wired it to the ignition circuit. Now the counter increments each time the ignition circuit is energized.

For the first couple of days, AGS is at the far end. All devices are powered off the Xanbus, and one possibility is that the Combox draws more power than the Gateway, and there is now marginal power at the AGS. So my first experiment has been to install the external power adapter for the Combox so it is powered externally rather than via the Xanbus. This experiment is ongoing now, so we will see what happens.

Next up will be to remove the Combox and reinstall the Gateway and see if the problem goes away.

Oh, and I noticed something else odd. I looked at the event log on the Combox today and it says that it rebooted at 03:05 this morning in response to a reboot command. WTF? Where could that be coming from?

mike95490 · November 2016

is your combox wired to the internet, or stand alone ? There is a buried hidden setting that takes months to overflow and cause errors and reboots.

Dave Angelini · November 2016

There also is a daily reboot sanity check that combox does.

twistedtree · November 2016

Dave Angelini said:

There also is a daily reboot sanity check that combox does.

OK, maybe that's all I'm seeing. If so, then the reboot is normal and has nothing to do with the phantom start attempts.

mike95490 · November 2016

Wow, I'm pretty sure my combox does not do a daily reboot, or is that an optional feature?

Dave Angelini · November 2016

mike95490 said:

Wow, I'm pretty sure my combox does not do a daily reboot, or is that an optional feature?

Pretty sure or positive? Happy Turkey Day Mike!

mike95490 · November 2016

Just pretty sure, I'm never there at midnight to watch the lights. I've seen it crash and reboot at all sorts of other times,
Is there a daily reboot setting, or is it a hard coded function ?

Dave Angelini · November 2016

If you are running the latest firmware, there should not be any of that unless it is your set-up interfering.
The testing I am doing now, they always tell me not to update firmware they send me at or about 3:05 AM.
That will never happen for me.....
Have you tried the Conext Insight which uses combox?

Image: https://us.v-cdn.net/6024911/uploads/editor/eq/e7gxmk2klmjy.png

mike95490 · November 2016

I have no data connection, except via cell phone as a hotspot. So I cannot connect to the insight. But insight is enabled by default and the error messages caused a Combox lockup every 4 months when the uSD card root directory overflowed !
Now that it's been OFF for a couple months, I'm looking forward to not having it crash on me.
So it goes into maint mode daily about 3am.

Dave Angelini · November 2016

I forgot Mike about the hotspot. I see your problem.
I notice the maint mode on an SCP that is not on the same screen I left it in at lights out time, the next morning.
It took our ISP 7 years and two broken bulldozers to get up to us so I understand.

twistedtree · November 2016

I think I'm zeroing in on the problem. Providing external power, and resetting the AGS to factory defaults and reconfiguring it made no difference either.

But when I look at the AGS status via the Combox, it says that the Stop Reason is Exercise End. But the generator hasn't exercised. But now suspecting something related to the exercise function, I started looking more closely at when the phantom start attempts occur, and they are all within the exercise time window. I have it set to exercise every 21 days at 09:00 for 60 minutes, so from 09:00 to 10:00. All the phantom start attempts occur in this window. Note that the exercise function has always worked correctly, save these phantom start attempts.

So at this point I think I have found a bug in the AGS. My next experiment will be to see if I can cause the phantom starts to happen by setting the exercise time and duration to something different. If the phantom starts happen in that new window, then I think I have figured out how to reproduce the problem. I'll also be away for a week or two, and plan to disable the exercise function while gone to see if the phantom starts cease, which I expect they will.

mike95490 · November 2016

> I think I'm zeroing in on the problem.
Awesome, writing bug free code is a lot tougher than it looks. At least it's not a bug that hits the glow plugs only and drains the battery.

twistedtree · November 2016

mike95490 said:

> I think I'm zeroing in on the problem.
Awesome, writing bug free code is a lot tougher than it looks. At least it's not a bug that hits the glow plugs only and drains the battery.

Oh, but it does hit the glow plugs. The ignition-on sequence activates them, though only for a second. The generator ended up with a dead battery a couple of years ago, and I think I now know why......

Dave Angelini · November 2016

This set-up did work correctly though so the AGS code did work.
Hard to believe that a code problem could do that. Not impossible but hard to believe.
For myself as one who does not use a generator, easy for me to believe that the AGS has been surged one too many times from generators or that something is getting into the xanbus and the really long cable run.

twistedtree · November 2016

Dave Angelini said:

This set-up did work correctly though so the AGS code did work.
Hard to believe that a code problem could do that. Not impossible but hard to believe.
For myself as one who does not use a generator, easy for me to believe that the AGS has been surged one too many times from generators or that something is getting into the xanbus and the really long cable run.

I can't say for 100% sure that this problem hasn't been present all along. It just seems unlikely that I never noticed it before. But it's possible.

As for other causes, it's certainly possible. This is just the first time that I've been able to narrow it down in any definitive way, so it seems the most promising explanation - at least so far.

Earlier this evening I disabled the exercise function. If the current theory holds, I would expect tomorrow to be free from phantom starts. It will be interesting to see what actually happens.

One experiment on my list is to remove the Combox and replace with the old Gateway, thereby putting the system back the way it was before I first noticed the phantom starts. It will be very interesting to see if the problem goes away when I do that. I can't offer any explanation for why installing the Combox would initiate this problem, but it does correlate with when I first noticed the problem, so seems a likely factor. But right now I'm making progress (I think) in making the problem reproducible which is essential to debugging a problem. So I'm going to keep on this path for now.

Stay tuned...

twistedtree · November 2016

After disabling the auto exercise feature, the phantom start attempts have stopped.

I finally found the support web page for this product, which is impossible starting at Schneider-electric.com, but google is my friend where the manufacturer otherwise fails. It looks like my AGS firmware is out of date, and although there is no mention of this issue in the release notes for the updates, updating seems like the next logical step. But at this point I'm pretty convinced this is an AGS firmware bug, and nothing to do with my long Xanbus or the addition of the Combox.

Dave Angelini · November 2016

The AGS has been out there a long time and the Engineers at Schneider use this device quite a bit.
If you have not updated the firmware.....
When you search, always add Solar to the Schneider search. The division is Schneider Electric Solar part of the 29 billion euro Schneider Electric

mike95490 · November 2016

at least you have the combox, and it can update firmware

Dave Angelini · December 2016

Combox will get new firmware soon (2 weeks) Mike. Might help your problem for someone else.

mike95490 · December 2016

Dave Angelini said:

Combox will get new firmware soon (2 weeks) Mike. Might help your problem for someone else.

Cool. I'll wait 2 more weeks for bug reports ! only problem was Insight was ON by default and having no internet, it filled up uSD card with error msgs

twistedtree · December 2016

mike95490 said:

at least you have the combox, and it can update firmware

That's what I thought....

Here's another update. It's been about a month and I'm back at this system. My generator counter advanced 6 times while I was away, corresponding to 3 run cycles. It counts both at the start and stop transition, hence the double count. The good news is there were zero phantom start cycles, so my issue has clearly been a bug in the AGS. Disabling exercising stopped the phantom start cycles.

Now the next problem. I downloaded the latest AGS firmware (V3.01), but the ComBox won't apply it to the AGS. I am able to upload the .xfo file to the ComBox successfully. That part seems to work. But when I select the AGS to update, then click the drop-down for a list of files to pick from, there are none. The file I just uploaded isn't listed.

Anyone have any ideas?

mike95490 · December 2016

use the method to load the file on a USB drivce, and load it from there, Using web interface may not work, I've never tried it, always from the USB on the combox

Odd XW AGS problem

Comments