Odd XW AGS problem

I've just noticed a very odd AGS issue with my Xantrex/Schneider XW system. System consists of an XW6048, 2x MPPT60, SCP, ComBox, and AGS.
Over the past week or so, I have happened to be in the same room as my generator, and noticed that it occasionally begins an uncommanded start cycle, but only for 1 to at most 2 seconds, then stops. When I say "uncommanded", I mean that none of the start criteria have been met that would normally cause the generator to start. If I start and stop the generator via the SCP/AGS using the Manual On and Manual Off settings, it works exactly as expected, and exactly as it has for the past 7-8 years.
It's a diesel generator, so the start cycle begins with ignition on and pre-heat on, normally for about 10 seconds before the starter is engaged. When this funky behavior happens, I hear a click, the fuel pump activates, and the pre-heat activates - all normal for a start cycle - but it then terminates after maybe 1 second. Their are no errors or warnings kicked off by any of the XW components, and no apparent start criteria has been met to trigger the AGS. I've hung around a couple of times when this has happened, and it has repeated itself, sometimes after a minute or two, and sometimes right away, and sometimes not at all (or at least as long as I waited).
This system has been in service for 7-8 years and I have never noticed this before. But the generator is in a barn and I honestly don't spend a lot of time in this particular room. So I suppose it might have always been doing this and I just didn't notice, but I think that's really unlikely since over the past week I have noticed it several times when I happened to be passing through the room. So I think it most likely has started recently.
What's changed in my system coincident with the emergence of this new phenomenon, is the replacement of a Xantrex Gateway with a Conext ComBox. Because these changes are coincident, it seems the most likely cause. But I can't for the life of me understand how installing a ComBox in place of a Gateway could cause such a thing.
Has anyone ever seen a problem similar to this? Anyone have any ideas? I'm scratching my head, and there isn't much hair left.
Over the past week or so, I have happened to be in the same room as my generator, and noticed that it occasionally begins an uncommanded start cycle, but only for 1 to at most 2 seconds, then stops. When I say "uncommanded", I mean that none of the start criteria have been met that would normally cause the generator to start. If I start and stop the generator via the SCP/AGS using the Manual On and Manual Off settings, it works exactly as expected, and exactly as it has for the past 7-8 years.
It's a diesel generator, so the start cycle begins with ignition on and pre-heat on, normally for about 10 seconds before the starter is engaged. When this funky behavior happens, I hear a click, the fuel pump activates, and the pre-heat activates - all normal for a start cycle - but it then terminates after maybe 1 second. Their are no errors or warnings kicked off by any of the XW components, and no apparent start criteria has been met to trigger the AGS. I've hung around a couple of times when this has happened, and it has repeated itself, sometimes after a minute or two, and sometimes right away, and sometimes not at all (or at least as long as I waited).
This system has been in service for 7-8 years and I have never noticed this before. But the generator is in a barn and I honestly don't spend a lot of time in this particular room. So I suppose it might have always been doing this and I just didn't notice, but I think that's really unlikely since over the past week I have noticed it several times when I happened to be passing through the room. So I think it most likely has started recently.
What's changed in my system coincident with the emergence of this new phenomenon, is the replacement of a Xantrex Gateway with a Conext ComBox. Because these changes are coincident, it seems the most likely cause. But I can't for the life of me understand how installing a ComBox in place of a Gateway could cause such a thing.
Has anyone ever seen a problem similar to this? Anyone have any ideas? I'm scratching my head, and there isn't much hair left.
Comments
The COMBOX really can't do this as you reasoned unless it is is at the end of its 130 feet system length and corrupting data.
I think that you just did not notice before, which to me is kind of scary with an AGS.
I need to use them sometimes but really discourage them with remote equipment installations.
You might be getting a warning from the big guy to be careful. I just did an insurance claim for a guy who came home to a burned down offgrid home. Good Luck!
htps://offgridsolar1.com/
E-mail [email protected]
That said, I have a confession to make. My bus is longer than 130'. But it's been 100% reliable for the past 7 years - at least in all ways I can see.
The more I think about it, this has to be a new problem. For me to happen to overhear it multiple times over a week, but to have never heard it previously over the preceding 7 years seems super unlikely.
I've ordered a counter that I can wire into the start circuit to count how often it happens. I think that will aid in debugging this. One test will be to swap back in the Gateway to see what happens. That seems like the best next step.
As for having autostart in the first place, I'm afraid it's necessity at this site. It can be unattended for stretches where the solar can't keep up, and some of the occupants aren't operationally savvy enough to monitor and run it manually when they are there. The AGS has worked great, save for this issue. But I agree it's not without risk.
I have been beta testing for them Schneider/xantrex/trace and know that bad things happen when xanbus bus gets too long. They even use comboxs in a master/slave config for getting more devices in very large systems. They have some very nice surprises in the pipeline. I can tell you that the XW system is here for a very long time and the XW + series has room to grow even at 11 years of maturity.
Have you looked at your system on the Conext Insight? Very safe way to look in from outside the firewall anywhere in the world. I have a client that I monitor their unskilled users. They too have an AGS but not because of limited solar resource. Their issue is limited intelligence and tons of money
htps://offgridsolar1.com/
E-mail [email protected]
|| Midnight Classic 200 | 10, Evergreen 200w in a 160VOC array ||
|| VEC1093 12V Charger | Maha C401 aa/aaa Charger | SureSine | Sunsaver MPPT 15A
solar: http://tinyurl.com/LMR-Solar
gen: http://tinyurl.com/LMR-Lister ,
My Xanbus experience is limited to installing and using this system, so I welcome hearing how it behaves under adverse conditions. I just estimated my xbus length and it's probably 250', so clearly over the 130' allowed. Maybe this has become the problem with the introduction of the ComBox in place of the gateway. It's certainly cause for suspicion. But all my non-xbus CAN experience says it would result in packets that have detected errors and get dropped.
If it is a result of the long bus, does anyone make a repeater? The bus is well contained in the house with all equipment within probably 30-50' of each other. Then there is a long run out to the barn where the gen and AGS are installed. That's what stretches the bus.
You may be right about the packets, detected errors, and drops. I just take exception anytime that someone says "always"
Always is a very big statement that often misses unknown exceptions.
htps://offgridsolar1.com/
E-mail [email protected]
My advice is of a general nature. When it becomes specific I am billing and there are plenty of others here who this is a hobby for.
Think about it and you will figure it out! Good Luck. I do plenty of pro bono, but not much on this continent...
htps://offgridsolar1.com/
E-mail [email protected]
Because these phantom starts only last a second, and seem to be randomly spaced, the only way I notice them is when I happen to be in the same room in the barn where the generator is housed. To get a better handle on how often this is happening, I got a little electronic counter and wired it to the ignition circuit. Now the counter increments each time the ignition circuit is energized.
For the first couple of days, AGS is at the far end. All devices are powered off the Xanbus, and one possibility is that the Combox draws more power than the Gateway, and there is now marginal power at the AGS. So my first experiment has been to install the external power adapter for the Combox so it is powered externally rather than via the Xanbus. This experiment is ongoing now, so we will see what happens.
Next up will be to remove the Combox and reinstall the Gateway and see if the problem goes away.
Oh, and I noticed something else odd. I looked at the event log on the Combox today and it says that it rebooted at 03:05 this morning in response to a reboot command. WTF? Where could that be coming from?
|| Midnight Classic 200 | 10, Evergreen 200w in a 160VOC array ||
|| VEC1093 12V Charger | Maha C401 aa/aaa Charger | SureSine | Sunsaver MPPT 15A
solar: http://tinyurl.com/LMR-Solar
gen: http://tinyurl.com/LMR-Lister ,
htps://offgridsolar1.com/
E-mail [email protected]
|| Midnight Classic 200 | 10, Evergreen 200w in a 160VOC array ||
|| VEC1093 12V Charger | Maha C401 aa/aaa Charger | SureSine | Sunsaver MPPT 15A
solar: http://tinyurl.com/LMR-Solar
gen: http://tinyurl.com/LMR-Lister ,
htps://offgridsolar1.com/
E-mail [email protected]
Is there a daily reboot setting, or is it a hard coded function ?
|| Midnight Classic 200 | 10, Evergreen 200w in a 160VOC array ||
|| VEC1093 12V Charger | Maha C401 aa/aaa Charger | SureSine | Sunsaver MPPT 15A
solar: http://tinyurl.com/LMR-Solar
gen: http://tinyurl.com/LMR-Lister ,
The testing I am doing now, they always tell me not to update firmware they send me at or about 3:05 AM.
That will never happen for me.....
Have you tried the Conext Insight which uses combox?
htps://offgridsolar1.com/
E-mail [email protected]
Now that it's been OFF for a couple months, I'm looking forward to not having it crash on me.
So it goes into maint mode daily about 3am.
|| Midnight Classic 200 | 10, Evergreen 200w in a 160VOC array ||
|| VEC1093 12V Charger | Maha C401 aa/aaa Charger | SureSine | Sunsaver MPPT 15A
solar: http://tinyurl.com/LMR-Solar
gen: http://tinyurl.com/LMR-Lister ,
I notice the maint mode on an SCP that is not on the same screen I left it in at lights out time, the next morning.
It took our ISP 7 years and two broken bulldozers to get up to us so I understand.
htps://offgridsolar1.com/
E-mail [email protected]
But when I look at the AGS status via the Combox, it says that the Stop Reason is Exercise End. But the generator hasn't exercised. But now suspecting something related to the exercise function, I started looking more closely at when the phantom start attempts occur, and they are all within the exercise time window. I have it set to exercise every 21 days at 09:00 for 60 minutes, so from 09:00 to 10:00. All the phantom start attempts occur in this window. Note that the exercise function has always worked correctly, save these phantom start attempts.
So at this point I think I have found a bug in the AGS. My next experiment will be to see if I can cause the phantom starts to happen by setting the exercise time and duration to something different. If the phantom starts happen in that new window, then I think I have figured out how to reproduce the problem. I'll also be away for a week or two, and plan to disable the exercise function while gone to see if the phantom starts cease, which I expect they will.
Awesome, writing bug free code is a lot tougher than it looks. At least it's not a bug that hits the glow plugs only and drains the battery.
|| Midnight Classic 200 | 10, Evergreen 200w in a 160VOC array ||
|| VEC1093 12V Charger | Maha C401 aa/aaa Charger | SureSine | Sunsaver MPPT 15A
solar: http://tinyurl.com/LMR-Solar
gen: http://tinyurl.com/LMR-Lister ,
Hard to believe that a code problem could do that. Not impossible but hard to believe.
For myself as one who does not use a generator, easy for me to believe that the AGS has been surged one too many times from generators or that something is getting into the xanbus and the really long cable run.
htps://offgridsolar1.com/
E-mail [email protected]
I can't say for 100% sure that this problem hasn't been present all along. It just seems unlikely that I never noticed it before. But it's possible.
As for other causes, it's certainly possible. This is just the first time that I've been able to narrow it down in any definitive way, so it seems the most promising explanation - at least so far.
Earlier this evening I disabled the exercise function. If the current theory holds, I would expect tomorrow to be free from phantom starts. It will be interesting to see what actually happens.
One experiment on my list is to remove the Combox and replace with the old Gateway, thereby putting the system back the way it was before I first noticed the phantom starts. It will be very interesting to see if the problem goes away when I do that. I can't offer any explanation for why installing the Combox would initiate this problem, but it does correlate with when I first noticed the problem, so seems a likely factor. But right now I'm making progress (I think) in making the problem reproducible which is essential to debugging a problem. So I'm going to keep on this path for now.
Stay tuned...
I finally found the support web page for this product, which is impossible starting at Schneider-electric.com, but google is my friend where the manufacturer otherwise fails. It looks like my AGS firmware is out of date, and although there is no mention of this issue in the release notes for the updates, updating seems like the next logical step. But at this point I'm pretty convinced this is an AGS firmware bug, and nothing to do with my long Xanbus or the addition of the Combox.
If you have not updated the firmware.....
When you search, always add Solar to the Schneider search. The division is Schneider Electric Solar part of the 29 billion euro Schneider Electric
htps://offgridsolar1.com/
E-mail [email protected]
|| Midnight Classic 200 | 10, Evergreen 200w in a 160VOC array ||
|| VEC1093 12V Charger | Maha C401 aa/aaa Charger | SureSine | Sunsaver MPPT 15A
solar: http://tinyurl.com/LMR-Solar
gen: http://tinyurl.com/LMR-Lister ,
htps://offgridsolar1.com/
E-mail [email protected]
|| Midnight Classic 200 | 10, Evergreen 200w in a 160VOC array ||
|| VEC1093 12V Charger | Maha C401 aa/aaa Charger | SureSine | Sunsaver MPPT 15A
solar: http://tinyurl.com/LMR-Solar
gen: http://tinyurl.com/LMR-Lister ,
|| Midnight Classic 200 | 10, Evergreen 200w in a 160VOC array ||
|| VEC1093 12V Charger | Maha C401 aa/aaa Charger | SureSine | Sunsaver MPPT 15A
solar: http://tinyurl.com/LMR-Solar
gen: http://tinyurl.com/LMR-Lister ,