Software that can read all devices ouputs

South Africa
South Africa Solar Expert Posts: 295 ✭✭✭
Hi,

Putting it out there.

I know most manufactures of solar components have data storage available for their equipment, some have the ability to upload the data, some need rather expensive additional devices to do that, but if you venture into separate components from various suppliers, there are not one software solution irrespective of equipment, that I can find that can combine the data.

So I am wondering if there is a need for generic software that can read data (from any solar devices that has a data output port) and store it somewhere on the web so that one can track what is happening on your system?

Data from i.e. charge controllers, a BMW battery monitor, inverters etc.

Then, because you have the data, you can now developed features like to inform user of anomalies, based on year to date values, where battery values are below average, panels are not preforming based on time of day or that the inverter appears to have a issue, via email / SMS? Once the data is recorded, the sky is the limit what to do with the data.

Is it worth my while to get my developers to look at this?

Notes:
The power the device needs must be as low as possible. So I am thinking of a Raspberry Pi2B.
Most device have serial connections, you may need up to 5 ports, so a USB hub.
Operating Software must be free, with developers not having to learn new skills, so Win10 IoT as the OS on the Pi 2B.
Data upload done with a UTP cable of the Pi 2B or added WiFi connector connected to users network.
SQL server to store the data on the web, that is not costing a arm and a leg.
5kVA Victron Multiplus II, 5.2kW array, 14kWh DIYLifepo4 bank, all grid-tied.


Comments

  • zoneblue
    zoneblue Solar Expert Posts: 1,220 ✭✭✭✭
    Commercially, there are roadblocks (proprietary stacks etc). That leaves you with industrial monitoring and automation systems like scada. Which you really have to know your stuff to implement because they are so complex.

    In the maker world, theres also quite a bit going on, examples:
    http://openenergymonitor.org
    https://github.com/zoneblue/theblackboxproject/blob/wiki/ProjectHome.md

    There the issue is diversity with lots of projects and duplication.
    See http://kb1uas.com/mnsforum/index.php#c13



    1.8kWp CSUN, 10kWh AGM, Midnite Classic 150, Outback VFX3024E,
    http://zoneblue.org/cms/page.php?view=off-grid-solar


  • karrak
    karrak Solar Expert Posts: 326 ✭✭✭✭
    zoneblue said:
    Commercially, there are roadblocks (proprietary stacks etc). That leaves you with industrial monitoring and automation systems like scada. Which you really have to know your stuff to implement because they are so complex.
    Because of this I wonder if it is easier to measure and log all the system parameters directly and generate your system from this.

    Although originally designed to measure individual battery cell voltages in my LFP (LiFePO4) battery, my project here
    https://github.com/simat/BatteryMonitor could be cut down to measure just overall battery voltage and battery current. It would need just a handful of components to measure battery voltage and battery current and although I currently have it running on a Beaglebone Black I don't see any reason that it couldn't run on any of the Raspberry Pis. Adding extra sensors to measure load current and other parameters is also fairly easy.

    Simon 

    Off-Grid with LFP (LiFePO4) battery, battery Installed April 2013
    32x90Ah Winston cells 2p16s (48V), MPP Solar PIP5048MS 5kW Inverter/80A MPPT controller/60A charger, 1900W of Solar Panels
    modified BMS based on TI bq769x0 cell monitors.
    Homemade overall system monitoring and power management  https://github.com/simat/BatteryMonitor
     

  • South Africa
    South Africa Solar Expert Posts: 295 ✭✭✭
    Thanks for the links.

    Close but not exactly what I am aiming for. For starters I want to read data direct from  a Morningstar TriStar Controller to see what the controller does, a Victron inverter to see what goes out via AC and a Victron BMV battery monitor to see what goes in and out of the batteries.

    And if you start with one device in the suppliers range, you can just as well do the development for the entire range. Only need a suppliers documentation to add more devices that the software can read data from.

    Currently data is read using each devices free proprietary software, installed on my laptop, and data is stored in CSV files.
    And all the above suppliers have the docs available for one to develop your own software. 

    This is what I am looking at, using the devices software I have currently installed. Idea is to have software package to read and display all data, once it has been stored in a database.
    1 - Charge controllers data
    2 - BMV battery monitor data
    3 - Inverter data

    5kVA Victron Multiplus II, 5.2kW array, 14kWh DIYLifepo4 bank, all grid-tied.


  • South Africa
    South Africa Solar Expert Posts: 295 ✭✭✭
    edited January 2016 #5
    To do it for Grid tied inverters is not needed for most of them have their own software included.

    Using separate devices like Midnite solar, Outback etc, you mix and match solar parts, can be very costly to read them individually and data is not stored in one central point.
    5kVA Victron Multiplus II, 5.2kW array, 14kWh DIYLifepo4 bank, all grid-tied.


  • zoneblue
    zoneblue Solar Expert Posts: 1,220 ✭✭✭✭
    The links i supplied will take you a small lifetime to digest. But they will do what you want. The question is how much time do you have to spend on it? There ARE out of the box solutions, which you can see listed in the documetnation section of theblackboxproject.
    1.8kWp CSUN, 10kWh AGM, Midnite Classic 150, Outback VFX3024E,
    http://zoneblue.org/cms/page.php?view=off-grid-solar


  • South Africa
    South Africa Solar Expert Posts: 295 ✭✭✭
    Zoneblue, my apologies. I am not the techie, just the one who thinks of the needs. Sorry if I missed anything.

    Do NOT need to re-invent anything, I would be extremely happy if it is available already, or can be done.

    Having looked again (quick), am I right that you use Microsoft SQL?
    Have you done, are you planning, interfaces for:
    http://www.morningstarcorp.com/products/tristar/ - the 45 model.
    http://https//www.victronenergy.com/inverters/phoenix-inverter-compact - i have the Phoenix 1600va inverter only.
    http://https//www.victronenergy.com/battery-monitors/bmv-700 - the 700 one. The BMV 700 is the key to my entire system.
    I have all the cables for all the devices, serial to USB.
    5kVA Victron Multiplus II, 5.2kW array, 14kWh DIYLifepo4 bank, all grid-tied.


  • Peter_V
    Peter_V Solar Expert Posts: 226 ✭✭✭
    Hi,

    Putting it out there.

    I know most manufactures of solar components have data storage available for their equipment, some have the ability to upload the data, some need rather expensive additional devices to do that, but if you venture into separate components from various suppliers, there are not one software solution irrespective of equipment, that I can find that can combine the data.

    So I am wondering if there is a need for generic software that can read data (from any solar devices that has a data output port) and store it somewhere on the web so that one can track what is happening on your system?

    Data from i.e. charge controllers, a BMW battery monitor, inverters etc.

    Then, because you have the data, you can now developed features like to inform user of anomalies, based on year to date values, where battery values are below average, panels are not preforming based on time of day or that the inverter appears to have a issue, via email / SMS? Once the data is recorded, the sky is the limit what to do with the data.

    Is it worth my while to get my developers to look at this?

    Notes:
    The power the device needs must be as low as possible. So I am thinking of a Raspberry Pi2B.
    Most device have serial connections, you may need up to 5 ports, so a USB hub.
    Operating Software must be free, with developers not having to learn new skills, so Win10 IoT as the OS on the Pi 2B.
    Data upload done with a UTP cable of the Pi 2B or added WiFi connector connected to users network.
    SQL server to store the data on the web, that is not costing a arm and a leg.

    South Africa said:
    Thanks for the links.

    Close but not exactly what I am aiming for. For starters I want to read data direct from  a Morningstar TriStar Controller to see what the controller does, a Victron inverter to see what goes out via AC and a Victron BMV battery monitor to see what goes in and out of the batteries.

    And if you start with one device in the suppliers range, you can just as well do the development for the entire range. Only need a suppliers documentation to add more devices that the software can read data from.

    Currently data is read using each devices free proprietary software, installed on my laptop, and data is stored in CSV files.
    And all the above suppliers have the docs available for one to develop your own software. 

    This is what I am looking at, using the devices software I have currently installed. Idea is to have software package to read and display all data, once it has been stored in a database.
    1 - Charge controllers data
    2 - BMV battery monitor data
    3 - Inverter data


  • Peter_V
    Peter_V Solar Expert Posts: 226 ✭✭✭
    Hi,

    Putting it out there.

    I know most manufactures of solar components have data storage available for their equipment, some have the ability to upload the data, some need rather expensive additional devices to do that, but if you venture into separate components from various suppliers, there are not one software solution irrespective of equipment, that I can find that can combine the data.

    So I am wondering if there is a need for generic software that can read data (from any solar devices that has a data output port) and store it somewhere on the web so that one can track what is happening on your system?

    I don't think that's practical.  Everything uses a different protocol/interface, etc.for communicating and many times this is a proprietary protocol.
    There is no way to make a "generic' interface.  The software would have to be specifically programmed to talk to each device.

    There is some opensource software that can take inputs from multiple sources, but you (or someone else) would have to write an interface for each device you want to use.  Even then, with some of the proprietary protocols, it's not possible to talk directly to the device.

    For example I use Enphase inverters.  These use a propriatary powerline protocol (they transmit their data over the power lines).  You need to use an "Envoy" to talk to the inverters and this will only send it's data back to Enphase where you can view it on the Enlighten website.

    Getting data from this system was such a pain in the butt that I went out and bought a Brultech ECM-1240 energy monitoring system and use that instead. I then went and wrote custom software for talking to the ECM-1240.
  • South Africa
    South Africa Solar Expert Posts: 295 ✭✭✭
    Peter, correct.

    Not one generic protocol, but per device, as required per suppliers protocol / interface for the device.

    In the end, user will have to select COM port per device connected, make and model, with the suppliers cable attached, to read the data.

    So far my experiences with my equipment, exposed me to Serial to USB with not too much effort required to read the data being sent.

    Enphase inverters I have not been exposed to, whereas Victron and Morningstar seems to encourage development as the docs are freely available.
    5kVA Victron Multiplus II, 5.2kW array, 14kWh DIYLifepo4 bank, all grid-tied.


  • mike95490
    mike95490 Solar Expert Posts: 9,583 ✭✭✭✭✭
    The Digi/Edgeport USB-Serial multi port adapter boxes are nice gear, 1 USB cable to your computer, and you can have 4-8 serial (COM) ports on the box. all powered from the USB.  Some good used buys on ebay for them. 
    Powerfab top of pole PV mount | Listeroid 6/1 w/st5 gen head | XW6048 inverter/chgr | Iota 48V/15A charger | Morningstar 60A MPPT | 48V, 800A NiFe Battery (in series)| 15, Evergreen 205w "12V" PV array on pole | Midnight ePanel | Grundfos 10 SO5-9 with 3 wire Franklin Electric motor (1/2hp 240V 1ph ) on a timer for 3 hr noontime run - Runs off PV ||
    || Midnight Classic 200 | 10, Evergreen 200w in a 160VOC array ||
    || VEC1093 12V Charger | Maha C401 aa/aaa Charger | SureSine | Sunsaver MPPT 15A

    solar: http://tinyurl.com/LMR-Solar
    gen: http://tinyurl.com/LMR-Lister ,

  • South Africa
    South Africa Solar Expert Posts: 295 ✭✭✭
    And it is done :D

    The focus was on the Voltronics range of UPS'es, glorified by adding a MPPT controller, with modules to read Victron devices using a Raspberry Pi or Arduino being developed, to also feed the software.

    http://www.solarmonitor.co.za/ for small fee.
    And this one http://aicc.org.za/ where the source code is now available.

    Both need a Victron BMV for more accurate SOC of the batteries.

    Cannot wait for the Rpi software that will read not only the Voltronic devices, but also the initial Victron inverters, BMV and MPPT controllers as well as the first Morningstar PWM charge controller.

    5kVA Victron Multiplus II, 5.2kW array, 14kWh DIYLifepo4 bank, all grid-tied.


  • rmcewen
    rmcewen Registered Users Posts: 8 ✭✭
    I have a system like this in my off-grid cabin but like many other posters mentioned it's customized for my particular setup and hardware.  All the data is collected and uploaded to a central server where it can be viewed/graphed over time via a web application.  The system consists of the following:

    1. Odroid U3 arm board (much more powerful than a Pi) as the central controller.
    2. Motion software running on controller monitors 4 security cameras and uploads images to web when motion detected.
    3. Controller reads all modbus values from Classic 150/WhizBangJr over ethernet to provide SOC, in/out amps, voltages, etc.
    4. Controller attached to an arduino Due via usb serial which reads three temperature sensors (inside, outside, utility room) and controls eight relays to turn specific loads on and off (i.e. water pump, fridge, inverter).
    5. Uploads to server with Sprint 3G modem and Cradlepoint router.
    6. Local web application runs on controller and allows control of relays and cameras as well as displaying real-time data.
    6. Remote web application runs on central server and allows viewing of camera images, real-time data, and historical data.

    So basically I can remotely monitor everything I need, including images from security cameras.  Everything runs linux and all the code except the arduino code is in python.  The data is stored in MongoDB which handles time series data better than a relational database.

    The entire remote system draws about 50W including the controller, four cameras, arduino, relays, and 3G modem.

    The one thing it currently lacks is the ability to manipulate the relays remotely or take a snapshot from one of the cameras.  I'm currently working on v2 which will have that capability.  The main differences in V2 are:

    1. Uses MQTT as a messaging protocol so will have two way communication between the controller at the cabin and central server.
    2. Uses an Odroid XU4 (8 core, 2GB ram, usb 3, gigbit ethernet ) as the controller ($75).
    3. Uses latching relays so relay coils consume no power.
    4. Uses Arduino Mega with ethernet shield.  The arduino has a MQTT client so I don't have to write my own protocol to communicate over serial like the previous version, it will just send and receive MQTT messages like everything else.
    5. Added some PIR sensors to the arduino that will be placed near the cameras.  Hopefully this will allow me to eliminate some false positives generated by using strictly image based motion detection.
    6. Wrote a native Android application that will display data and control relays, cameras, etc.  This replaces the local web application.

    So that's my current and in progress solution to data collection and system control.  The v1 system has been running for about two years and has performed flawlessly.  I'm usually only at the cabin on weekends so it's great to be able to monitor everything remotely.


  • Dave Angelini
    Dave Angelini Solar Expert Posts: 6,891 ✭✭✭✭✭✭
    Good Job!
    Can your system communicate to the ODROID the state of a switch?  I guess what I am saying is I am doing this now but I need the internet to be working.  I would like to do something similar on a IOS or droid phone but on the wi-fi side of the firewall. You can e-mail me if you have some time.  Thanks!
    "we go where power lines don't" Sierra Nevada mountain area
       htps://offgridsolar1.com/
    E-mail offgridsolar@sti.net

  • rmcewen
    rmcewen Registered Users Posts: 8 ✭✭
    Yes, the Odroid can read the state of a switch using one of the digital inputs on the connected arduino.  For example, in the v2 system, the PIR sensors are just switches that close when motion is detected.  The v2 system can query any of the digital or analog input pins via an MQTT command sent to the arduino or you could program the arduino to send an MQTT command when the switch state changes.  For example, when the PIR sensors detect motion the arduino sends an MQTT message that it was triggered and another message when motion is no longer detected.  As I mentioned, I'm going to hopefully use this to improve the motion detection of the security cameras and I have a siren and flashing light I may connect as well to ideally act as a deterrent to would be thieves.

    If the switch is actually switching something and thus carrying a voltage on the switched circuit you'd want to either use resistors and create a voltage divider to bring the voltage down to the arduino's input voltage of 5V or use some type of transistor or optoisolater.

    As for the internet and firewall, I basically wrote two different web applications, one for the local network and another for the internet.  Version 2 will have the android app for the local network and a web application for the internet.  In v2, the odroid will run a local MQTT server which is bridged with an MQTT server on the internet server so all messages are received in both places and can be sent from inside or outside the firewall.  It greatly simplifies the transfer of data and I won't have to write any code to pass data between the two locations.  

    The current system would require updating the arduino code to add a message sending trigger but the idea with version 2 is to specify a range of inputs that will send an mqtt message when the state has changed so when adding something new I can just hook it up to one of the free digital inputs.  I wired up 8 digital and 6 analog inputs to be usable as general purpose inputs.  Currently the PIR sensors use 2 of the 8 digital inputs.

    The arduino's ehternet shield also has a micro sd card reader so I have the code read a lot of the configuration information from the sd card on startup.  So if I need to change the ip address or add another PIR sensor I can just update an .ini file on the sd card and the arduino will recognize it as a special type of sensor and send the corresponding messages as appropriate.  For example, if I add a 3rd PIR sensor I just need to connect the sensor and add the corresponding pin number to the config file on the sd card.  No reprogramming required.

    In v2 the arduino and the odroid aren't physically connected, other than being on the same network.  So when the arduino sends an MQTT message like a PIR sensor being triggered, code on the odroid receives the message and can then start capturing video or snapshots from the cameras, switch a relay to activate the siren, etc.  Eventually I hope to have it send a text message to my phone with a snapshot from the camera also so I can determine the source of the motion.

    With the current system I can go to the web application and see snapshots taken every second from the cameras when they detect motion.  I have a lot of pictures of birds flying in front of the cameras and spiders building webs.  At night the IR leds in the cameras attract flying insects which then attracts the spiders.  That's why I'm adding the PIR sensors to hopefully reduce false positives.

    Here's a blog post with some pictures of the current system:  http://blog.flgator.com/2014-12-17-controller.html

    The v2 controller board is about 1/3 the size of the current version you see in the pictures.
  • Dave Angelini
    Dave Angelini Solar Expert Posts: 6,891 ✭✭✭✭✭✭
    Thanks Rick!  I left an e-mail at your Blog.  Really nice workmanship.
    "we go where power lines don't" Sierra Nevada mountain area
       htps://offgridsolar1.com/
    E-mail offgridsolar@sti.net

  • South Africa
    South Africa Solar Expert Posts: 295 ✭✭✭
    So, we did it. Reading Victron MPPT's, BMV's and inverters, Morningstar PWM controller and most of the Voltronic range of inverters sold here in SA.

    We built generic reader, can run on Windows 7-10 (soon a Raspberry Pi3b with Win10 IoT as OS).
    Data read is sent to a SQL database on the web.

    Here I am reading 4 devices on my laptop. The two blocks you see blank, those devices are being red by my tablet for they are a bit to far from the laptop.


    System is web based. The is my Dashboard, showing what I need to see, using Google Chrome. Can also view same on my Cellphone.


    Then we have the generic Dashboard, for standard users:
    This is the dashboard for the Voltronix (Axpert) device.


    The stats per device i.e:
    1 x Morningstar Controller for system 1
    1 x BMV700 battery Monitor for system 1
    1 x Victron MPPT - for system 2
    1 x Victron BMV600S for system 2
    1 x Victron MPPT - for system 3
    1 x Not shown here, is the 6th device, i.e. a Victron Phoenix inverter.

    Can view up to 6 devices, for most people will not have my mismatched system.  :D


    And the Graphs:


    Once all is tested and confirmed, for we pick up interesting anomalies late nights, early mornings and rainy days, we will continue to expand the software to switch relays on / off, send warnings to users if problems are picked up and any other ideas we can add to make the software add value.
    5kVA Victron Multiplus II, 5.2kW array, 14kWh DIYLifepo4 bank, all grid-tied.


  • Dave Angelini
    Dave Angelini Solar Expert Posts: 6,891 ✭✭✭✭✭✭
    I would add the ability to not just switch relays but control appliances over wifi based on conditions.
    Nice job!
    "we go where power lines don't" Sierra Nevada mountain area
       htps://offgridsolar1.com/
    E-mail offgridsolar@sti.net

  • South Africa
    South Africa Solar Expert Posts: 295 ✭✭✭
    Thanks Dave ... Wifi  ... thank you for the idea!
    5kVA Victron Multiplus II, 5.2kW array, 14kWh DIYLifepo4 bank, all grid-tied.