Enphase Data Access

24567

Comments

  • ag4wake
    ag4wake Registered Users Posts: 18
    Re: Enphase Data Access

    Did you get your system installed by a company. If so bets are they have a customer that would have the older enphase. The company i'm dealing with probably has a customer or two with the older unit, though since I haven't yet signed anything I'm thinking it maybe too soon to ask around to borrow his customers devices.
    andyman wrote: »
    Thats exactly my plan. Take a dump of shadow and brute it. Problem is getting in first. The person who got into it did it because he had the older unit that allowed bootup from an alternative flash. Enphase disabled this ability or atleast I haven't figured it out yet. So no shadow at this time.
  • System2
    System2 Posts: 6,290 admin
    Re: Enphase Data Access

    Has anyone looked at capturing the data between the inverters and the Envoy? e.g. the powerline comms..
  • ag4wake
    ag4wake Registered Users Posts: 18
    Re: Enphase Data Access

    I'm getting closer to taking the plunge with a solar pv system. It looks like the older envoys are still available. Maybe I'll buy both and see what we can do with them. Overall it's a drop in the bucket on the cost of the system.
  • sandeen
    sandeen Solar Expert Posts: 48 ✭✭✭
    Re: Enphase Data Access

    So, I got a unit today. :) Not sure what to do with it yet, but for now here are some photos. Note the COM port on the back side.

    http://www.flickr.com/photos/39926580@N00/4662038196/
    http://www.flickr.com/photos/39926580@N00/4662039404/
  • drees
    drees Solar Expert Posts: 482 ✭✭✭
    Re: Enphase Data Access

    Looks like a 2GB SanDisc microSD card on the front side - have you pulled it yet?

    Is that a USB port in between the SD and Ethernet card, too?
  • sandeen
    sandeen Solar Expert Posts: 48 ✭✭✭
    Re: Enphase Data Access
    drees wrote: »
    Looks like a 2GB SanDisc microSD card on the front side - have you pulled it yet?

    Of course. :) It's an empty ext2 partition. Must be for data storage while offline.
    drees wrote: »
    Is that a USB port in between the SD and Ethernet card, too?

    It is; the docs say "don't use it" ;) Wifi would be nice...

    FWIW, absolutely nothing in the box says anything about the GPL license. Grr.
  • System2
    System2 Posts: 6,290 admin
    Re: Enphase Data Access

    The problem is trying to scrub the data from the html interface is they don't provide enough of the info. It would actually be completely fine if they gave you the current power number for each panel, instead the unit only shows you the power generation for the entire system... It would have been pretty simple for them to display each panel as well.. but for whatever reason they choose not to... so you have to get at the data some other way...

    I was able to hack getting the data out of it with a different route I haven't seen discussed yet. I played with my hack for a couple weeks, and then quit since I decided that what enphase was charging for the access was cheap enough. And with my hack, you couldn't see your system on enphase site until the night... (you'll see why after you read how it works)

    Here is the hack I did:
    Change your dhcp so that it assignes out a DNS server to the unit that you have control over. Set (I'm going from memory now) reports.enphaseenergy.com to point to a web server you have control over... So that when the unit attempts to report home it goes to your webserver instead. (https enabled)

    Then on your webserver it will attempt to goto (from memory again, verify these from the apache web logs):
    emu_reports/performance_report

    So you create a file emu_reports/performance_report.php

    Inside there you can put: (You need to get your apache to interpert it as php even if it the URL doesn't have the extension on it.)

    <?

    if($_SERVER >= 1) {
    print "Syntax check\n";
    exit;
    }

    $con = mysql_connect ('localhost', 'XXXX', 'XXXX');
    mysql_select_db('emu');

    $date = date("r");
    $time = time();
    $out = file_get_contents("php://input");
    $out1 = gzuncompress($out);
    $wfd = fopen("/tmp/out","a");
    fwrite($wfd,"\nHere: $date $time\n");
    fwrite($wfd,"$out1\n");
    fclose($wfd);

    // <interval eqid='110934067806.1' end_date='1259949300' stat_duration='300' stats='2528,245490,60018,34685,316,6,0,0' id='203' interval_duration='300'/>
    // <interval eqid='110937088168.1' end_date='1259949300' stat_duration='300' stats='890,245980,59942,33531,334,10,0,0' id='204' interval_duration='300'/>

    $wfd = fopen("/tmp/cur","w");
    $P = explode("\n",$out1);
    foreach ($P as $line) {
    ...
    ...

    A bunch of code to parse the xml data...
    }

    ----

    The only thing I couldn't figure out was how to get it to quit sending the data.. so once a night you change the DNS entry to the real reports.enphaseenergy.com and let it talk to there... that clears the buffer and the next morning you get your data again... (And sends all the data to enphase)

    One issue is that if it talks to reports.enphase *instead* of you, then you lose that record... They send some command back to tell the unit that they have received the data, so it quits transmitting that data to you.. so you can't let it talk to reports.enphaseenergy.com unless you know there is no "real" data to send (like at night).

    Also, you have to configure an https apache web server since the unit wants to talk https... It isn't smart enough to verify the certificates though, so you can just self-sign the certificates...

    (This is not an easy hack, but for you guys who are talking about doing embedded linux stuff, it shouldn't be to hard to throw together).

    Good luck, hope somebody breaks in and gets us real access. I would love to be able to get the data out on the fly... without racing enphase to it. I'd be happy enough with a setting in their web interface that allows you to point it to a secondary web upload site and published specs on what the data means that is in the XML and what commands you can send back to it.


    Laters,
    -d.
  • drees
    drees Solar Expert Posts: 482 ✭✭✭
    Re: Enphase Data Access
    sandeen wrote: »
    Of course. :) It's an empty ext2 partition. Must be for data storage while offline.
    Figures! Would have been nice if they kept an image of the system on it...
    sandeen wrote: »
    It is; the docs say "don't use it" ;) Wifi would be nice...
    Duh - would help if I actually looked at my EMU here... :blush:
    sandeen wrote: »
    FWIW, absolutely nothing in the box says anything about the GPL license. Grr.
    Nope, nothing about the GPL here or where to get the source for the GPL software shipping on it.

    It appears that at least some of the software written on it is in ruby from some of the stack traces you can get out of the EMU if you hit the local web server too fast or at the wrong time.
    Also, you have to configure an https apache web server since the unit wants to talk https... It isn't smart enough to verify the certificates though, so you can just self-sign the certificates...
    Nice detective work - should be pretty simple then to actually write a transparent SSL proxy using the DNS spoofing (since the EMU doesn't check the validity of SSL certs) which grabs a copy of the data, saves it for processing, then passes it along to Enphase and any response back from Enphase back to the EMU.

    Possible steps would be:
    1. Setup split DNS using bind to provide EMU with local server IP address instead of actual IP address of reports.enphaseenergy.com
    2. Setup Apache with a self-signed cert - if it doesn't check the validity of the SSL cert, it might even accept any SSL cert (though it's easy to create your own self-signed cert for any domain)
    3. Write program to handle requests from EMU, save data, pass on to Enphase, read response from Enphase, pass back to EMU.

    I could write that - just need a bit of free time... :D

    At the very least, that would be useful as then you can easily grab a copy of all data being uploaded to Enphase for your own data plotting. Though getting access to the data directly on the EMU would still be nice...
  • russ
    russ Solar Expert Posts: 593 ✭✭
    Re: Enphase Data Access

    İ thinl they are trying to insure an income from each custoöer for the next 30 years to come! Nice guys!
  • sandeen
    sandeen Solar Expert Posts: 48 ✭✭✭
    Re: Enphase Data Access
    russ wrote: »
    İ thinl they are trying to insure an income from each custoöer for the next 30 years to come! Nice guys!

    They're welcome to do that of course, and if there's value in the offering (and I really do think there is) then it's a great strategy. I have no problem with it.

    But we seem to be hackers on this thread and there's a certain element of "why? because it's there!" - I don't think any of this activity will result in any significant loss to Enphase, they have a good product and a good price, they'll do fine.

    (they're not playing by the rules w.r.t. the GPL license of the code they're using, though.)
  • russ
    russ Solar Expert Posts: 593 ✭✭
    Re: Enphase Data Access

    The company that comes along and allows free access to your own data will kick their butt! One will arrive soon enough then we see a new policy.

    İ just don't like that type of business model. Saw similar for an energy monitoring system for homes - had saved their information but went back and deleted it.
  • sandeen
    sandeen Solar Expert Posts: 48 ✭✭✭
    Re: Enphase Data Access
    russ wrote: »
    The company that comes along and allows free access to your own data will kick their butt! One will arrive soon enough then we see a new policy.

    I think it'd be a niche market of a niche market, and not worthwhile as any sort of business venture. Anything useful would require custom firmware (to push elsewhere) or a server running inside your firewall to pull data from the device... I just don't see it as viable on any serious scale. Just my $0.02 ;)
  • sandeen
    sandeen Solar Expert Posts: 48 ✭✭✭
    Re: Enphase Data Access
    andyman wrote: »
    No.. It won't tftp a new image unless you can pass redboot the commands to do it. The bootp that you see initially is just a very low level telnet server redboot sets up on port 9000 for low level debug access. You still need to hit ctrl c to get into redboot. Problem is Enphase decided to make the bootup time 1 second and somehow disabled ctrl -c ..

    If i can get to the redboot prompt i could just do this:

    load -r -b 0x100000 /tftpboot/zImage
    exec -b 0x100000 -l 0x200000 -c "noinitrd console=ttymxc0,115200 root=/dev/nfs nfsroot=10.29.244.99:/tftpboot/rootfs init=/linuxrc ip=10.29.241.6:10.29.244.99"


    My life would be a lot easier if someone can just let me borrow their old Envoy unit (big square box, not the little one) for a day.

    andy

    You know, if someone asked for GPL sources they'd have to (in theory) tell us how they changed redboot. Maybe it's something other than ctrl-c now? Of course if the patch they provide simply shows it as disabled we'd never know the difference ....
  • drees
    drees Solar Expert Posts: 482 ✭✭✭
    Re: Enphase Data Access
    sandeen wrote: »
    You know, if someone asked for GPL sources they'd have to (in theory) tell us how they changed redboot.
    Hey Eric, you're the open source expert here! ;)
  • System2
    System2 Posts: 6,290 admin
    Re: Enphase Data Access
    drees wrote: »
    Nice detective work - should be pretty simple then to actually write a transparent SSL proxy using the DNS spoofing (since the EMU doesn't check the validity of SSL certs) which grabs a copy of the data, saves it for processing, then passes it along to Enphase and any response back from Enphase back to the EMU.

    Possible steps would be:
    1. Setup split DNS using bind to provide EMU with local server IP address instead of actual IP address of reports.enphaseenergy.com
    2. Setup Apache with a self-signed cert - if it doesn't check the validity of the SSL cert, it might even accept any SSL cert (though it's easy to create your own self-signed cert for any domain)
    3. Write program to handle requests from EMU, save data, pass on to Enphase, read response from Enphase, pass back to EMU.

    I could write that - just need a bit of free time... :D

    At the very least, that would be useful as then you can easily grab a copy of all data being uploaded to Enphase for your own data plotting. Though getting access to the data directly on the EMU would still be nice...

    Great thread guys, I was about to sign up for a solar system based on the Enphase microinvertors, and stumbled across this thread trying to verify that access to the system's data is open.

    Regarding the idea above of a transparent proxy, could you just use something like squid to do that, rather than writing one yourself? I see the squid docs point out the problems with transparent https proxying, but I think as long as the EMU doesn't require any particular signed cert, you'd probably be okay? Just a thought...
  • drees
    drees Solar Expert Posts: 482 ✭✭✭
    Re: Enphase Data Access
    Regarding the idea above of a transparent proxy, could you just use something like squid to do that, rather than writing one yourself? I see the squid docs point out the problems with transparent https proxying, but I think as long as the EMU doesn't require any particular signed cert, you'd probably be okay? Just a thought...
    I thought of that - but does Squid in transparent proxy setup let you capture all the data being transmitted? Admittedly I haven't checked or used Squid except for regular proxy type use.
  • System2
    System2 Posts: 6,290 admin
    Re: Enphase Data Access
    drees wrote: »
    I thought of that - but does Squid in transparent proxy setup let you capture all the data being transmitted? Admittedly I haven't checked or used Squid except for regular proxy type use.

    I'm not sure. I looked a little more into it, and there appear to be a couple issues. First, caching isn't really what we want, so much as complete data logging. Second, transparent proxying of SSL appears to be a frequent question asked about squid...always with the answer "no, it doesn't do that...that's a man in the middle attack".

    So, I took a look to see if there was software specifically to do that sort of thing...use the "evil" software out there for good purposes. :) It looks like there are several packages that might work. "dsniff" is one package that looks promising, and is even in the Ubuntu repository. Installing it, it looks like what you'd want. It has "dnsspoof" if you need it for dns redirection, and webmitm for transparent proxying of SSL connections. By default, it looks like it only is capturing "webmail logins and form submissions" (for legitimate purposes only??), but the source is BSD licensed. Seem like it would be pretty trivial to start with that and just log all data instead, then write separate code to parse the (yay) XML data records in your text/XML friendly language of choice.

    I'm itching to pull the trigger on this solar system, but really don't want to be stuck without access to my own data (and I don't really see a better alternative to Enphase's products). Their site sure looks nice, and I plan on getting the subscription, but I'd like to play with the data myself as well. It looks like a lot of the Enlighten displays are in Flash, which would make even screen-scraping their service to generate data logs problematic. So, I hope a solution like this would work...
  • dwh
    dwh Solar Expert Posts: 1,341 ✭✭✭
    Re: Enphase Data Access
    sandeen wrote: »
    Maybe it's something other than ctrl-c now?


    Sorry, I haven't been around much and I'm playing catchup. I love this thread. :D

    Mebbe this will help:

    http://www.dd-wrt.com/phpBB2/viewtopic.php?p=386316

    "The reason you can't hit Ctrl-C in RedBoot is because linemode is disabled in telnet by default. On a *nix box simply make a .telnetrc file in your home folder and put this in the file:
    Code:
    192.168.1.254 9000
    mode line

    to enable linemode."

    AND

    "I got the linemode tip from the OpenWrt documentation."


    [EDIT: I don't think you actually put the "Code:" part in .telnetrc.]
  • dwh
    dwh Solar Expert Posts: 1,341 ✭✭✭
    Re: Enphase Data Access
    So, I took a look to see if there was software specifically to do that sort of thing...use the "evil" software out there for good purposes. :)

    You white hat you.
    It looks like there are several packages that might work. "dsniff" is one package that looks promising, and is even in the Ubuntu repository. Installing it, it looks like what you'd want. It has "dnsspoof" if you need it for dns redirection, and webmitm for transparent proxying of SSL connections. By default, it looks like it only is capturing "webmail logins and form submissions" (for legitimate purposes only??), but the source is BSD licensed. Seem like it would be pretty trivial to start with that and just log all data instead, then write separate code to parse the (yay) XML data records in your text/XML friendly language of choice.

    What you left out is ssldump - that's how you do the logging - gotta have that. :)


    [EDIT: BTW, I recently setup a server where the client specified Ubuntu 10.04 Server and I *HATED* it. Firing up X on every runlevel was my first clue that these Canonical guys had screwed up badly...and then finding out that ufw can't do a redirect and having to rip out ufw so I could run a normal iptables setup just added to the fun. And don't even get me started on that goofy gnome ethernet config dialog that requires you to manually input the MAC and does NOT stick its config info in /etc/network/interfaces where it BLOODY WELL BELONGS!

    GAH! h8@ubuntu

    FYI, BackTrack Linux (what I run on my laptop and netbook) comes with all the um..."penetration testing" tools installed. Very handy for um...penetration testing...yea...that's the ticket!

    http://en.wikipedia.org/wiki/BackTrack

    ]
  • System2
    System2 Posts: 6,290 admin
    Re: Enphase Data Access
    Their site sure looks nice, and I plan on getting the subscription, but I'd like to play with the data myself as well. It looks like a lot of the Enlighten displays are in Flash, which would make even screen-scraping their service to generate data logs problematic. So, I hope a solution like this would work...


    You can access a flash-free (but somewhat limited) version of their site if you pull it up on an iPhone (and probably other mobile devices). For the purposes of screen-scraping you just need to provide an appropriate User-Agent string and Accept: header. From there you can get the day's power output, monthly total, and some other information.

    The raw datapoints are embedded within some JSON so you can also discover the direct URLs to that data and perhaps use it to do something interesting.

    I screen-scrape my data and broadcast it to twitter for my personal amusement. http://twitter.com/Solar10300
  • sandeen
    sandeen Solar Expert Posts: 48 ✭✭✭
    Re: Enphase Data Access
    PxT. wrote: »
    You can access a flash-free (but somewhat limited) version of their site if you pull it up on an iPhone (and probably other mobile devices). For the purposes of screen-scraping you just need to provide an appropriate User-Agent string and Accept: header. From there you can get the day's power output, monthly total, and some other information.

    The raw datapoints are embedded within some JSON so you can also discover the direct URLs to that data and perhaps use it to do something interesting.

    I screen-scrape my data and broadcast it to twitter for my personal amusement. http://twitter.com/Solar10300

    I screenscrape the built-in webserver for data, and upload it to pachube.com:

    http://www.pachube.com/feeds/8145

    (not very interesting yet since I'm still waiting 15 business days for the new meter to get installed, grrrr)

    If anyone is interested in the perl script to do this I'll happily share it. I just launch it every 5min from a cron job.
  • sandeen
    sandeen Solar Expert Posts: 48 ✭✭✭
    Re: Enphase Data Access

    This doesn't further the cause of getting into the unit, but you can make pretty graphs just by screenscraping the Envoy. I wrote up a blog post about how to use pachube.com to make some pretty graphs of current and historical output.

    Screen-shot-2010-07-03-at-9.29.35-AM-300x211.png
  • AntronX
    AntronX Solar Expert Posts: 462 ✭✭
    Re: Enphase Data Access

    Hey Sandeen, nice blog. Do you host everything from your house?
  • sandeen
    sandeen Solar Expert Posts: 48 ✭✭✭
    Re: Enphase Data Access
    AntronX wrote: »
    Hey Sandeen, nice blog. Do you host everything from your house?

    Yeah, for now. It doesn't get so much traffic that it's a huge problem yet. ;) It's on the do-everything machine in the basement.
  • AntronX
    AntronX Solar Expert Posts: 462 ✭✭
    Re: Enphase Data Access

    I browsed through your projects, are you working on anything solar related now, or planning to?
  • sandeen
    sandeen Solar Expert Posts: 48 ✭✭✭
    Re: Enphase Data Access
    AntronX wrote: »
    I browsed through your projects, are you working on anything solar related now, or planning to?

    (hate to get too off topic for the thread so I'll send you a priv msg)
  • dak664
    dak664 Registered Users Posts: 14 ✭✭
    Re: Enphase Data Access

    I got an M190 to see if I could pick up data from the power line. For detection a 15nf cap and 1:1 transformer, ultimately a sound card could decode traffic.
    But I don't see any modulation on the oscilloscope so I suspect it is silent until queried :(

    Oh well, it makes use of two old SP75 panels and should pay back in 20 years.
  • drees
    drees Solar Expert Posts: 482 ✭✭✭
    Re: Enphase Data Access
    dak664 wrote: »
    Oh well, it makes use of two old SP75 panels and should pay back in 20 years.
    I'm guessing you hooked up those panels in series to the inverter?

    And yes, from what I've seen, it looks like the Enphase EMU polls the inverters periodically to grab the data from them.
  • dak664
    dak664 Registered Users Posts: 14 ✭✭
    Re: Enphase Data Access

    Yes. in series. During the heat of the day the mppt is selecting 23 volts at 2 amps! Hopefully it will harvest more power this fall.
    Would be nice if someone could capture some of the power line traffic...
  • System2
    System2 Posts: 6,290 admin
    Re: Enphase Data Access

    I have to get out my thinking cap on embedded Linux... Like many here, I am deep in the throughs of trying to determine what system I want to use. For some reason I really really don't want to go down the EnPhase route if I cannot gain access to the data. I personally don't like that type of corporate policy and I spend my every day at work writing software so go figure.

    Anyhow, I did contact EnPhase today because they never responded to a request about how to get data within the Envoy. The support guy was polite but curt. He said they would never provide the data that I was looking for. It was only available if I subscribed to Enlighten.

    I would love to hear if anyone has anymore news. Ditto on the use of GPL. I almost asked the rep today but had to go to a meeting.