Easyfind not updating changed IP

[bIO]

Hello,

I have a problem with the Easyfind service and I hope someone can help me troubleshoot this situation.

My B3 is working in Server mode behind a combined router/modem device. When I activate the Easyfind service through the Web interface I can access my router by supplying the <name>.myownb3.com address.

The moment my IP changes (assigned by ISP), this change seems not to be propagated to the Easyfind service, however, the B3 looks like it is aware of the change.

Here is a snippet from syslog. The IP addresses are (of course) made up, but you see that the change is noticed (18:41:26):

Code: Select all

Sep 26 18:40:01 b3 /USR/SBIN/CRON[17895]: (root) CMD (/etc/init.d/dovecot status >/dev/null 2>&1 || /etc/init.d/dovecot restart)
Sep 26 18:40:01 b3 /USR/SBIN/CRON[17896]: (root) CMD (test -x /usr/lib/web-admin/notify-dispatcher.pl && /usr/lib/web-admin/notify-dispatcher.pl)
Sep 26 18:40:01 b3 /USR/SBIN/CRON[17897]: (root) CMD (test -x /usr/bin/php && /usr/bin/php /usr/share/horde3/scripts/alarms.php)
Sep 26 18:41:26 b3 twisted: [HTTP11ClientProtocol,client] Got new IP '111.111.111.111' which is not the same as the last one '000.000.000.000'
Sep 26 18:41:26 b3 bubba-networkmanager: Starting up
Sep 26 18:42:26 b3 twisted: [HTTP11ClientProtocol,client] Got new IP '111.111.111.111' which is the same as the last one '111.111.111.111'
Sep 26 18:42:26 b3 bubba-networkmanager: Server timed out, terminating
Sep 26 18:42:26 b3 bubba-networkmanager: Daemon terminating
Sep 26 18:42:26 b3 bubba-networkmanager: Shutting down
Sep 26 18:43:26 b3 twisted: [HTTP11ClientProtocol,client] Got new IP '111.111.111.111' which is the same as the last one '111.111.111.111'
Sep 26 18:44:26 b3 twisted: [HTTP11ClientProtocol,client] Got new IP '111.111.111.111' which is the same as the last one '111.111.111.111'
Sep 26 18:45:01 b3 /USR/SBIN/CRON[17912]: (root) CMD (/etc/init.d/dovecot status >/dev/null 2>&1 || /etc/init.d/dovecot restart)
Sep 26 18:45:01 b3 /USR/SBIN/CRON[17913]: (root) CMD (test -x /usr/lib/web-admin/notify-dispatcher.pl && /usr/lib/web-admin/notify-dispatcher.pl)
Sep 26 18:45:01 b3 /USR/SBIN/CRON[17914]: (root) CMD (test -x /usr/bin/php && /usr/bin/php /usr/share/horde3/scripts/alarms.php)
Sep 26 18:45:26 b3 twisted: [HTTP11ClientProtocol,client] Got new IP '111.111.111.111' which is the same as the last one '111.111.111.111'

If I would let it sit like this, all the messages would go on, saying "new IP is the same as the last one", but the Easyfind service is still pointing to my old IP address.

If I just click "Update Easyfind" under Settings>Identity in the Web interface (without disabling and enabling), the following appears in syslog. It seems to detect the change again, but this time the change is propagated to the Easyfind service and I can access the B3 through <name>.myownb3.com:

Code: Select all

Sep 26 18:46:26 b3 twisted: [HTTP11ClientProtocol,client] Got new IP '111.111.111.111' which is not the same as the last one '000.000.000.000'
Sep 26 18:46:26 b3 bubba-networkmanager: Starting up
Sep 26 18:47:26 b3 twisted: [HTTP11ClientProtocol,client] Got new IP '111.111.111.111' which is the same as the last one '111.111.111.111'
Sep 26 18:47:26 b3 bubba-networkmanager: Server timed out, terminating
Sep 26 18:47:26 b3 bubba-networkmanager: Daemon terminating
Sep 26 18:47:26 b3 bubba-networkmanager: Shutting down

I have read that for the B3 to be able to detect changes it has to be connected through the WAN port (which is not the case in my situation) or to be able to communicate to my router/modem. The corresponding setting (Services>Other>Remote access through other router) is enabled in B3, but there is no UPnP related setting in my router. I tried opening all ports for the B3 (Server mode in router) but that also did not help.

Is there anything I can do to get Easyfind working? Maybe someone can help me with the inner workings of what service invokes what methods so I can go on from there.
---
bIO

[Gordon]

I think I wrote that comment about Easyfind not working as expected when only connected on the LAN port. I can't remember the specifics at this time though. Your logs show that in fact something is happening and the clue appears to be in the timeout message, which does seem to suggest that something is in fact trying to accomplish something by accessing the WAN interface that is not connected to anything.

What I do know is that the twisted process is the bubba-easyfind service and that it is a python script. It is readable therefore and by reading it you may find what's causing your issue - like a hardcoded eth0 or a call to `bubba-networkmanager getwanif` which also resolves to eth0.

[Ubi]

but you see the timeout also in the logs after a manual update to easyfind, which apparently does succeed. This suggests the timeout is not directly responsible for the lack of automatic updates

[Gordon]

Hadn't looked that far. Still with the easyfind service being oriented on eth0 one would expect the manual update to behave the same way as the automated method, so seeing the timeout there is not that much of a surprise. Quite possibly though the manual method, which does not make use of twisted.py, has been programmed to take the secondary route when the first one ends in error. Twisted.py not seeming to have this fallback is just one of those typical oversights that do happen when using multiple methods for a single goal.

[Ubi]

Code: Select all

    interface = netifaces.ifaddresses(WAN)
    if netifaces.AF_LINK in interface:
        mac0 = interface[netifaces.AF_LINK][0]['addr']
    d = httpRequest(
        "https://easyfind.excito.org",
        {
            'key': key,  
            'mac0': mac0,
        },
        method='POST',
        headers={'Content-Type': ['application/x-www-form-urlencoded']}
    )
    d.addCallback(easyfind_ip_updated)
    d.addErrback(err)

This seems to be the interesting part. It seems that mac addr of the wan is submitted to the easyfind server, which is hardcoded halfway the script for some reason. But the request itself does not seem to be forced over the wan interface, suggesting that having the b3 connected via lan should not be the issue.

[bIO]

Thanks Gordon and Ubi for looking into the issue.

I have taken some steps yesterday and flashed my modem/router combo with another image that allowed me to enable UPnP configuration. There are three different configuration options now, from letting the router/modem supply status infos, to allowing security settings to be updated by external programs.

Before, syslog showed the following after disabling/enabling Easyfind through the web interface:

Code: Select all

1127 Sep 28 17:44:44 b3 bubba-igd: Stopping IGD UPNP service
1128 Sep 28 17:44:44 b3 bubba-igd: Application starting
1129 Sep 28 17:44:44 b3 bubba-igd: Starting IGD UPNP service
1130 Sep 28 17:44:54 b3 bubba-igd: Stopping IGD UPNP service
1131 Sep 28 17:44:54 b3 bubba-igd: Application starting
1132 Sep 28 17:44:54 b3 bubba-igd: Starting IGD UPNP service

With some options enabled, it showed:

Code: Select all

Sep 29 05:10:07 b3 bubba-igd: Stopping IGD UPNP service
Sep 29 05:10:07 b3 bubba-igd: Application starting
Sep 29 05:10:07 b3 bubba-igd: Starting IGD UPNP service
Sep 29 05:10:07 b3 bubba-igd: Device available: uuid:<uuid> at <local-IP-of-router/modem>
Sep 29 05:10:07 b3 bubba-igd: Service available: uuid:<uuid> at <external-IP>
Sep 29 05:10:07 b3 bubba-igd: Error: Not available Action
Sep 29 05:10:07 b3 bubba-igd: Error: Not available Action
Sep 29 05:10:07 b3 bubba-igd: Opened port mapping <local-IP-of-B3>:80 <-> <external-IP>:80
Sep 29 05:10:07 b3 bubba-igd: Error: Not available Action
Sep 29 05:10:07 b3 bubba-igd: Error: Not available Action
Sep 29 05:10:07 b3 bubba-igd: Opened port mapping <local-IP-of-B3>:443 <-> <external-IP>:443
Sep 29 05:10:19 b3 twisted: [HTTP11ClientProtocol,client] Got new IP '<external-IP>' which is the same as the last one '<external-IP>'
Sep 29 05:10:19 b3 bubba-igd: Stopping IGD UPNP service
Sep 29 05:10:19 b3 bubba-igd: Service unavailable: uuid:<uuid>
Sep 29 05:10:19 b3 bubba-igd: Error: Not available Action
Sep 29 05:10:19 b3 bubba-igd: Error: Not available Action
Sep 29 05:10:19 b3 bubba-igd: Closed port mapping <local-IP-of-B3>:80 <-> <external-IP>:80
Sep 29 05:10:19 b3 bubba-networkmanager: Server timed out, terminating
Sep 29 05:10:19 b3 bubba-networkmanager: Daemon terminating
Sep 29 05:10:19 b3 bubba-networkmanager: Shutting down
Sep 29 05:10:19 b3 bubba-igd: Error: Not available Action
Sep 29 05:10:19 b3 bubba-igd: Error: Not available Action
Sep 29 05:10:19 b3 bubba-igd: Closed port mapping <local-IP-of-B3>:443 <-> <external-IP>:443
Sep 29 05:10:19 b3 bubba-igd: Device unavailable: uuid:<uuid>
Sep 29 05:10:19 b3 bubba-igd: Application starting
Sep 29 05:10:19 b3 bubba-igd: Starting IGD UPNP service
Sep 29 05:10:19 b3 bubba-igd: Device available: uuid:<uuid> at <local-IP-of-router/modem>
Sep 29 05:10:19 b3 bubba-igd: Service available: uuid:<uuid> at <external-IP>
Sep 29 05:10:19 b3 bubba-igd: Error: Not available Action
Sep 29 05:10:19 b3 bubba-igd: Error: Not available Action
Sep 29 05:10:19 b3 bubba-igd: Opened port mapping <local-IP-of-B3>:80 <-> <external-IP>:80
Sep 29 05:10:20 b3 bubba-igd: Error: Not available Action
Sep 29 05:10:20 b3 bubba-igd: Error: Not available Action
Sep 29 05:10:20 b3 bubba-igd: Opened port mapping <local-IP-of-B3>:443 <-> <external-IP>:443
Sep 29 05:10:21 b3 bubba-igd: Got easyfind data: {"error":true,"opcode":16,"msg":"Unable to validate from production."}

With the two UPnP security options enabled, the B3 succeeded in opening TCP and UDP ports 80 and 443:

Code: Select all

Sep 29 05:25:19 b3 bubba-igd: External IP changed: uuid:<uui> at <old-external-IP>
Sep 29 05:25:19 b3 bubba-igd: Opened port mapping <local-IP-of-B3>:80 <-> <old-external-IP>:80
Sep 29 05:25:19 b3 bubba-igd: Opened port mapping <local-IP-of-B3>:443 <-> <old-external-IP>:443
Sep 29 05:25:19 b3 twisted: [HTTP11ClientProtocol,client] Got new IP '<new-external-IP>' which is the same as the last one '<new-external-IP>'
Sep 29 05:25:20 b3 bubba-igd: Got easyfind data: {"error":true,"opcode":16,"msg":"Unable to validate from production."}
Sep 29 05:25:20 b3 bubba-igd: External IP changed: uuid:<uui> at 0.0.0.0
Sep 29 05:25:20 b3 bubba-igd: Opened port mapping <local-IP-of-B3>:80 <-> 0.0.0.0:80
Sep 29 05:25:20 b3 bubba-igd: Opened port mapping <local-IP-of-B3>:443 <-> 0.0.0.0:443
Sep 29 05:25:20 b3 bubba-igd: Got easyfind data: {"error":true,"opcode":16,"msg":"Unable to validate from production."}
Sep 29 05:25:20 b3 bubba-igd: External IP changed: uuid:<uui> at <new-external-IP>
Sep 29 05:25:20 b3 bubba-igd: Opened port mapping <local-IP-of-B3>:80 <-> <new-external-IP>:80
Sep 29 05:25:20 b3 bubba-igd: Opened port mapping <local-IP-of-B3>:443 <-> <new-external-IP>:443
Sep 29 05:25:21 b3 bubba-igd: Got easyfind data: {"error":true,"opcode":16,"msg":"Unable to validate from production."}

I suppose the error that is thrown now ("Unable to validate from production."), relates to the MAC address not being found on the remote server?

Nevertheless, despite opening the four ports in the modem/router, there is still no automatic update of a changed external IP.

Like Gordon said, it seems there are many ways to go wrong, when you do not adhere to KISS. I understand that Excito wanted to do a smart check for a changed IP and not rely on frequent IP checks. But it is a pity that most other dynamic DNS services work like a charm by a simple URL GET and Easyfind does not.

[Ubi]

In principle the Easyfind way is better as it saves a lot of traffic on the server, and may be faster to adapt to changes than a predefined window of 'pings' to retreive an IP address. But with every technology comes a liability and this seems to be it. Would be interesting to speculate whether a 'backup' routine that pings every 6 hours or so would benefit the user experience. Maybe even a choice of methods?

[Gordon]

This seems to be the interesting part. It seems that mac addr of the wan is submitted to the easyfind server, which is hardcoded halfway the script for some reason. But the request itself does not seem to be forced over the wan interface, suggesting that having the b3 connected via lan should not be the issue.

I don't think that reference has any relation to the issue. It's most likely just another method of verification, next to the hardware defined "key" parameter, that Excito uses to keep the easyfind service exclusive to B2/B3 owners.

[Gordon]

In principle the Easyfind way is better as it saves a lot of traffic on the server, and may be faster to adapt to changes than a predefined window of 'pings' to retreive an IP address. But with every technology comes a liability and this seems to be it. Would be interesting to speculate whether a 'backup' routine that pings every 6 hours or so would benefit the user experience. Maybe even a choice of methods?

That is correct. In fact all those other dynamic DNS services specifically state not to appreciate "hammering" and that they will ban you from their service if you do so anyway. The choice made here seems extremely smart and does in fact appear to work as the service logs the change right after it occurred. It's just the actual updating of the DNS server that somehow fails in the automated method.

[Gordon]

Okay, I had some spare time and took a quick look.

First off: the issue with easyfind only working on eth0 appears to be outdated and not related to the twisted method.

Looking at the script (which is not called twisted.py but bubba-easyfind.tac) I see it constructing a web call with a timeout value set at 10 seconds. The actual execution of that call is probably the one responsible for the timeout message that pops up in the log. So I started thinking whether that timeout value could be set too low, but turning my browser to that address I get almost instant response, so that's not the issue. What I did find however is that the site uses a selfsigned certificate and that most definitely explains why a scripted client would not see a HTML response within a certain time.

Looking at the web-admin code, which is PHP, I see an additional parameter passed to the web client, which is to ignore ssl verification. I do not see a similar construct in the Python script and that does explain why the manual method can work where the automated method doesn't. Also the PHP method uses GET to pass the parameters where the Python script uses POST, but if that would pose to be an issue this will likely be logged as an error.

The quick fix as I see it is to get into the scripts and change https://easyfind... to become http://easyfind... The more secure way is to somehow load the offending certificate (or its root certificate) in the trust chain of the twisted client, which I currently don't know how. That could be a nice target for software version 2.6.1

PS I'm also missing a firewall rewrite attached to this method of sniffing out a change to the public IP address. While that may not be a problem for everyone, it would be a problem for someone like myself. Being able to add hook scripts to the twisted method would be a great addition therefore.

[Ubi]

considering the price of a certificate it may be a good idea to get a proper certificate.

The firewall connection is interesting. I was always under the impression these things would be behind a router anyway, but they need not be, and then you do indeed need a firewall update upon a public IP change. This does however give some risk that a bork of the twisted script locks out the internet connection completely, which for regular users will be difficult to diagnose.

[Gordon]

The firewall connection is interesting. I was always under the impression these things would be behind a router anyway, but they need not be, and then you do indeed need a firewall update upon a public IP change. This does however give some risk that a bork of the twisted script locks out the internet connection completely, which for regular users will be difficult to diagnose.

Not quite...

If the machine you're trying to reach is holding the public IP itself then there's no need for a firewall rewrite because that public IP does in fact give direct access to that machine. The point why you would need to rewrite those rules is when you have NAT rules in your firewall to make that public IP translate into a local address - e.g. the B3 itself. You'll find this very same concept in one of the dhcp-exit-hook scripts, only there the rewrite tracks changes of the eth0 IP address inside the firewall rules.

[bIO]

The quick fix as I see it is to get into the scripts and change https://easyfind... to become http://easyfind... The more secure way is to somehow load the offending certificate (or its root certificate) in the trust chain of the twisted client, which I currently don't know how. That could be a nice target for software version 2.6.1

OK, I tried to change to "http" in the bubba-easyfind.tac Python script. That actually did not solve the problem. I then started digging deeper and tried to understand how most stuff works.

Beginning with the PHP file settings.php (CodeIgniter controller for the web-admin front-end), I realized the functions called in lines 649 and 651 are defined in networkmanager.php (CodeIgniter model) and they use a PHP HTTP request (the one with the ssl_verify_peer = false Gordon mentioned) to verify MAC address and if a given Easyfind name is free. So nothing updates the DNS at this point.

But there is a function (easyfind_setname) in networkmanager.php that calls another one in adminfunctions.php (in /usr/lib/web-admin). Now things get tricky and my head started spinning after going through all the different files (PHP, Perl and Python, this calls for errors, doesn't it ). Function set_easyfind in adminfunctions.php calls a Perl script (backend.pl) which is used to give some functions elevated privileges. This calls the easyfind.pl script with appropriate arguments.

To cut a long story somewhat shorter: Maybe nothing new, but the actual update process (at least when settings are made through the web interface) is done by easyfind.pl. Now by just calling easyfind.pl without arguments as root I was able to get Easyfind to update the DNS.

Now to get it working automatically I thought about modifying bubba-easyfind.tac and replacing easyfind_set_ip() with this piece of code, as it should be called after twisted recognizes an IP change:

Code: Select all

def easyfind_set_ip(new_ip):
    easycall = subprocess.Popen(['/usr/lib/web-admin/backend.pl', 'easyfind', 0, 0], stdout=subprocess.PIPE)
#    if easycall == 0:
#        log.msg("Easyfind update succeeded!")
#    else:
#        log.msg("Easyfind update failed!")

Unfortunately, this does not work. I also tried subprocess.call to no avail. If you have an idea, let me know.
---
bIO

[Ubi]

THis all smells like a permission problem.
What happens when you su to apache and run the backend.pl manually?

[Gordon]

THis all smells like a permission problem.
What happens when you su to apache and run the backend.pl manually?

But bubba-easyfind runs as root, so that shouldn't be an issue. Also backend.pl does not elevate rights, as the web-admin is run through fastcgi accessing the bubba-adminphp service (in a nutshell - the latter service provides three php instances running as root). Backend.pl is most likely an earlier effort to concentrate all functions and methods around a single acting module and it is somewhat weird that the bubba-easyfind service does not use this central element to perform the update.

In fact there is one huge difference between the two methods: the bubba-easyfind will update the config file regardless of whether the update fails (in fact it starts by updating that file and only then it will attempt to do the https request) and backend.pl will only update the config file if the https request is successful.

I also need to come back on this answer

That is correct. In fact all those other dynamic DNS services specifically state not to appreciate "hammering" and that they will ban you from their service if you do so anyway. The choice made here seems extremely smart and does in fact appear to work as the service logs the change right after it occurred. It's just the actual updating of the DNS server that somehow fails in the automated method.

After inspecting the python script a bit more thoroughly I have to conclude that some type of hammering is actually still happening. And from this I do not get what all these other settings have to do with this service. What the script does is perform a web request to a page that returns the originating IP address every 60 seconds and compare that address to the one stored in /etc/network/easyfind.conf. If the address is different then it calls the update procedure. So instead of trying to fix that script you could also implement this in bash and use cron to perform the update:

Code: Select all

#!/bin/bash
old_ip=$(grep '^ip' /etc/network/easyfind.conf | awk {'print $3'})
new_ip=$(wget -o /dev/null -O /dev/stdout http://ef.excito.org/ip.json| \
    grep '{"ip_address":"[0-9\.]*"}'| sed 's/{"ip_address":"\([0-9\.]*\)"}/\1/')
if [ ! -z "$new_ip" ] && [ "$old_ip" != "$new_ip" ]; then 
    /usr/lib/web-admin/easyfind.pl
fi