Can no longer access B3 server

[jeff]

A couple of weeks ago we had a power outage. Afterwards I had some issues with my modem/router when I tried to plug the network cable into my LAN port on the B3 the modem would continually reset and try to reconnect to the internet and attached devices.

Thinking the issue was with the modem/router I managed a factory reset and it started to behave again.

Almost straight away I started experiencing issues accessing my B3 over wifi, I could access the web admin interface via my local network and also via the internet but was unable to connect directly to the shared files from my windows laptop. I constantly had an error after entering my username and password (I saved it somewhere but I can't find it now).

I did another restart of my B3 but now unfortunately I can't access it at all. I can see through my modem/routers DHCP list that it cannot connect to the B3 server at all. I can see my SSID but cannot connect.

Now the only way I can connect to my B3 server is via booting the rescue disk. Using SSH I can see that my server appears to have been in bad shape for quite some time. There are constant "No space left on device" warning messages mostly from postfix:

Code: Select all

postfix/cleanup[1559]: warning: 3559E4A241: write queue file: No space left on device

but also from other systems:

Code: Select all

Jan 18 23:36:59 b3 bubba-igd: Application starting
Jan 18 23:36:59 b3 bubba-igd: Starting IGD UPNP service
Jan 18 23:37:01 b3 dnsmasq[1243]: started, version 2.55 cachesize 150
Jan 18 23:37:01 b3 dnsmasq[1243]: compile time options: IPv6 GNU-getopt DBus I18N DHCP TFTP
Jan 18 23:37:01 b3 dnsmasq[1243]: warning: interface br0 does not currently exist
Jan 18 23:37:01 b3 dnsmasq-dhcp[1243]: DHCP, IP range 192.168.10.50 -- 192.168.10.100, lease time 12h
Jan 18 23:37:01 b3 dnsmasq[1243]: no servers found in /etc/resolv.conf, will retry
Jan 18 23:37:01 b3 dnsmasq[1243]: read /etc/hosts - 7 addresses
Jan 18 23:37:14 b3 Exception thrown :(#012Traceback (most recent call last):#012  File "/usr/sbin/bubba-album-inotifyd", line 200, in 

<module>#012    notifier.loop(daemonize=True)#012  File "/usr/lib/python2.6/dist-pack
ages/pyinotify.py", line 1397, in loop#012    self.__daemonize(**args)#012  File "/usr/lib/python2.6/dist-packages/pyinotify.py", line 

1355, in __daemonize#012    os.write(fd_pid, str(os.getpid()) + '\n')#012OSError: [E
rrno 28] No space left on device

Jan 22 16:18:11 b3 avahi-daemon[1275]: write(): No space left on device
Jan 22 16:18:11 b3 avahi-daemon[1275]: Failed to create PID file: No space left on device

After mounting the drive I can see from a df that the drive appears to be almost full

Code: Select all

# df -h /mnt
Filesystem                Size      Used Available Use% Mounted on
/dev/sda1                 9.8G      9.7G         0 100% /mnt
#

I have tried removing some files here and there (mainly logs) but so far nothing has helped (I'm reluctant to delete much in fear of making things worse)

I am confident there is nothing wrong with the drive (my external backup still runs successfully) I just can't get network access.

If someone could offer some ideas it would be most appreciated?

Cheers
Jeff

[Gordon]

Did you try the search button?

A good candidate for solving your problem is here: Why is /dev/sda1 full?

[jeff]

Hi Gordon

Thanks for the suggestion, I was actually trying to follow this thread viewtopic.php?f=9&t=4245 but I've had no luck.

My /mnt/var/spool/postfix/maildrop/ directory is only 205.9M (857 files) and I can't run any commands to clear them as I can't access the server without the rescue disk.

The largest directories I can find include:

Code: Select all

6.0G        /mnt/var/lib/album/thumbs/hdtv
1.1G        /mnt/var/lib/album/thumbs/rescaled
212.6M        /mnt/var/lib/album/thumbs/thumbs

I do have a lot of media on the server but I'm not sure if this is normal?
Is it ok to delete the files in these directories?
I assume there is a setting somewhere I can use to turn off the generation of these files?

I did also try removing the 358 files from the /mnt/root/Mail location as was suggested in the post you recommended but that doesn't look to have made much difference as I still see the same messages in the syslog.

Cheers
Jeff

[Gordon]

The search given will return other results as well.

Seems you are using bubba-album? Those thumbs will do it. The size of the root partition is only 10GiB in the original B3 system.

It should be okay to simply delete those files in /var/spool/postfix/maildrop, although you might want to take a peek in some of them to see what message they contain and to who they are supposed to be send. Once you can start your system again your first goal should be to move those thumbs dirs to the storage partition and create a link to that new location from where they are now.

[jeff]

I've cleared the maildrop and moved the album to the storage partition. I am now using less than 3gb of the root partition.

I have also discovered that if I plug my network cable from the router to WAN port in the B3 and start the server without the rescue USB, the device is picked up by my router in the DHCP list. I can access SSH and the web interface (although not as the Administrator, I get an error saying this is not allowed while connected via WAN). I cannot access the file share or connect to WIFI.

If I start the B3 with the network cable plugged into the LAN port (as it should be) I cannot see the device in my routers DHCP list and cannot access the web interface, WIFI or the file share.

How can I verify the networking configuration is correct?

[Gordon]

Was wifi enabled? Do you see its SSID? Then you must have a fixed IP on the LAN side. If you can ssh into the B3 from the WAN side you can verify network addresses with command

Code: Select all

ip addr

In the list returned you'll see adapters 'lo' (localhost), 'eth0' (WAN), 'eth1' (LAN), 'wlan0' (wifi). If wifi is correctly enabled there should be no address on eth1 and wlan0 and there will be a fifth adapter named 'br0' that owns the LAN IP.

If that is all good check the firewall by using the command (you must be root for this)

Code: Select all

iptables-save

Verify that the output contains a line that says

-A INPUT -i br0 -j ACCEPT

Missing? run

Code: Select all

iptables -I INPUT 4 -i br0 -j ACCEPT

You should now be able to access the SSID

[jeff]

I can confirm that the SSID is available, I just can't connect.

The output of 'ip addr' is as follows:

Code: Select all

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 00:22:02:00:28:5c brd ff:ff:ff:ff:ff:ff
    inet 10.1.1.2/24 brd 10.1.1.255 scope global eth0
    inet6 fe80::222:2ff:fe00:285c/64 scope link
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state UNKNOWN qlen 1000
    link/ether 00:22:02:00:28:5d brd ff:ff:ff:ff:ff:ff
4: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master br0 state U                                                             NKNOWN qlen 32
    link/ether 00:0b:6b:e4:ee:a3 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::20b:6bff:fee4:eea3/64 scope link
       valid_lft forever preferred_lft forever
5: mon.wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UNKNOWN                                                              qlen 32
    link/ieee802.11/radiotap 00:0b:6b:e4:ee:a3 brd ff:ff:ff:ff:ff:ff
6: br0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN
    link/ether 00:0b:6b:e4:ee:a3 brd ff:ff:ff:ff:ff:ff

The output of 'iptables-save' is as follows:

Code: Select all

*nat
:PREROUTING ACCEPT [585:67119]
:INPUT ACCEPT [3:188]
:OUTPUT ACCEPT [496:26587]
:POSTROUTING ACCEPT [5:260]
-A POSTROUTING -o eth0 -j MASQUERADE
COMMIT
# Completed on Thu Jan 26 14:21:28 2017
# Generated by iptables-save v1.4.8 on Thu Jan 26 14:21:28 2017
*filter
:INPUT DROP [588:68056]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [1303:93613]
-A INPUT -p tcp -m tcp --tcp-flags SYN,ACK SYN,ACK -m state --state NEW -j REJECT --reject-with tcp-reset
-A INPUT -p tcp -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -m state --state NEW -j DROP
-A INPUT -i eth0 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -i br0 -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -i eth0 -p icmp -m icmp --icmp-type 11 -j ACCEPT
-A INPUT -i eth0 -p icmp -m icmp --icmp-type 8 -j ACCEPT
-A INPUT -p icmp -m icmp --icmp-type 3/4 -j ACCEPT
-A INPUT -i eth0 -p tcp -m tcp --dport 22 -j ACCEPT
-A INPUT -i eth0 -p tcp -m tcp --dport 80 -j ACCEPT
-A INPUT -i eth0 -p tcp -m tcp --dport 443 -j ACCEPT
-A INPUT -i eth0 -p tcp -m tcp --dport 143 -j ACCEPT
-A INPUT -i eth0 -p tcp -m tcp --dport 993 -j ACCEPT
-A FORWARD -i br0 -j ACCEPT
-A FORWARD -m state --state RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -p icmp -m icmp --icmp-type 3/4 -j ACCEPT
COMMIT

I can see all the references mentioned. Is there anything else in the output that could help?

[Gordon]

There's no IP address assigned to br0

Your best option for fixing this is probably to enable admin login on the WAN interface so you can use the web frontend to set an IP. To do this use 'nano' to edit the file /home/admin/.bubbacfg and add the following line:

Code: Select all

AllowRemote=yes

[jeff]

I did as you suggested and changed the Network setting to "Router - Firewall - Server". On clicking update immediately I lost the connection to the web interface and cannot get back.

I can still access the server via SSH when connected to the WAN port but no longer through the web interface (connecting via WAN or LAN).

Any ideas?

[Gordon]

Code: Select all

iptables -A INPUT -i eth0 -p tcp --dport 80 -j ACCEPT

[jeff]

Thanks Gordon, that appears to have done the trick.

My wifi, web access and network access are all working as before.

Is there any way I can offer some payment as appreciation for your help?

[Gordon]

No worries there. Good to hear you got it sorted.