Oh well. A few days after I’ve left Berlin for a while, I lost connectivity to my appartment’s equipment. While the DSL modems were still working (and accessible, thanks to their VPN), I totally lost access to everything else. Fortunately, I was able to resurrected it remotely in the end…
I was using a Seagate Dockstar as my router and OpenVPN termination point since I moved to Berlin part-time 3+ years ago (well, today, I do have DSL connectivity there, including ISDN and VoIP lines, but initially, the setup described in July 2010 worked nicely ;)), but just on the night before I was heading out of town for summer vacation, it died on me. It’s totally dead, nothing even on the serial console …
Fortunately, I already had prepared and tested a filesystem image for a Raspberry Pi some time ago, but I wasn’t too excited about it’s performance initially — OpenVPN really sucks on it’s CPU, and the USB performance is … well, OK-ish for a school project I assume. But, well, at a time like that, there’s not much choice ;)
So I quickly installed a RPi as a replacement router, and it worked flawlessly all the three weeks of my summer vacation. Unfortunately, I had to go to the hospital shortly after vacation, and that’s when this RPi decided to die on me, as it turned out.
Initially, I thought the bloody Netgear GS108T braindied again, and tonight I wanted to find out what I could get to work again remotely.
What still was working were the VPNs between my Fritz!Box 7570 in Gütersloh and both the 7570 (Alice 10/1 ADSL2) and the 7360 (congstar 50/10 VDSL2) in Berlin; connectivity between them was severed, though.
wusel@death:~$ traceroute 192.168.176.1 traceroute to 192.168.176.1 (192.168.176.1), 64 hops max, 40 byte packets 1 albert.uu.org (220.127.116.11) 0 ms 0 ms 0 ms 2 FB-VDSL-GTSO.uu.org (192.168.177.1) 1 ms 0 ms 1 ms 3 FB-VDSL-BRLN.uu.org (192.168.176.1) 44 ms 44 ms 43 ms wusel@death:~$ traceroute 192.168.175.1 traceroute to 192.168.175.1 (192.168.175.1), 64 hops max, 40 byte packets 1 albert.uu.org (18.104.22.168) 0 ms 0 ms 0 ms 2 FB-VDSL-GTSO.uu.org (192.168.177.1) 1 ms 0 ms 0 ms 3 FB-DSL-BRLN.uu.org (192.168.175.1) 66 ms 65 ms 82 ms
By using some trickery, I was able to enable telnetd on the remote Fritz!Boxes – you need to enter #96*7* via a phone connected via ISDN or POTS to the Fritz!Box — tough, when being 400 km away ;) But there’s a well known trick to get the FB to dial this for you, so Google was my fried again to refresh my memories –, which gave me access to that remote site at least. A Fritz!Box is a nifty, Linux-driven device, and it comes with ping, traceroute, wget and nc from BusyBox, so it’s at least a starting point to investigate a remote network …
Pinging the Raspberry’s address from either FBs again pointed to the Netgear switch being braindead: no response. Fortunately though, from the times I used the Dockstar as router, my laptop had Fritz!Box-local IPs as well – the Dockstar kernel from 2010 wasn’t set up for source routing, so I did this on the laptop for some time – and these did ping from both FBs. Which, thankfully, ruled out the Netgear (which will be replaced once I’m back nonetheless; it’s a piece of crap).
So, where am I? I was connected via telnet over VPN to a Fritz!Box in Berlin, and the only protocol my laptop speaks is SSH. Which, surprisingly ;), is not built into the Fritz!Box’ busybox binary (nor is telnet).
There is “nc”, but only in it’s active form (no “-l” option). I can port-forward from the public DSL IP to my laptop — but as that one has the default gateway set to the Raspberry, it’s no use either: it will sent it’s answers into the big bad bit bucket.
If I could only have ssh for the Fritz!Box … Oh, wait. That one is actually available: AVM’s Fritz!Box is kind of the CPE in Germany ever since we got DSL, and there’s a project called “Freetz” which allows compilation of tools to run in the Fritz!Box environment.
To make a long story short: I found a working dropbear (lightweight ssh/sshd/scp implementation) binary in a post on ip-phone-forum, downloaded it, moved it to a publically accessible web space, and the rest is straight forward:
# cd /var/tmp # wget http://death.uu.org/~wusel/dropbearmulti_mipsel_static Connecting to death.uu.org (22.214.171.124:80) dropbearmulti_mipsel 100% |*******************************| 355k 00:00:00 ETA # chmod +x dropbearmulti_mipsel_static # ./dropbearmulti_mipsel_static Dropbear multi-purpose version 2012.55 Make a symlink pointing at this binary with one of the following names: 'dropbear' - the Dropbear server 'dbclient' or 'ssh' - the Dropbear client 'dropbearkey' - the key generator 'scp' - secure copy # ln -s dropbearmulti_mipsel_static dropbear # ln -s dropbearmulti_mipsel_static ssh # ./ssh firstname.lastname@example.org ./ssh: Warning: failed creating //.ssh: Read-only file system Host '192.168.175.4' is not in the trusted hosts file. (fingerprint md5 7e:[…]:63) Do you want to continue connecting? (y/n) yes email@example.com's password: Linux greebo 2.6.32-31-generic #61-Ubuntu SMP Fri Apr 8 18:24:35 UTC 2011 i686 GNU/Linux […]
So, that at least gave me access to my laptop, a much better suited device for checking what went wrong. I added a route to one of my boxes via the Fritz!Box’ internal IP, enabled port forwarding on that FB and that finally gave me comfortable access to my laptop, to see what went wrong there.
Fortunately, before I started to reconfigure the laptop as a temporary router (including OpenVPN termination, DHCP services etc.), I tried to ping the Raspberry from this side:
root@greebo:~# ping gw PING gw.berlin.uu.org (126.96.36.199) 56(84) bytes of data. 64 bytes from gw.berlin.uu.org (188.8.131.52): icmp_seq=1 ttl=64 time=0.529 ms 64 bytes from gw.berlin.uu.org (184.108.40.206): icmp_seq=2 ttl=64 time=0.494 ms 64 bytes from gw.berlin.uu.org (220.127.116.11): icmp_seq=3 ttl=64 time=0.462 ms ^C --- gw.berlin.uu.org ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 1998ms rtt min/avg/max/mdev = 0.462/0.495/0.529/0.027 ms
WTF?! And indeed, from THAT side of the network, the RPi was happily accessible even via SSH. No b0rked ext4 filesystem (which I was kind of expecting), but there were some interface aliases missing, which was the reason for the Raspberry not being accessible from the Fritz!Boxes. And that seems to be caused by a bloody USB issue on 2013-08-25:
Aug 25 20:37:21 gw-pi kernel: [2714301.574854] asix 1-1.3.4:1.0: eth1: unregister 'asix' usb-bcm2708_usb-1.3.4, ASIX AX88178 USB 2.0 Ethernet Aug 25 20:37:23 gw-pi kernel: [2714304.225990] asix 1-1.3.4:1.0: eth1: register 'asix' at usb-bcm2708_usb-1.3.4, ASIX AX88178 USB 2.0 Ethernet, 00:22:75:d6:be:f2
Since most devices these days are GBit ones, I decided to not use the build-in (USB connected) 100 MBit port of the Raspberry but use a GBit USB dongle at a powered USB hub instead. Even at only 480 MBit/sec USB 2.0 speeds, this should be much faster that the (USB 2.0 connected) 100 MBit/sec ethernet port, so hopefully the bottleneck would be lesser — the Dockstar featured a PCIe connected GBit port … Turns out I lowered the long-term average bandwidth by this decision *sigh*.
The “fix” to restore connectivity was simple: “ifup eth1:3”, “ifup eth1:4” — and some seconds later normality was restored.
Me and the Raspberry. It remains complicated …
As soon as I’m back in my flat in Berlin, I’ll check whether the Buffalo WZR-HP-AG300H would be a proper substitute as a main router (including OpenVPN termination); I’ve put OpenWRT on it already, but transforming Debian-style network configuration to OpenWRT’s way of doing things, well, that will take time. I’m not too excited, though, as trying to replace one Dockstar by another OpenWRT-capable router, the TL-WR1043ND, turned out to be a no-go option already in terms of OpenVPN performance.
I’m currently waiting for the quad-core RockChip 3188 stuff to settle (I’ve already secure a MK 802 IV), which looks like being an option in the not too distant future. If you see a dual- or quad-core board with non-USB connected Gigabit Ethernet and the option of running (a Debian-flavor of) Linux for less that EUR/USD 100, please drop me a note …