RAID recovery fails after reinstall

[freeman2501]

Hello forum,

I had to reinstall my B3 this week using the usb recovery method. This worked without a hassle. I had been mirroring my B3 on a Bubba storage disk using raid so after the reinstall I wanted to recover the data from my storage disk to get everything back in my home folders. Unfortunately this fails.

When I do a RAID recovery I select the storage disk (/dev/smb1) and start the recovery. Unfortunately the recovery stops at 66.67% with the following error: "Error: Operation failed: /usr/sbin/diskmanager md add /dev/md0 /dev/sda2" (cfr attached screenshot). It leaves my system borked (red led) after this failure.

This appeared in the syslog when I dit the recovery

Code: Select all

Nov  5 14:46:18 b3 kernel: [  545.934381] Out of memory: Kill process 2376 (diskdaemon.pl) score 804 or sacrifice child
Nov  5 14:46:18 b3 kernel: [  545.942530] Killed process 2376 (diskdaemon.pl) total-vm:1660240kB, anon-rss:429836kB, file-rss:44kB

The command

Code: Select all

mdadm --detail /dev/md0

gave me this (I read this in another forum post about raid)

Code: Select all

/dev/md0:
        Version : 0.90
  Creation Time : Sun Nov 28 15:04:27 2010
     Raid Level : raid1
     Array Size : 1950367680 (1860.02 GiB 1997.18 GB)
  Used Dev Size : 1950367680 (1860.02 GiB 1997.18 GB)
   Raid Devices : 2
  Total Devices : 1
Preferred Minor : 0
    Persistence : Superblock is persistent

    Update Time : Sat Nov  5 14:56:08 2011
          State : clean, degraded
 Active Devices : 1
Working Devices : 1
 Failed Devices : 0
  Spare Devices : 0

           UUID : 36875b51:59ff88cf:bd94c120:a359a404
         Events : 0.7

    Number   Major   Minor   RaidDevice State
       0       0        0        0      removed
       1       8       17        1      active sync   /dev/sdb1

Any info is welcome. As far as I know the raid has always worked without a problem so my data should be on it somewhere. Are there other options I can try to recover my data? Am I missing a step somewhere? I followed the guidelines from the B3 manual (page 99/100 ... Recover RAID array).

I'm not a server specialist but I know my way around a little bit using a terminal/ssh.

Regards,

Guy

[Ubi]

the raid on the B3 is just a simple linux software RAID, with lots of howtos available on the net. The fact that the reconstruction borked your B3 suggests it restored by using the remote drive as a template, unlikely as this seems. As this is a fresh install, I'd say re-run the installation fro USB, and do the recovery from the console rather than from the GUI.

[freeman2501]

Hi Ubi,

Thx for the reply. I've been checking howto's but I'm not really keen on trying a manual recovery as I don't want to make my backup unusable by fiddling too much with it. My sysadmin knowledge is too limited (which is the reason I needed to reinstall my Bubba in the first place).

Nevertheless, I'll keep on reading and try to learn sth from it.

Guy

[Ubi]

Are you telling us you are using half of the old RAID as a backup system?

[freeman2501]

Hi Ubi,

I'm not sure what you are referring too when you speak of "half of the old RAID".

Guy

[Ubi]

are you using the "Bubba Storage" module as a backup while you are reinstalling the B3, and then trying to put data back on the B3 via a RAID recovery using the Bubba Storage as a master?

if so; STOP!!!!

[ryz]

Ubi I think you have to define what you mean by backup. From my understanding, which might be wrong since i do not now to much of RAID, is that he uses an software raid set up as an mirror raid where the same data is written to both hard drive. I guess that it was only the /home partition that was raided. So both the B3 and Bubba storage should contain his /home folder.
He then reinstalled the B3 and is no trying to get his /home folder back by doing an raid recovery from the bubba storage so yes you could say he is using the bubba storage as an backup.

[freeman2501]

Hello Ubi,

Thanks for your reply. That's what I was trying to do. When my B3 got borked I disconnected the Bubba storage disk and reinstalled the B3. When the B3 was reinstalled I wanted to recover my home folder from the Bubba storage disk using the menu "Disk > Raid > Recover Raid Array". For "Select which external disk to recover RAID data from." I select the Bubba storage disk and hit "Recover Internal Disk".
When I tried this the first time I rebooted with the Bubba storage disk already connected. This failed with the error you can find in my first post. After browsing through the forum I found a post describing that you first have to boot the B3, then connect the Bubba storage and then do the "Recover Raid Array". Again I got the same error.

I haven't done more fiddling since last weekend. I have searched around for howto's but those I found describe the "classic" RAID setup where the complete disk is mirrored, system files inclusive. If I understand the RAID implementation on the B3 correctly the system files are not mirrored but only the home folder with it's contents.

Don't really know what I'm doing wrong and I hope I have stopped in time. Any more help how to fix this would be sweet

Guy

[Ubi]

You cannot achieve what you want because /home is mounted at the time of recovery.

First, mount the storage in normal non-RAID mode, and copy all files to the B3. Then, borrow a portable HD somewhere and also copy everything on that. IF this is all good, then you have a backup. Now follow the regular procedure of creating a RAID.

On the other hand, why don't you make your Storage into something that resembles Time Machine, by using Rsync and incremental backups. It is much more versatile than a RAID1.

[freeman2501]

Excito confirmed this week that the "Recover RAID array" functionality is currently broken and will be fix in a future software upgrade. I managed to recover my backup manually (thank you for the hints Ubi) by doing the following steps:

* Disconnect your backup disk and reboot your B3
* Once the B3 has booted reconnect your backup disk. You should see it popping up in the disk list in the B3 admin
* ssh to your B3 and login as root (it's possible that the next steps are possible as a regular user ... haven't tested it)
* create a new folder anywhere on your B3 eg mkdir /bubba_backup
* fdisk -l and look for your backup disk in the listing it generates (mine was in /dev/sdb1)
* mount the backup disk in the new folder you created: mount -t ext3 /dev/sdb1 /bubba_backup
* The files on your backup disk will now be available in /bubba_backup
* cp all files you want to restore from /bubba_backup to the appropriate folders on your B3. Don't forget to reset ownership if you copy files to specific home directories
* Once you are done you can unmount your backup disk: umount /bubba_backup
* You got all your files available again on your B3. Setup a backup solution again as your RAID array is still broken off course.

The thing I missed before when I wanted to manually mount the backup disk was the -t flag. Without it mounting will give the following error: "unknown filesystem type 'linux_raid_member'". That's why you have to provide a filesystem using the -t flag when you want to mount the disk.

Ubi, thanks for the support. I'm going to step away from the RAID setup and use another backup solution as you suggested. The current RAID setup provided by Excito just mirrors the contents of the home folder. If the complete disk is not mirrored you loose the main advantage of RAID imho eg if a disk breaks down, you boot from the other disk, replace the broken disk, ... Due to the lack of system files in the current RAID setup this is not possible and a non-raided backup solution seems much more versatile.

Over & out for this thread. I got my backup back so all is fine again

Regards,

Guy