problem mdadm Raid6 help

J

jhon614

What is this storage?
Joined
May 19, 2011
Messages
1
Location
Warsaw, Ohio
I have a large Raid6 array built using mdadm with 6 components, each 2TB, making the array size around 8TB. Recently, there was a storm, and after a restart Ubuntu's Disk Utility says the state of the array is "Not running, partially assembled".

I tried the following

Code: 
$ sudo mdadm --assemble --scan

No change. So I tried

Code: 
$ sudo mdadm --assemble --scan --verbose

and got

Code: 
mdadm: looking for devices for /dev/Raid6
mdadm: no RAID superblock on /dev/sdh1
mdadm: /dev/sdh1 has wrong uuid.
mdadm: no RAID superblock on /dev/sdh
mdadm: /dev/sdh has wrong uuid.
mdadm: cannot open device /dev/sdg1: Device or resource busy
mdadm: /dev/sdg1 has wrong uuid.
mdadm: cannot open device /dev/sdg: Device or resource busy
mdadm: /dev/sdg has wrong uuid.
mdadm: cannot open device /dev/sdf1: Device or resource busy
mdadm: /dev/sdf1 has wrong uuid.
mdadm: cannot open device /dev/sdf: Device or resource busy
mdadm: /dev/sdf has wrong uuid.
mdadm: cannot open device /dev/sde1: Device or resource busy
mdadm: /dev/sde1 has wrong uuid.
mdadm: cannot open device /dev/sde: Device or resource busy
mdadm: /dev/sde has wrong uuid.
mdadm: cannot open device /dev/sdd1: Device or resource busy
mdadm: /dev/sdd1 has wrong uuid.
mdadm: cannot open device /dev/sdd: Device or resource busy
mdadm: /dev/sdd has wrong uuid.
mdadm: cannot open device /dev/sdc1: Device or resource busy
mdadm: /dev/sdc1 has wrong uuid.
mdadm: cannot open device /dev/sdc: Device or resource busy
mdadm: /dev/sdc has wrong uuid.
mdadm: cannot open device /dev/sdb1: Device or resource busy
mdadm: /dev/sdb1 has wrong uuid.
mdadm: cannot open device /dev/sdb: Device or resource busy
mdadm: /dev/sdb has wrong uuid.
mdadm: cannot open device /dev/sda6: Device or resource busy
mdadm: /dev/sda6 has wrong uuid.
mdadm: cannot open device /dev/sda5: Device or resource busy
mdadm: /dev/sda5 has wrong uuid.
mdadm: no RAID superblock on /dev/sda2
mdadm: /dev/sda2 has wrong uuid.
mdadm: no RAID superblock on /dev/sda1
mdadm: /dev/sda1 has wrong uuid.
mdadm: cannot open device /dev/sda: Device or resource busy
mdadm: /dev/sda has wrong uuid.

/dev/sdh is actually a bare drive that was going to be the new spare, so it's fine that there's no useful data on it (and /dev/sdh1). /dev/sda is the system drive and not part of the array, so it's fine too.

My first question was why mdadm says all the devices are busy and their uuids are wrong. Any thoughts?

I checked the individual device partitions using the following

Code: 
$ sudo mdadm --misc --examine /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sde1 /dev/sdf1 /dev/sdg1

and got

Code: 
/dev/sdb1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : f4d0374a:0aaa0563:a6fed9f2:9931a37f
           Name : :Raid6
  Creation Time : Mon Jan 31 23:19:09 2011
     Raid Level : raid6
   Raid Devices : 6

 Avail Dev Size : 3906933253 (1862.97 GiB 2000.35 GB)
     Array Size : 15627732992 (7451.88 GiB 8001.40 GB)
  Used Dev Size : 3906933248 (1862.97 GiB 2000.35 GB)
    Data Offset : 272 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 4b0dd105:badb53a9:94ae0343:bd94f237

    Update Time : Sat May 14 03:40:25 2011
       Checksum : 6bfbe3cd - correct
         Events : 120501

         Layout : left-symmetric
     Chunk Size : 64K

   Device Role : Active device 2
   Array State : AAAAAA ('A' == active, '.' == missing)
/dev/sdc1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : f4d0374a:0aaa0563:a6fed9f2:9931a37f
           Name : :Raid6
  Creation Time : Mon Jan 31 23:19:09 2011
     Raid Level : raid6
   Raid Devices : 6

 Avail Dev Size : 3906933248 (1862.97 GiB 2000.35 GB)
     Array Size : 15627732992 (7451.88 GiB 8001.40 GB)
    Data Offset : 243 sectors
   Super Offset : 8 sectors
          State : active
    Device UUID : dfdfb2a4:70e12706:e8474464:9467c519

    Update Time : Sat May 14 03:40:25 2011
       Checksum : 6ac524f2 - correct
         Events : 120501

         Layout : left-symmetric
     Chunk Size : 64K

   Device Role : Active device 5
   Array State : AAAAAA ('A' == active, '.' == missing)
/dev/sdd1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : f4d0374a:0aaa0563:a6fed9f2:9931a37f
           Name : :Raid6
  Creation Time : Mon Jan 31 23:19:09 2011
     Raid Level : raid6
   Raid Devices : 6

 Avail Dev Size : 3906933287 (1862.97 GiB 2000.35 GB)
     Array Size : 15627732992 (7451.88 GiB 8001.40 GB)
  Used Dev Size : 3906933248 (1862.97 GiB 2000.35 GB)
    Data Offset : 272 sectors
   Super Offset : 8 sectors
          State : active
    Device UUID : 66b10d93:354a565e:388d4ab8:09629b1b

    Update Time : Sat May 14 03:40:25 2011
       Checksum : 72a9f43 - correct
         Events : 120501

         Layout : left-symmetric
     Chunk Size : 64K

   Device Role : Active device 1
   Array State : AAAAAA ('A' == active, '.' == missing)
/dev/sde1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : f4d0374a:0aaa0563:a6fed9f2:9931a37f
           Name : :Raid6
  Creation Time : Mon Jan 31 23:19:09 2011
     Raid Level : raid6
   Raid Devices : 6

 Avail Dev Size : 3906933253 (1862.97 GiB 2000.35 GB)
     Array Size : 15627732992 (7451.88 GiB 8001.40 GB)
  Used Dev Size : 3906933248 (1862.97 GiB 2000.35 GB)
    Data Offset : 272 sectors
   Super Offset : 8 sectors
          State : active
    Device UUID : 34258aad:8ac3939f:438b66af:73c98d3f

    Update Time : Sat May 14 03:40:25 2011
       Checksum : 7df2f1bd - correct
         Events : 120501

         Layout : left-symmetric
     Chunk Size : 64K

   Device Role : Active device 4
   Array State : AAAAAA ('A' == active, '.' == missing)
/dev/sdf1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : f4d0374a:0aaa0563:a6fed9f2:9931a37f
           Name : :Raid6
  Creation Time : Mon Jan 31 23:19:09 2011
     Raid Level : raid6
   Raid Devices : 6

 Avail Dev Size : 3906933287 (1862.97 GiB 2000.35 GB)
     Array Size : 15627732992 (7451.88 GiB 8001.40 GB)
  Used Dev Size : 3906933248 (1862.97 GiB 2000.35 GB)
    Data Offset : 272 sectors
   Super Offset : 8 sectors
          State : active
    Device UUID : 522a13c0:e41315ca:8070933c:4fa37f46

    Update Time : Sat May 14 03:40:25 2011
       Checksum : 4f1c066d - correct
         Events : 120501

         Layout : left-symmetric
     Chunk Size : 64K

   Device Role : Active device 0
   Array State : AAAAAA ('A' == active, '.' == missing)
/dev/sdg1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : f4d0374a:0aaa0563:a6fed9f2:9931a37f
           Name : :Raid6
  Creation Time : Mon Jan 31 23:19:09 2011
     Raid Level : raid6
   Raid Devices : 6

 Avail Dev Size : 3906933287 (1862.97 GiB 2000.35 GB)
     Array Size : 15627732992 (7451.88 GiB 8001.40 GB)
  Used Dev Size : 3906933248 (1862.97 GiB 2000.35 GB)
    Data Offset : 272 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : f74ec4e7:e01c916d:3003d7fa:f15761a5

    Update Time : Wed May 11 22:45:23 2011
       Checksum : 376b895d - correct
         Events : 117920

         Layout : left-symmetric
     Chunk Size : 64K

   Device Role : Active device 0
   Array State : AAAAAA ('A' == active, '.' == missing)

Everything here looks good except some of the "State" lines appear different and some of the "Update Time" lines appear different.

Does anyone know what I should do to get this thing back online? Any help would be greatly appreciated.
 
Mercutio

Mercutio

Fatwah on Western Digital
Joined
Jan 17, 2002
Messages
21,601
Location
I am omnipresent
Honestly, if my arrays were doing something like that, what I would do is recreate the arrays and restore from backup. Most of the time the *nix way would be to find the problem and fix it and if you can stand to have 6TB of your stuff offline there's probably a fun project in figuring out what the hell happened, but for the most part when I have a giant lump of data like that I'm more worried about getting it back on line again.
 
LiamC

LiamC

Storage Is My Life
Joined
Feb 7, 2002
Messages
2,016
Location
Canberra
Search this and the Tech Support forum with keyword mdadm. There's a number of threads that may be of use. Gilbo's thread?
 
S

sor

What is this storage?
Joined
May 23, 2011
Messages
7
Weird. This sort of looks like the array is already started (or thinks it is), that may be why they look busy.

What does /proc/mdstat contain? Have you tried manually starting the array?

I've had some luck recreating arrays exactly as they were, and the data all still being there. I've never done that on an md raid6. I'd test the procedure first with some scratch disks, and save it for a last resort before recovering from backup, but that might get your array back.
 
S

sor

What is this storage?
Joined
May 23, 2011
Messages
7
Oh. From your 'assemble scan' output it looks like you've got /dev/Raid6 perhaps defined in /etc/mdadm.conf (or similar), and there is a UUID in that file not matching what's defined on the arrays (maybe the file is corrupt or something?). Perhaps if you edit that file to match "f4d0374a:0aaa0563:a6fed9f2:9931a37f" for UUID, or just move that file out of the way for the moment and recreate with "mdadm --examine --scan > /etc/mdadm.conf". At any rate I'm wondering if the array is started under a name you're not looking for or something.
 
blakerwry

blakerwry

Storage? I am Storage!
Joined
Oct 12, 2002
Messages
4,203
Location
Kansas City, USA
Website
justblake.com
sor said:
At any rate I'm wondering if the array is started under a name you're not looking for or something.
Click to expand...

Agreed, is the file system on the array mounted? How do you normally access it?

Does dmesg show any error on boot relating to /dev/Raid6, mdadm, or any of the drives?
 
You must log in or register to reply here.
Top