One of the volumes on bbsto01 failed to mount with a bad superblock today when the machine was booted. I ran xfs_check and xfs_repair, the latter which resulted in this output: ...................................................................................................................................found candidate secondary superblock... verified secondary superblock... writing modified primary superblock xfs_repair: size check failed xfs_repair: cannot repair this filesystem. Sorry. Any ideas? -- Ken Raffenetti Systems Administration Associate MCS Division - Argonne National Laboratory
It looks like this might be part of it: [ 665.305421] 3w-9xxx: scsi0: AEN: WARNING (0x04:0x0036): Verify fixed data/parity mismatch:unit=0. There are a few interesting notes here: http://xfs.9218.n7.nabble.com/xfs-fixing-wrong-xfs-size-td6364.html -nld On Nov 27, 2012, at 6:29 PM, Ken Raffenetti wrote:
One of the volumes on bbsto01 failed to mount with a bad superblock today when the machine was booted. I ran xfs_check and xfs_repair, the latter which resulted in this output:
...................................................................................................................................found candidate secondary superblock... verified secondary superblock... writing modified primary superblock xfs_repair: size check failed xfs_repair: cannot repair this filesystem. Sorry.
Any ideas?
-- Ken Raffenetti Systems Administration Associate MCS Division - Argonne National Laboratory
I believe this means there is a discrepancy between the block device size and the size stored in the sb. xfs_db can read the sb. Do we know the last time it was grown? We could update the sb to match the block device and then it may mount, but there will be data loss. --- Daniel Murphy-Olson Systems Administrator Mathematics & Computer Science Division Argonne National Laboratory 630-252-0055
the filesystem has never been grown; it has been the same size since day one. (and we haven't even had a drive replacement in the last year) -nld On Nov 27, 2012, at 6:48 PM, Dan Olson wrote:
I believe this means there is a discrepancy between the block device size and the size stored in the sb. xfs_db can read the sb. Do we know the last time it was grown? We could update the sb to match the block device and then it may mount, but there will be data loss.
--- Daniel Murphy-Olson Systems Administrator Mathematics & Computer Science Division Argonne National Laboratory 630-252-0055
I'd say there is more going on, maybe with the root volume as well. The machine is down again. We do have good backups, there used to be a couple spare sto boxes in breadboard. Anyone know if they are still available, or did they get incorporated into mg-rast? ---- Daniel Murphy-Olson Systems Administrator Mathematics & Computer Science Division Argonne National Laboratory 630-252-0055 ----- Original Message -----
From: "Narayan Desai" <[email protected]> To: "Dan Olson" <[email protected]> Cc: "Ken Raffenetti" <[email protected]>, "core-admins >> \"[email protected]\"" <[email protected]> Sent: Tuesday, November 27, 2012 6:59:20 PM Subject: Re: xfs help? the filesystem has never been grown; it has been the same size since day one. (and we haven't even had a drive replacement in the last year) -nld
On Nov 27, 2012, at 6:48 PM, Dan Olson wrote:
I believe this means there is a discrepancy between the block device size and the size stored in the sb. xfs_db can read the sb. Do we know the last time it was grown? We could update the sb to match the block device and then it may mount, but there will be data loss.
--- Daniel Murphy-Olson Systems Administrator Mathematics & Computer Science Division Argonne National Laboratory 630-252-0055
That was me. The XFS repair commands eat memory like candy. Provided that the device scrub doesn't reveal anything bad in the raid device, Ken is going to make a new XFS filesystem on the block device and restore from backups. There are only 4 users that have logged into the system in the last 2 months that have home directories on that volume, so we should be able to get things into good shape pretty quickly. (there are so few users on fd that it makes this a lot easier) We've got a pile of spare gear in the bb fileserver rack; Ken has the skinny. -nld On Nov 27, 2012, at 9:47 PM, Dan Olson wrote:
I'd say there is more going on, maybe with the root volume as well. The machine is down again.
We do have good backups, there used to be a couple spare sto boxes in breadboard. Anyone know if they are still available, or did they get incorporated into mg-rast?
---- Daniel Murphy-Olson Systems Administrator Mathematics & Computer Science Division Argonne National Laboratory 630-252-0055
----- Original Message -----
From: "Narayan Desai" <[email protected]> To: "Dan Olson" <[email protected]> Cc: "Ken Raffenetti" <[email protected]>, "core-admins >> \"[email protected]\"" <[email protected]> Sent: Tuesday, November 27, 2012 6:59:20 PM Subject: Re: xfs help? the filesystem has never been grown; it has been the same size since day one. (and we haven't even had a drive replacement in the last year) -nld
On Nov 27, 2012, at 6:48 PM, Dan Olson wrote:
I believe this means there is a discrepancy between the block device size and the size stored in the sb. xfs_db can read the sb. Do we know the last time it was grown? We could update the sb to match the block device and then it may mount, but there will be data loss.
--- Daniel Murphy-Olson Systems Administrator Mathematics & Computer Science Division Argonne National Laboratory 630-252-0055
I'm going to start the restore this morning unless anyone objects. Is TSM in good shape to do this? It's reporting a good backup from yesterday morning. 1 11/27/2012 08:15:06 XFS /media/data0/nfs-homes02 Ken Raffenetti Systems Administration Associate MCS Division - Argonne National Laboratory On 11/27/2012 09:57 PM, Narayan Desai wrote:
That was me. The XFS repair commands eat memory like candy.
Provided that the device scrub doesn't reveal anything bad in the raid device, Ken is going to make a new XFS filesystem on the block device and restore from backups. There are only 4 users that have logged into the system in the last 2 months that have home directories on that volume, so we should be able to get things into good shape pretty quickly. (there are so few users on fd that it makes this a lot easier)
We've got a pile of spare gear in the bb fileserver rack; Ken has the skinny. -nld
On Nov 27, 2012, at 9:47 PM, Dan Olson wrote:
I'd say there is more going on, maybe with the root volume as well. The machine is down again.
We do have good backups, there used to be a couple spare sto boxes in breadboard. Anyone know if they are still available, or did they get incorporated into mg-rast?
---- Daniel Murphy-Olson Systems Administrator Mathematics & Computer Science Division Argonne National Laboratory 630-252-0055
----- Original Message -----
From: "Narayan Desai" <[email protected]> To: "Dan Olson" <[email protected]> Cc: "Ken Raffenetti" <[email protected]>, "core-admins >> \"[email protected]\"" <[email protected]> Sent: Tuesday, November 27, 2012 6:59:20 PM Subject: Re: xfs help? the filesystem has never been grown; it has been the same size since day one. (and we haven't even had a drive replacement in the last year) -nld
On Nov 27, 2012, at 6:48 PM, Dan Olson wrote:
I believe this means there is a discrepancy between the block device size and the size stored in the sb. xfs_db can read the sb. Do we know the last time it was grown? We could update the sb to match the block device and then it may mount, but there will be data loss.
--- Daniel Murphy-Olson Systems Administrator Mathematics & Computer Science Division Argonne National Laboratory 630-252-0055
participants (3)
-
Dan Olson -
Ken Raffenetti -
Narayan Desai