RAID Array Recovery - arkk NAS¶

This runbook combines the original recovery guide with the additional degraded-start steps discovered during recovery when the array appeared as inactive and members were marked as spares (S).

Array: /dev/md0 (RAID5, 5 disks, ~15TB usable)

Current Situation Template¶

Use this section to record real-time state during incidents.

Array status from cat /proc/mdstat
Failed disk device and serial number
Replacement disk device and serial number
Any observed I/O errors from dmesg

Pre-Recovery Checklist¶

[ ] Backup critical data if possible
[ ] Cold spare drive ready (same size or larger)
[ ] Record serial number of the replacement drive
[ ] Confirm maintenance window and power-off plan
[ ] Keep commands in a shell history log during recovery

Device Mapping Checklist (Fill Before Destructive Steps)¶

Do this once per recovery session and reuse the same values through Steps 5-9.

Capture disk inventory by serial:

lsblk -o NAME,SERIAL,SIZE,MODEL,TYPE
sudo mdadm --detail /dev/md0

Fill this table:

Purpose	Example	Verified by serial?
New replacement disk (whole disk)	/dev/sdf	[ ]
New replacement partition	/dev/sdf1	[ ]
Known-good member for partition clone	/dev/sdb	[ ]
Failed/removed slot number	3	[ ]

Export variables to reduce typo risk:

export NEW_DISK=/dev/sdX
export NEW_PART=${NEW_DISK}1
export GOOD_DISK=/dev/sdY

Sanity check before running write operations:

echo "NEW_DISK=$NEW_DISK NEW_PART=$NEW_PART GOOD_DISK=$GOOD_DISK"
lsblk -o NAME,SERIAL,SIZE,MODEL "$NEW_DISK" "$GOOD_DISK"

Only proceed when serial numbers match your worksheet.

Recovery Steps¶

Step 1: Verify Current State Before Shutdown¶

ssh arkk
cat /proc/mdstat
sudo mdadm --detail /dev/md0
lsblk -o NAME,SERIAL,SIZE,MODEL
sudo smartctl -i "$NEW_DISK" 2>/dev/null | grep "Serial Number" || true

Confirm data is currently accessible if possible:

ls -la /mnt/arkk

Step 2: Unmount and Stop Array¶

sudo umount /mnt/arkk || true
sudo mdadm --stop /dev/md0 || true
cat /proc/mdstat

Step 3: Power Down¶

sudo shutdown -h now

Step 4: Physical Disk Swap¶

Disconnect power.
Locate failed disk by serial number, not SATA port.
Remove failed disk and install replacement.
Reconnect and boot.

Step 5: Verify New Disk Detection¶

ssh arkk
lsblk -o NAME,SERIAL,SIZE,MODEL
sudo smartctl -H "$NEW_DISK"

If the replacement drive was previously used, old RAID metadata can exist. That is handled below.

Step 6: Start Array (Degraded) - Updated Procedure¶

If /proc/mdstat shows md0 : inactive ... (S), metadata is present but the array is not running yet.

Step 6a: Stop inactive assembly first¶

sudo mdadm --stop /dev/md0

Step 6b: Assemble degraded, read-only, and force-run¶

Use all candidate RAID member partitions for this host, excluding devices with no md superblock.

sudo mdadm --assemble --readonly --force --run /dev/md0 /dev/sd[a-f]1

Notes: - --force may be required when one member has stale Events. - If a replacement disk is currently marked spare, degraded read-only start can still succeed for recovery.

Step 6c: Validate array is truly active¶

cat /proc/mdstat
sudo mdadm --detail /dev/md0

Expected degraded recovery shape: - md0 : active (read-only) raid5 - [5/4] [UUU_U] or equivalent

Step 6d: Mount read-only for data extraction¶

sudo mkdir -p /mnt/arkk
sudo mount -o ro /dev/md0 /mnt/arkk

If mount fails, verify if filesystem is on a partitioned md node:

lsblk -f
sudo blkid /dev/md0 /dev/md0p1

Then mount /dev/md0p1 if required.

Step 6e: If assembly still fails¶

Check metadata consistency and event counters:

sudo mdadm --examine /dev/sd[a-z]1 | egrep -i "Array UUID|Raid Level|Raid Devices|Device Role|Events|State"

Check kernel logs for read/I/O failures:

dmesg -T | egrep -i "md0|I/O error|raid|sd[a-f]" | tail -n 100

Common causes: - Omitted required member in assemble command - Stale member rejected without --force - True disk I/O error on one of the remaining members

Step 7: Clean Replacement Drive Metadata (if needed)¶

cat /proc/mdstat
lsblk "$NEW_DISK"
sudo mdadm --examine "$NEW_PART" || true
sudo mdadm --zero-superblock "$NEW_PART" || true
sudo wipefs -a "$NEW_DISK"

Step 8: Partition Replacement Disk¶

Copy partition layout from a known-good member disk.

sudo sfdisk -d "$GOOD_DISK" | sudo sfdisk "$NEW_DISK"
lsblk "$NEW_DISK"

Step 9: Add New Disk to Array¶

sudo mdadm --manage /dev/md0 --add "$NEW_PART"
cat /proc/mdstat

Step 10: Monitor Rebuild¶

watch -n 5 cat /proc/mdstat
sudo mdadm --detail /dev/md0

Step 11: Verify Completion¶

cat /proc/mdstat
sudo mdadm --detail /dev/md0

Target state: - [5/5] [UUUUU] - State : clean

Step 12: Persist Assembly at Boot¶

sudo cp /etc/mdadm/mdadm.conf /etc/mdadm/mdadm.conf.backup

# Check if md0 is already defined to avoid duplicate ARRAY lines.
grep -E "^ARRAY /dev/md0" /etc/mdadm/mdadm.conf

# If md0 is not present, append scan output.
sudo mdadm --detail --scan | sudo tee -a /etc/mdadm/mdadm.conf

# If md0 is already present, replace that ARRAY line manually with fresh scan output.
# sudo mdadm --detail --scan
# sudo editor /etc/mdadm/mdadm.conf

sudo update-initramfs -u

Data Recovery Workflow (Recommended Before Rebuild)¶

When the array is degraded, prioritize extraction first.

Keep the array read-only for the entire backup (assemble/mount ro).
Mount the array read-only on arkk and export it to the backup host.
Pull data to the backup host. Two transports are supported:
NFS over a direct link (preferred here): mount the array on the backup host over a dedicated/bonded Ethernet link and rsync from the local mountpoint (no ssh overhead). This is how the 2026 incident backup runs.
rsync over SSH: when no direct mount is available.
Copy selectively — skip regenerable/redownloadable data and thin periodic training checkpoints, so only irreplaceable data consumes rescue capacity.
Re-run the copy to catch partials (rsync skips already-complete files).
Verify the copy, then rebuild parity (never rebuild before the backup is verified — a resilver runs with zero redundancy).

Selective-copy helper scripts in this repo (docs/machines/arkk/scripts/): - run_backup.sh — runs both passes in order with logging and preflight checks - rsync_selected_from_arkk.sh + targets.txt / excludes.txt — bulk selective copy - thin_checkpoints_from_arkk.sh + checkpoint_dirs.txt — thinned checkpoint ladder

For the incident-specific keep/drop plan and run procedure, see BACKUP_TRIAGE_2026.md.

Troubleshooting Quick Reference¶

New disk not detected¶

echo "- - -" | sudo tee /sys/class/scsi_host/host*/scan
dmesg -T | tail -n 100

mdadm add fails¶

sudo mdadm --examine "$NEW_PART"
sudo mdadm --zero-superblock "$NEW_PART"
sudo mdadm --manage /dev/md0 --add "$NEW_PART"

Array will not start¶

sudo mdadm --assemble --readonly --force --run /dev/md0 /dev/sd[a-f]1
sudo mdadm --examine /dev/sd[a-f]1 | grep -i Events

Reference¶

mdadm manual: man mdadm
SMART manual: man smartctl