一.问题描述
某一台跑批服务器硬盘无法正常读写，提示input/output error，服务器每天均需使用，询问情况后发现服务器首先为硬盘故障，更换硬盘后提示此错误（RAID已正常同步）
二.排查问题
出现问题，先查看日志，收集日志进行分析查看，日志分析结果如下：
[12922471.544897] smartpqi 0000:5e:00.0: reset of scsi 14:1:0:3: SUCCESS
[12922471.545034] sd 14:1:0:3: [sdd] Medium access timeout failure. Offlining disk!
…
[12922471.546144] blk_update_request: I/O error, dev sdd, sector 2351217920
[12922471.546473] sd 14:1:0:3: rejecting I/O to offline device
[12922471.547836] XFS (sdd1): metadata I/O error: block 0x8bbac400 (“xlog_iodone”) error 5 numblks 512
[12922471.547840] XFS (sdd1): xfs_do_force_shutdown(0x2) called from line 1200 of file fs/xfs/xfs_log.c. Return address = 0xffffffffc07a1ea0
[12922471.547866] XFS (sdd1): Log I/O Error Detected. Shutting down filesystem
[12922471.547868] XFS (sdd1): Please umount the filesystem and rectify the problem(s)
[12922471.547870] XFS (sdd1): metadata I/O error: block 0x8bbac600 (“xlog_iodone”) error 5 numblks 512
[12922471.547872] XFS (sdd1): xfs_do_force_shutdown(0x2) called from line 1200 of file fs/xfs/xfs_log.c. Return address = 0xffffffffc07a1ea0
[12922471.547891] XFS (sdd1): metadata I/O error: block 0x2bc1a6c0 (“xfs_trans_read_buf_map”) error 5 numblks 32
[12922471.547898] XFS (sdd1): xfs_imap_to_bp: xfs_trans_read_buf() returned error -5.
[12922471.548349] XFS (sdd1): metadata I/O error: block 0xc65b63f8 (“xfs_trans_read_buf_map”) error 5 numblks 8
[12922471.548390] XFS (sdd1): metadata I/O error: block 0x8bdb5820 (“xfs_trans_read_buf_map”) error 5 numblks 32
[12922471.548408] XFS (sdd1): xfs_imap_to_bp: xfs_trans_read_buf() returned error -5.
[12922471.548412] XFS (sdd1): metadata I/O error: block 0x11771540 (“xfs_trans_read_buf_map”) error 5 numblks 32
[12922471.548417] XFS (sdd1): xfs_imap_to_bp: xfs_trans_read_buf() returned error -5.
…
[15351852.339037] sd 14:1:0:3: rejecting I/O to offline device
- 查看日志发现磁盘已经offline，并且文件系统已经异常.
三.解决方案
- 1.手动将此硬盘设置为online
# echo running /sys/block/sdd/device/state - 2.查询是否为running
cat /sys/block/sdd/device/state - 3.修复文件系统，并确认硬盘处于umount状态（视情况而定，如无法umount则只能进行重启，我是进行的重启操作）
- 4.开始修复
XFS : Corruption detected. Unmount and run xfs_repair
官方文档如下:https://access.redhat.com/solutions/1194613 - 5.按照上述方法修复完成后，再进行mount操作
原创文章，作者：AIRF，如若转载，请注明出处：http://www.ai-rf.com/index.php/2021/11/21/linux-ls-%e5%91%bd%e4%bb%a4%e6%8f%90%e7%a4%ba-input-output-error/
