固态移动硬盘频繁掉线问题排查
错误
[ 1570.328430] blk_update_request: I/O error, dev sdd, sector 0 op 0x1:(WRITE) flags 0x800 phys_seg 0 prio class 0
[ 1570.342148] sd 6:0:0:0: [sdd] Synchronizing SCSI cache
[ 1570.591625] sd 6:0:0:0: [sdd] Synchronize Cache(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK
[ 1603.208458] EXT4-fs error (device sdd1): ext4_get_inode_loc:4461: inode #3932392: block 15728686: comm gogs: unable to read itable block
[ 1603.208469] EXT4-fs error (device sdd1) in ext4_reserve_inode_write:5751: IO failure
[ 1603.208474] EXT4-fs error (device sdd1): ext4_dirty_inode:5953: inode #3932392: comm gogs: mark_inode_dirty error
[ 1603.208478] ------------[ cut here ]------------
[ 1603.208480] bdi-(unknown) not registered
[ 1603.208499] WARNING: CPU: 0 PID: 5844 at fs/fs-writeback.c:2307 __mark_inode_dirty+0x247/0x2f0
[ 1603.208500] Modules linked in: uas usb_storage nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set nf_tables nfnetlink vboxnetadp(OE) vboxnetflt(OE) vboxdrv(OE) bnep binfmt_misc intel_rapl_msr intel_rapl_common snd_hda_codec_realtek snd_hda_codec_generic snd_hda_codec_hdmi btusb x86_pkg_temp_thermal intel_powerclamp xfs btrtl iwlmvm snd_hda_intel btbcm coretemp uvcvideo btintel snd_intel_dspcfg mei_hdcp videobuf2_vmalloc mei_wdt bluetooth soundwire_intel videobuf2_memops kvm_intel videobuf2_v4l2 mac80211 soundwire_generic_allocation snd_soc_core jitterentropy_rng videobuf2_common libarc4 kvm snd_compress drbg videodev soundwire_cadence irqbypass ansi_cprng snd_hda_codec cpufreq_ondemand joydev ecdh_generic mc ecc iwlwifi snd_hda_core rapl snd_hwdep intel_cstate soundwire_bus cfg80211 thinkpad_acpi snd_pcm nvram snd_timer iTCO_wdt rmi_smbus mei_me ledtrig_audio
[ 1603.208598] intel_pmc_bxt intel_uncore snd tpm_tis iTCO_vendor_support rmi_core serio_raw pcspkr at24 soundcore tpm_tis_core mei watchdog sg rfkill tpm wmi_bmof evdev rng_core ac parport_pc ppdev lp parport fuse configfs ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c crc32c_generic raid1 raid0 multipath linear md_mod dm_mirror dm_region_hash dm_log dm_mod sd_mod t10_pi crc_t10dif crct10dif_generic hid_logitech_hidpp hid_logitech_dj hid_generic usbhid hid i915 crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel ghash_clmulni_intel rtsx_pci_sdmmc mmc_core i2c_algo_bit ahci drm_kms_helper libahci aesni_intel xhci_pci libata cec xhci_hcd libaes ehci_pci crypto_simd ehci_hcd drm psmouse cryptd scsi_mod e1000e glue_helper usbcore i2c_i801 i2c_smbus rtsx_pci lpc_ich ptp pps_core usb_common wmi battery video button
[ 1603.208717] CPU: 0 PID: 5844 Comm: gogs Tainted: G OE 5.10.0-9-amd64 #1 Debian 5.10.70-1
[ 1603.208720] Hardware name: LENOVO 20AWS1XF01/20AWS1XF01, BIOS GLET85WW (2.39 ) 09/29/2016
[ 1603.208725] RIP: 0010:__mark_inode_dirty+0x247/0x2f0
[ 1603.208729] Code: ff ff 48 8b 38 48 89 c5 f6 47 44 01 74 1e 48 8b 40 08 a8 01 75 16 e8 f8 81 f3 ff 48 c7 c7 1b 99 2f 9d 48 89 c6 e8 2c 59 58 00 <0f> 0b 48 8b 05 f0 f1 30 01 49 89 85 c8 00 00 00 45 85 ff 74 0e 48
[ 1603.208731] RSP: 0018:ffffae4a02827d90 EFLAGS: 00010286
[ 1603.208735] RAX: 0000000000000000 RBX: 0000000000000001 RCX: ffff8b6c4e618a08
[ 1603.208737] RDX: 00000000ffffffd8 RSI: 0000000000000027 RDI: ffff8b6c4e618a00
[ 1603.208739] RBP: ffff8b69baac0400 R08: 0000000000000000 R09: ffffae4a02827bb0
[ 1603.208740] R10: ffffae4a02827ba8 R11: ffffffff9d8cb428 R12: ffff8b695e7f8668
[ 1603.208742] R13: ffff8b695e7f85e0 R14: 0000000000000000 R15: 0000000000000000
[ 1603.208746] FS: 00007f4ce65e2700(0000) GS:ffff8b6c4e600000(0000) knlGS:0000000000000000
[ 1603.208748] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1603.208750] CR2: 00007ff1ec5129c8 CR3: 000000017a9fe006 CR4: 00000000001706f0
[ 1603.208752] Call Trace:
[ 1603.208762] generic_update_time+0x68/0xc0
[ 1603.208766] file_update_time+0x123/0x140
[ 1603.208771] ? generic_write_checks+0x61/0xc0
[ 1603.208812] ext4_buffered_write_iter+0x5a/0x160 [ext4]
[ 1603.208820] new_sync_write+0x11c/0x1b0
[ 1603.208824] vfs_write+0x1c2/0x260
[ 1603.208827] ksys_write+0x5f/0xe0
[ 1603.208834] do_syscall_64+0x33/0x80
[ 1603.208837] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 1603.208841] RIP: 0033:0x495ffb
[ 1603.208845] Code: fe ff eb bd e8 26 49 fe ff e9 61 ff ff ff cc e8 9b 16 fe ff 48 8b 7c 24 10 48 8b 74 24 18 48 8b 54 24 20 48 8b 44 24 08 0f 05 <48> 3d 01 f0 ff ff 76 20 48 c7 44 24 28 ff ff ff ff 48 c7 44 24 30
[ 1603.208847] RSP: 002b:000000c00006db10 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 1603.208850] RAX: ffffffffffffffda RBX: 000000c000044000 RCX: 0000000000495ffb
[ 1603.208852] RDX: 000000000000008d RSI: 000000c000474f00 RDI: 0000000000000003
[ 1603.208854] RBP: 000000c00006db60 R08: 000000c000474f01 R09: 0000000000000280
[ 1603.208856] R10: 00007ffc8bb15080 R11: 0000000000000202 R12: ffffffffffffffff
[ 1603.208857] R13: 0000000000000002 R14: 0000000000000001 R15: 0000000000000040
[ 1603.208862] ---[ end trace 04478d6216aad623 ]---
[ 1603.208897] EXT4-fs error (device sdd1): ext4_get_inode_loc:4461: inode #3932392: block 15728686: comm gogs: unable to read itable block
[ 1603.208901] EXT4-fs error (device sdd1) in ext4_reserve_inode_write:5751: IO failure
[ 1603.208905] EXT4-fs error (device sdd1): ext4_da_write_end:3101: inode #3932392: comm gogs: mark_inode_dirty error
[ 1603.208933] EXT4-fs error (device sdd1): ext4_get_inode_loc:4461: inode #3932392: block 15728686: comm gogs: unable to read itable block
[ 1603.208936] EXT4-fs error (device sdd1) in ext4_reserve_inode_write:5751: IO failure
[ 1603.208939] EXT4-fs error (device sdd1): ext4_dirty_inode:5953: inode #3932392: comm gogs: mark_inode_dirty error
[ 1603.209809] EXT4-fs error (device sdd1): ext4_get_inode_loc:4461: inode #3932392: block 15728686: comm gogs: unable to read itable block
[ 1603.212148] EXT4-fs warning (device sdd1): htree_dirblock_to_tree:1003: inode #2: lblock 0: comm git: error -5 reading directory block
[ 1603.215180] EXT4-fs warning (device sdd1): htree_dirblock_to_tree:1003: inode #2: lblock 0: comm git: error -5 reading directory block
[ 1603.217274] EXT4-fs warning (device sdd1): htree_dirblock_to_tree:1003: inode #2: lblock 0: comm git: error -5 reading directory block
[ 1603.219118] EXT4-fs warning (device sdd1): htree_dirblock_to_tree:1003: inode #2: lblock 0: comm git: error -5 reading directory block
[ 1603.220876] EXT4-fs warning (device sdd1): htree_dirblock_to_tree:1003: inode #2: lblock 0: comm git: error -5 reading directory block
[ 1603.222464] EXT4-fs warning (device sdd1): htree_dirblock_to_tree:1003: inode #2: lblock 0: comm git: error -5 reading directory block
[ 1603.224087] EXT4-fs warning (device sdd1): htree_dirblock_to_tree:1003: inode #2: lblock 0: comm git: error -5 reading directory block
[ 1603.225799] EXT4-fs warning (device sdd1): htree_dirblock_to_tree:1003: inode #2: lblock 0: comm git: error -5 reading directory block
[ 1603.227401] EXT4-fs warning (device sdd1): htree_dirblock_to_tree:1003: inode #2: lblock 0: comm git: error -5 reading directory block
[ 1603.229007] EXT4-fs warning (device sdd1): htree_dirblock_to_tree:1003: inode #2: lblock 0: comm git: error -5 reading directory block
[ 1608.992108] Aborting journal on device sdd1-8.
[ 1608.992124] Buffer I/O error on dev sdd1, logical block 28868608, lost sync page write
[ 1608.992130] JBD2: Error -5 detected when updating journal superblock for sdd1-8.
[ 1611.108274] EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: (null)
[ 1749.930856] EXT4-fs error: 569 callbacks suppressed
[ 1749.930858] EXT4-fs error (device sdd1): __ext4_find_entry:1534: inode #3932197: comm gogs: reading directory lblock 0
[ 1749.930873] EXT4-fs error (device sdd1): __ext4_find_entry:1534: inode #3932197: comm gogs: reading directory lblock 0
[ 1749.930877] EXT4-fs error (device sdd1): __ext4_find_entry:1534: inode #3932197: comm gogs: reading directory lblock 0
[ 1749.930882] EXT4-fs error (device sdd1): __ext4_find_entry:1534: inode #3932197: comm gogs: reading directory lblock 0
[ 1749.930889] EXT4-fs error (device sdd1): __ext4_find_entry:1534: inode #3932197: comm gogs: reading directory lblock 0
[ 1749.930892] EXT4-fs error (device sdd1): __ext4_find_entry:1534: inode #3932197: comm gogs: reading directory lblock 0
[ 1749.930919] EXT4-fs error (device sdd1): __ext4_find_entry:1534: inode #3932161: comm gogs: reading directory lblock 0
[ 1749.930930] EXT4-fs error (device sdd1): __ext4_find_entry:1534: inode #3932161: comm gogs: reading directory lblock 0
[ 1749.930937] EXT4-fs error (device sdd1): __ext4_find_entry:1534: inode #3932161: comm gogs: reading directory lblock 0
[ 1749.930945] EXT4-fs error (device sdd1): __ext4_find_entry:1534: inode #3932161: comm gogs: reading directory lblock 0
[ 1846.614756] usb 4-2: new SuperSpeed Gen 1 USB device number 3 using xhci_hcd
[ 1846.635853] usb 4-2: New USB device found, idVendor=152d, idProduct=0583, bcdDevice= 2.09
[ 1846.635860] usb 4-2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[ 1846.635864] usb 4-2: Product: External HDD
[ 1846.635867] usb 4-2: Manufacturer: External USB3.0
[ 1846.635870] usb 4-2: SerialNumber: DD56419883891
[ 1846.641116] scsi host6: uas
[ 1846.641647] scsi 6:0:0:0: Direct-Access External USB3.0 0209 PQ: 0 ANSI: 6
[ 1846.642510] sd 6:0:0:0: Attached scsi generic sg3 type 0
[ 1848.616280] sd 6:0:0:0: [sdd] 468862128 512-byte logical blocks: (240 GB/224 GiB)
[ 1848.616283] sd 6:0:0:0: [sdd] 4096-byte physical blocks
[ 1848.616402] sd 6:0:0:0: [sdd] Write Protect is off
[ 1848.616404] sd 6:0:0:0: [sdd] Mode Sense: 5f 00 00 08
[ 1848.616711] sd 6:0:0:0: [sdd] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[ 1848.616939] sd 6:0:0:0: [sdd] Optimal transfer size 33553920 bytes not a multiple of physical block size (4096 bytes)
[ 1848.636911] sdd: sdd1
[ 1848.668497] sd 6:0:0:0: [sdd] Attached SCSI disk
[ 1848.832889] EXT4-fs (sdd1): recovery complete
[ 1848.833743] EXT4-fs (sdd1): mounted filesystem with ordered data mode. Opts: (null)
[ 1958.091238] EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: (null)
两种可能
- 固态硬盘坏了
- ext4分区有损坏。
尝试修复ext4分区
$ sudo dumpe2fs -h /dev/sdd1
dumpe2fs 1.46.2 (28-Feb-2021)
Filesystem volume name: dddd
Last mounted on: /media/wii/dddd
Filesystem UUID: b75bb3db-d2a7-4245-b581-35cbeaf4b7c2
Filesystem magic number: 0xEF53
Filesystem revision #: 1 (dynamic)
Filesystem features: has_journal ext_attr resize_inode dir_index filetype needs_recovery extent 64bit flex_bg sparse_super large_file huge_file dir_nlink extra_isize metadata_csum
Filesystem flags: unsigned_directory_hash
Default mount options: user_xattr acl
Filesystem state: clean
Errors behavior: Continue
Filesystem OS type: Linux
Inode count: 14655488
Block count: 58607510
Reserved block count: 2930375
Free blocks: 36428795
Free inodes: 14391455
First block: 0
Block size: 4096
Fragment size: 4096
Group descriptor size: 64
Reserved GDT blocks: 1024
Blocks per group: 32768
Fragments per group: 32768
Inodes per group: 8192
Inode blocks per group: 512
Flex block group size: 16
Filesystem created: Sat May 18 15:07:01 2019
Last mount time: Tue Nov 30 09:42:01 2021
Last write time: Tue Nov 30 09:42:01 2021
Mount count: 316
Maximum mount count: -1
Last checked: Sat May 18 15:07:01 2019
Check interval: 0 (<none>)
Lifetime writes: 557 GB
Reserved blocks uid: 0 (user root)
Reserved blocks gid: 0 (group root)
First inode: 11
Inode size: 256
Required extra isize: 32
Desired extra isize: 32
Journal inode: 8
Default directory hash: half_md4
Directory Hash Seed: 6ee2f640-e2e1-4b41-a9f1-a809fda08387
Journal backup: inode blocks
Checksum type: crc32c
Checksum: 0xc93fe4e6
Journal features: journal_incompat_revoke journal_64bit journal_checksum_v3
Total journal size: 1024M
Total journal blocks: 262144
Max transaction length: 262144
Fast commit length: 0
Journal sequence: 0x0006f568
Journal start: 0
Journal checksum type: crc32c
Journal checksum: 0xb05abb12
$ sudo mkfs.ext4 -n /dev/sdd1
mke2fs 1.46.2 (28-Feb-2021)
/dev/sdd1 contains a ext4 file system labelled 'dddd'
last mounted on /media/wii/dddd on Tue Nov 30 09:42:01 2021
Proceed anyway? (y,N) y
Creating filesystem with 58607510 4k blocks and 14655488 inodes
Filesystem UUID: 749e1ee2-dc2f-4b5f-9092-ff6fb796ce3b
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424, 20480000, 23887872
分区可以成功过载，尝试使用备份块修复文件系统。
$ sudo fsck.ext4 -b 32768 /dev/sdd1
e2fsck 1.46.2 (28-Feb-2021)
dddd was not cleanly unmounted, check forced.
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
/lost+found not found. Create<y>? yes
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Free blocks count wrong for group #0 (23484, counted=23488).
Fix<y>? yes
Free blocks count wrong for group #1 (31715, counted=26452).
Fix<y>? yes
Free blocks count wrong for group #2 (32768, counted=451).
Fix<y>? yes
Free blocks count wrong for group #3 (31715, counted=153).
Fix<y>? yes
Free blocks count wrong for group #4 (32768, counted=0).
Fix<y>? yes
Free blocks count wrong for group #5 (31715, counted=4529).
Fix<y>? yes
Free blocks count wrong for group #6 (32768, counted=2588).
Fix<y>? yes
Free blocks count wrong for group #7 (31715, counted=2054).
Fix<y>? yes
Free blocks count wrong for group #8 (32768, counted=261).
Fix ('a' enables 'yes' to all) <y>? yes to all
Free blocks count wrong for group #9 (31715, counted=483).
Fix? yes
...
Free inodes count wrong for group #1584 (8192, counted=7854).
Fix? yes
Directories count wrong for group #1584 (0, counted=53).
Fix? yes
Free inodes count wrong (14652303, counted=14391454).
Fix? yes
Padding at end of inode bitmap is not set. Fix? yes
dddd: ***** FILE SYSTEM WAS MODIFIED *****
dddd: 264034/14655488 files (0.2% non-contiguous), 22178716/58607510 blocks
SSD硬盘使用寿命
sudo apt-get install smartmontools
# -a 打印磁盘所有的SMART信息
# -s on/off 打开或关闭磁盘的SMART功能
sudo smartctl -a /dev/sdb -s on
smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.10.0-9-amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Number: KINGSTON SA1000M8240G
Serial Number: 50026B7682BFBD0F
Firmware Version: E8FK11.T
PCI Vendor/Subsystem ID: 0x2646
IEEE OUI Identifier: 0x0026b7
Total NVM Capacity: 240,057,409,536 [240 GB]
Unallocated NVM Capacity: 0
Controller ID: 0
NVMe Version: 1.2
Number of Namespaces: 1
Namespace 1 Size/Capacity: 240,057,409,536 [240 GB]
Namespace 1 Formatted LBA Size: 512
Namespace 1 IEEE EUI-64: 0026b7 682bfbd0f5
Local Time is: Mon Dec 6 21:48:30 2021 CST
Firmware Updates (0x02): 1 Slot
Optional Admin Commands (0x0007): Security Format Frmw_DL
Optional NVM Commands (0x001e): Wr_Unc DS_Mngmt Wr_Zero Sav/Sel_Feat
Log Page Attributes (0x04): Ext_Get_Lg
Maximum Data Transfer Size: 512 Pages
Warning Comp. Temp. Threshold: 84 Celsius
Critical Comp. Temp. Threshold: 88 Celsius
Supported Power States
St Op Max Active Idle RL RT WL WT Ent_Lat Ex_Lat
0 + 7.90W 0.0790W - 0 0 0 0 600 600
1 + 7.90W 0.0790W - 0 0 0 0 600 600
2 + 7.90W 0.0790W - 0 0 0 0 600 600
3 - 0.1000W 0.0790W - 3 3 3 3 1000 1000
4 - 0.0050W 0.0790W - 4 4 4 4 400000 90000
Supported LBA Sizes (NSID 0x1)
Id Fmt Data Metadt Rel_Perf
0 + 512 0 1
1 - 4096 0 0
=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
SMART/Health Information (NVMe Log 0x02)
Critical Warning: 0x00
Temperature: 27 Celsius
Available Spare: 100%
Available Spare Threshold: 10%
Percentage Used: 0% <-------使用百分比估算
Data Units Read: 1,225,875 [627 GB]
Data Units Written: 1,259,166 [644 GB]
Host Read Commands: 15,108,813
Host Write Commands: 9,069,076
Controller Busy Time: 16
Power Cycles: 770
Power On Hours: 1,211
Unsafe Shutdowns: 235
Media and Data Integrity Errors: 0
Error Information Log Entries: 0
Warning Comp. Temperature Time: 0
Critical Comp. Temperature Time: 0
Temperature Sensor 2: 27 Celsius
Error Information (NVMe Log 0x01, 16 of 16 entries)
No Errors Logged