NVMe Identify Controller failed after 9 minutes reading directory '.': Input/output error

1

Hi All, I have been chasing my tail on and off for months.

I have a Raspberry Pi 5 Model B 16GB with PCIe Expansion Board For Raspberry Pi 5, Raspberry Pi 5 M.2 HAT, Raspberry Pi AI HAT+ 26 TOPS for Raspberry Pi 5, Raspberry Pi 5 Active Cooler, Raspberry Pi 5 White Power Supply (Official) USB-C PD 5.1V 27W with a SD card boot and a 1 tb Corsair MP600 MICRO NVMe SSD mounted as storage for video. It is running frigate NVR without issues for around 9 minutes then the NVMe becomes no longer accessible. The only solution is reboot the pi. Disk testing for I/O reveals no errors and speeds are as expected for a PCIE gen 2. Running Frigate NVR on the SD card without mounting the SSD for storage works as expected.

NVMe formatted as:

/dev/nvme0n1: UUID=“da24ca08-80c5-4286-84eb-81dffd3fb015” BLOCK_SIZE=“4096” TYPE=“ext4”

Errors seen after failure from testing:

Read NVMe Identify Controller failed: NVME_IOCTL_ADMIN_CMD: Input/output error

Smartctl open device: /dev/nvme0 failed: Resource temporarily unavailable

I have reinserted the PCI cable and NVMe SSD drive in the HAT.

I am lost to what to try next.

Any ideas please?

Cheers Matt

2

Hey there, @Matt217152,

With a Pi setup with as many HATs are you have at play here, there could be many potential things that are going wrong.

Let’s start by trying to remove excess variables at play, starting with the physical layer. You have the PCIe Expansion Board between the M2 HAT and the Pi, which is going to add a lot of potential failure points.

Is the behaviour still occurring during the following situations:

  • When the M2 HAT is connected directly to the Pi, without the PCIe Expansion Board?
  • When a different NVMe Drive is used with the M2 HAT? Could you also tell us what NVMe drive you are using? Some have known issues with the Pi.
3

Hi @Jane , Thanks for replying.

I have a 1 tb Corsair MP600 MICRO NVMe as recommend at the time of purchase.

I have not tried without the 2CH PCI hat as I expect the NVMe wont fail because it will not be written to like it is when the AI hat is being used for video recording. Nor have i tied another NVMe. I dont have access to one.

I can try without the 2CH PCI Hat and AI Hat but I will have nothing to run then to test out the NVMe.

Do you have a suggestion on how to stress test the NVMe.

I have been using Sudo smartctl -t short /dev/nvme0 - Sudo smartctl -l selftest /dev/nvme0 to do short tests. It has never failed doing this testing.

Thanks Matt

4

Hey, @Matt217152 I get that it’s a little frustrating, but I’d still strongly recommend breaking the stack down first. With all of these units together, there are just too many variables to tell whether this is a configuration limit or one component misbehaving.

First we need to prove that the NVMe + M.2 HAT combi is stable directly on the Pi. That gives a clean baseline and helps rule out a faulty ribbon, HAT, etc.

To give it a good testing, you should use something like fio command on a file. It will make a controlled disk workload that’s more likely to expose I/O errors, power issues, or device problems. Try using the below:

sudo apt install -y fio

fio --name=nvme_stress \
  --directory=/mnt/nvme/fio-test \
  --filename=testfile \
  --size=20G \
  --time_based \
  --runtime=20m \
  --rw=write \
  --bs=1M \
  --iodepth=16 \
  --direct=1 \
  --numjobs=1 \
  --group_reporting

Also, please keep sudo dmesg -w running during the test. It will help reveal any faults.

1 Like
5

​​​Hello @Jane .

Thank you for your detailed information.

I did as suggested. No logs in dmesg with only the m2 hat. Ran the same test with all hats. No Dmesg logs. Ran frigate and I get this after a few minutes.

[ 1506.321960] nvme nvme0: controller is down; will reset: CSTS=0xffffffff, PCI_STATUS=0x10

[ 1506.321985] nvme nvme0: Does your device have a faulty power saving mode enabled?

[ 1506.322121] nvme nvme0: Try “nvme_core.default_ps_max_latency_us=0 pcie_aspm=off pcie_port_pm=off” and report a bug

[ 1506.389964] nvme 0001:04:00.0: enabling device (0000 → 0002)

[ 1506.390029] nvme nvme0: Disabling device after reset failure: -19

[ 1506.401938] EXT4-fs warning (device nvme0n1): ext4_end_bio:368: I/O error 10 writing to inode 12058636 starting block 159431988)

[ 1506.401966] EXT4-fs warning (device nvme0n1): ext4_end_bio:368: I/O error 10 writing to inode 12058637 starting block 159432192)

[ 1506.401971] EXT4-fs (nvme0n1): failed to convert unwritten extents to written extents – potential data loss! (inode 12058636, error -5)

[ 1506.402005] EXT4-fs warning (device nvme0n1): ext4_end_bio:368: I/O error 10 writing to inode 12058635 starting block 159432204)

etc etc

So I suspecting this “nvme nvme0: Try “nvme_core.default_ps_max_latency_us=0 pcie_aspm=off pcie_port_pm=off” and report a bug”

so by sudo nano /boot/firmware/cmdline.txt and adding pcie_aspm=off pcie_port_pm=off to the end of the string of existing commands.

So far it has been running 30 minutes without fail. That sudo dmesg -w is just what i needed.

Do you think I need to look at anything else?

Thanks Matt

2 Likes
6

Hi Matt,

At a glance these error’s indicate an issue with driver availability, and the power mode is the most common type (and it would be uncommon that a network storage device would drop out for other reasons).

But I hope this fixes it! Please let us know if you run into any future issues :smiley:
Liam

1 Like
7

Hey Matt,

Glad you were able to find a solution. As Liam says, power mode issues can be pretty common with peripherals, I’m glad dmesg pointed you in the right direction.

1 Like