- Joined
- Dec 12, 2011
- Messages
- 111
- Motherboard
- Gigabyte Z590 Vision G
- CPU
- i7-11700K
- Graphics
- RX 6900 XT
- Mobile Phone
General NVMe Drive Problems (Fatal)
- Thread starter c-o-pr
- Start date
- Joined
- Jan 30, 2011
- Messages
- 458
- Motherboard
- ASUS ROG RAMPAGE VI EXTREME
- CPU
- i9-10980XE
- Graphics
- RX 5700 XT
Last month, I replaced a hardware component and had to unplug/replug one of my NVME drives connected via PCIe Riser cable. Over the next days and weeks, I experienced 4-5 random kernel panics, but I didn't know what was causing them. After reading the KPs, I suspected a hardware failure of the NVMe drive, as 3 of them referred to a "3rd party NVMe controller. Loss of MMIO space. Delete IO submission queue. fBuiltIn=1 MODEL=Samsung SSD 960 PRO 2TB". Weeks later, I decided to turn off the computer, wipe the PCIe connectors of the drive, and plug it back in, and the kernel panics stopped.
It seems that in my case, the issue was a hardware failure/defect due to the drive not being properly connected to the riser cable.
- Joined
- May 22, 2011
- Messages
- 914
- Motherboard
- Gigabyte Z370N-WiFi
- CPU
- i7-8700
- Graphics
- RX 6800 XT
- Mac
- Classic Mac
- Mobile Phone
@Thireus
Very interesting -- thank you for sharing that.
So in that case it was just sort of an occasionally not perfect physical hardware connection?
Man -- this makes it SO hard to troubleshoot something like this..
It's dusty where I live (Idaho) and I wonder if dust can get in to connections and cause just enough an issue here.
My issues have been so inconsistent
Very interesting -- thank you for sharing that.
So in that case it was just sort of an occasionally not perfect physical hardware connection?
Man -- this makes it SO hard to troubleshoot something like this..
It's dusty where I live (Idaho) and I wonder if dust can get in to connections and cause just enough an issue here.
My issues have been so inconsistent
- Joined
- Jan 30, 2011
- Messages
- 458
- Motherboard
- ASUS ROG RAMPAGE VI EXTREME
- CPU
- i9-10980XE
- Graphics
- RX 5700 XT
Correct. I cannot be 100% certain, but it seems to have been the case. Currently I've cummulated 10days of uptime, something I didn't manage to achieve since the first NVME KP appeared last month.
trs96
Moderator
- Joined
- Jul 30, 2012
- Messages
- 25,553
- Motherboard
- Gigabyte B460M Aorus Pro
- CPU
- i5-10500
- Graphics
- RX 570
- Mac
- Mobile Phone
I've had situations where I shipped a PC across country, 2,000 miles via ground shipping. Everything tested as working perfectly before packing it. When it got to the recipient it was throwing all kinds of blinking light and beep code errors that I had to decipher. Would not boot at all. There was no physical damage to the box it was packed in. The PC inside was packed well and nothing happened to that either.
After some research I discovered that the ram, all four modules, had to be re-seated. After that it booted normally and it hasn't been a problem since then. The DIMMs were all secure in their sockets so I think that something related to the changing temps and pressure on the long trip through the mountains, somehow affected the board and ram slots.
Last edited:
- Joined
- May 22, 2011
- Messages
- 914
- Motherboard
- Gigabyte Z370N-WiFi
- CPU
- i7-8700
- Graphics
- RX 6800 XT
- Mac
- Classic Mac
- Mobile Phone
ha! touché indeed..
I will say though, they'd I've been building PCs for decades and never had anything like this happening.
An odd one.
Last edited:
- Joined
- Apr 12, 2021
- Messages
- 903
- Motherboard
- Asus z590 ROG Maximus XIII Hero
- CPU
- i9-11900K
- Graphics
- RX 6600 XT
- Mac
- Classic Mac
- Mobile Phone
Over at the thread...
Choosing a Compatible SSD For a Boot Drive
www.tonymacx86.com
...a poster (OP) asked if spontaneous boot failures he was seeing, without any other system changes, might be related to an incompatible drive?
The OP question wasn't quite appropriate to that thread, but it's a fair question in context, as the the thread originator, @trs96, offers views on Apple SSD design, hazards, trends in the consumer NVMe market, and helpful recommendations based on experience from the forums.
I took a fair amount of effort to assist OP, with many suggestions for troubleshooting, which I wrote in a way I hoped to be helpful to anyone with storage failures. Unfortunately for me, my effort was dumped down the memory hole when my posts were moderated out of existence due to being too far off-topic for that thread. That was fair, as I had only skimmed threads origin post, so my mistake. But I am not a moderator and I have no power to organize, threads, so I just offer help in context where I can. Frustrating.
In order to make my frustration productive, I re-read the origin post and recalled many other articles and experiences on NVMe SSDs and macOS.
I began to wonder about the idea of SSD "compatibility." Given the limited information available on drive operation and macOS, I think we can only discuss "incompatibility," as meaning applications of drives which haven't failed yet.
I noticed some patterns on incompatibility, as follows:
Three main issues:
A—Power (excessive power consumption in laptops)
B—Kernel panics due to command timeouts
C—Trim stalls at boot (and in my experience also during installation and snapshot mgmt).
In rare cases there are two other problem areas:
D—Total device failure (rare but I've seen it myself repeatedly, and mentioned elsewhere)
E—Problems with external NVMe hanging
"A & B" have never affected me, so I have no comments. I assume NVMeFix is helpful. I don't happen to use NVMeFix any more on my build (SK-Hynix P41 Platinum boot drvie and WD SN750 backup) with no ill effects. But I was using it with my failed Rocket4s.
"D & E" have both affected me directly and gave rise to creating this thread. I suffered complete disaster of a pair of new Sabrent Rocket 4s under Big Sur when I was first assembling my build, and later with Monterey I had problems with 980 Pro hanging with used with a Type-C ext enclosure. The ext-hanging caused me to go on a long review and rebuild of my USB config, but I was never able to attribute the problem to USB mapping. My problems with ext NVMe hanging abated since a later Monterey release and Ventura has always worked fine.
This leaves "C", for which I went through a major headache when an otherwise perfectly fine 980 Pro I'd purchased to resolve my Sabrent Rocket 4 disasters started taking 15 minutes to boot. It acted up after a Samsung firmware update under Monterey in early 2022. Early in my build I also used a 970 EVO without problems on Big Sur, but I needed a bigger drive. That 970 EVO predates a Samsung controller change to the model. I now use it every day as a USB-attached scratch drive. I've also worked with an WD Black SN750 which has always be perfectly reliable and speedy, but after an upgrade from z590 Comet Lake to Rocket Lake, PCI4, I wanted to feel what the new bus might offer, so I choose the SK-Hynix P41 Platinum based on study of reviews. it was the top performing drive in 2022.
As an aside, my conclusion re PCIe4 storage is it certainly gives the best Mac response I have seen. But it doesn't offer a meaningful usability advantage over PCIe3. Moreover, if you have Comet Lake dialed in, there's no value to upgrading to Rocket Lake. Advances for 12th and 13th generation are more significant steps forward.
Today on the forums, I see Samsung Trim stalls "C" are the most commonly reported drive issue. But my point of view may be mostly conformation bias—I don't know.
@trs96 is single-handedly winning the forums over to WD, which seems fine. WD has been consistently reliable for years, while SSD prices have fallen by half or more since 2021, so it's hard to go wrong. But there's evidence that many other choices that can work.
At the same time, I see no solid evidence that there's any firm idea of "macOS compatibility". What we have is mostly anecdotal evidence of incompatibility (this which includes my own stories) that at best amount to "for some reason, certain drives don't work well—or don't work at all."
While there are various lists (linked below). I can't come up with any clear pattern other than Samsung+Trim ("C") stall issues.
It gets confusing because I have seen posts which claim that a certain generation of Phison controller is no good, but then seen other posts which contradict this saying Phison good, Samsung Phoenix bad, etc.
The Compatibility thread above is agreeably preoccupied with WD. But it's justifications are not satisfying. And while Apple is clearly going its own way on everything, I see Trim as the lynch-pin for incompatible hack NVMe.
Trim is topic aggravation for me because in the late 2000s I was a regular on the now bygone Macintouch, and everyone there including the site owner was going cattywumpas over Apple's lack of support for Trim on 3rd party drives, as if it were some great betrayal of the Mac user community. But even supposedly expert opinion could not coherently explain what Trim did, why it was needed, and any downside for not supporting it. The best article of that age was written for Ars Technica, but its conclusions begged the question: Trim is needed or it would. I did a lot searching, and the best information I could come up with was academic papers justifying Trim for wear levelling, given contrived scenarios.
Old timers wlll recall "Trim enablers" and Apple ultimately offering the trimforce command, with a big warning "USE AT OWN RISK." Today, I think it's truly ironic that after Mac cognoscenti finally got the Trim support they always wanted, it's biting a new generation of enthusiasts in the backside. And still nobody really knows what Trim is good for or when to use it.
Obviously, I've got a bee in my bonnet on this stuff, so the rest of this post will be a compendium of annotated links pertaining to hazard "C", its workarounds, and my opinions on the general issue of SSD Trim. I hope this will provide a further helpful point of view on the situation from a particular perspective, and I welcome all thoughts and criticisms.
Off to the races...!
NVMe TRIM SPEC
In NVMe, Trim support is handled by an optional feature called "Dataset Management (DSM)."
You will find this referred to in the OpenCore documentation for "SetApfsTrimTimeout" quirk.
(Please read below the fold.)
NVM-Express-NVM-Command-Set-Specification (PDF)
Pg 28, section 3.2.3 — NVM Commandset: Dataset Management.
The DSM has an interesting range of attributes (opcodes):
Interestingly, Ubuntu (and maybe other Linux) expose NVME DSM operations as a user command nvme-dsm. (see LINUX section at end)
OPENCORE TRIM QUIRK
OpenCore Configuration Guide (as of 0.9.1) — SetApfsTrimTimeout.
Users with boot stalls can look for Spacemanger console boot log entries, accompanied by high drive activity for the duration of the stall. Try watching your mainboard's drive access LED.
The vocabulary is in the documentation is imprecis and doesn't exactly pertain to macOS. "Unmapped" and "deallocated" are figures of speech, not details about the design of Spaceman.
Moreover, the last sentence is a contradiction of the preceding sentence: the first says macOS has no state of "unmapped," it includes only "used or free," then it goes on to say all free space is "unmapped" at boot.
I read this as 'where other filesystems may track unmapped status, APFS does not.'
Meaning that Spaceman Trims everything marked as free without regard to previous status. The point about "fragmentation" seems incidental. It will never be explained how any particular number of DSM commands might lead to a slowdown.
So read the timeout option as "if it's not finished in 10 seconds" Triming is discontinued. The details are never explained.
...the time required for the drive to complete all the Spaceman DSM requests may far exceed the timeout.
Again, the terms "unmapped" and "deallocated" are used interchangeably in the documentation without a clear definition of either, so it's difficult to interpret these claims. In this OC documentation,"deallocate" is likely jargon. As the too-commonly mis-used term "garbage collection" in other literature.
In the literature, the drive operations are know as follows:
"Unmap" is the SCSI command set term.
"Trim" is the ATA command set term.
"DSM" is the NVMe command set term.
"Discard" is a Lunix term—I mention this because it's the only SW where we have visibility into the code.
I'm going to use discard from here on to mean the OSes, inc macOS, invocation of drive IO commands.
With this clarification, we can move on to observe that we can't easily study the specific discards done by Spaceman, nor do we have details on Samsung's implementation of DSM functionality (which is optional), so there's no way to know why Spaceman stalls.
One great reason that Samsung is slow is because it's DSM commands are actually implemented, whereas on other drives they may not be! But there are many possibilities.
APFS Fragmentation may or may not be relevant. This is never discussed.
As Spaceman doesn't track discard status, there's no basis for any idea of Spaceman Trim progress outside the current boot cycle. In the event of a timeout, there can be no resumption of interrupted Trim progress. So we can assume that Spaceman issues discards for all free space, which goes as far as it can under the time limit. And this interpretation fits the reported behavior perfectly.
Here, do not read "non-functional" as that the drive is rendered inoperative. Read it as once Trim cycles cause boot to take minutes, the system is effectively non-functional because re-booting hacks is routine to troubleshooting and minutes-long reboots are impractical.
At this point the patch is creating problems as much as it's working around them: DSM commands are issued for all free space, but why is this good? What is the "problem" exactly?
This next comment I find especially interesting:
The "found by the controller" simply makes no sense. The controller doesn't have awareness of the volume structure. That would be a compatibility nightmare. I discuss why below.
A little detour here... The article is in Russian, so here's the translation:
Can an SSD be used effectively without TRIM support? | Notes of an IT specialist
interface31-ru.translate.goog
It's a decent article. Please read it. Tldr Answer: Yes! Very much so.
As it mentioned in the OC documentation. Since Monterey, Only the value of zero (0) and -1 are respected.
My experience with Samsung 980 Pro, as of macOS 12 and 13, is that a timeout value of 0 is not reliable to disable discards:
—Spaceman may discard even if a timeout of 0 is supplied.
—If Spaceman is not allowed to complete, it will be run the next boot.
—If Spaceman is allowed to complete, it may not attempt discard again until some unknown threshold event. There is some trigger condition which will cause a Spaceman discard pass over all free space in the volume irrespective of the timeout. I believe trigger is related to system volume snapshot state changes. This means full discard pass(es) are run during macOS update installation. When the discard pass runs, the time required depends primarily on the amount of volume free space.
On the 980 Pro, the discard DSM operations count against wear, according to Samsung Magician.
MY PERSONAL OBSERVATIONS ON THE HISTORY OF TRIM (DSM, UNMAP)
The essence of Trim is the assumption that it is helpful to drive performance for the host to inform the drive of LBA allocation status. The root of this assumption are following facts about flash storage devices:
—The drive presents its store as an LBAs (logical block addresses, 0 to N-1) blocks for the advertised extent of the drive. Drive commands are issued for ranges of LBAs.
—The drive must handle LBA writes at a flash-block level, where flash blocks are large multiples (10s or more) of LBA blocks. Internally, the controler translates LBAs into page and cell addresses according to structure of flash and whatever proprietary juju the controller implements.
—In order to write to an LBA block, the portion of the flash block to be updated must be in a fresh state.
—When fresh, the write request for an LBA-block-sized area (which can be orders of magnitude smaller than the affected flash block) can be written just once, after which it must be refreshed. Other fresh regions of the same flash block can be written, at any time.
—Refreshing written blocks can only be done by refreshing the entire flash block which contains them. So if any LBA block within a flash black is to be rewritten, the entire flash block must first be refreshed.
—Flash blocks tolerate a limited number of refresh cycles (around 1000) after which the block becomes unreliable.
—These features of flash storage leads to a performance effect widely termed write-amplification: small writes from the host may produce orders of magnitude larger effects internal to the drive, affecting throughput and wear. A drive-internal control control stratum called the "flash translation layer (FTL)" determines how LBAs are mapped onto flash blocks. Using various levels of caching and allocation, the controller manages the wear and performance implications of write-amplification.
The history of Trim is with Compaq and Microsoft Windows. Flash was introduced in the mid 80s and SSDs used very simple controllers, basically directly exposing the flash structure to the storage driver. Microsoft extended the ATA commandset to add Trim, which allowed the driver to supply allocation hints to the drive.
Overview of ATA (Wikipedia)
en.m.wikipedia.org
Compact flash was introduced in 1986, at the same time the 80386, which is the birth-point of the modern PC.
The design of flash memory and accommodations for Trim predates generations of PC storage interface design, but at the time Microsoft was adding ATA Trim, it was a common design approach to place complex device mgmt cores in SW to gain value from the PC's most-expensive component, the CPU. As a case in point, in the 90s, end-user cost of PC telephone modems was reduced by putting the modulator/demodulator function in a device driver, rather than include a dedicated component in the modem.
See Softmodem / Winmodem
en.wikipedia.org
In a similar vein but occurring a decade later than Softmodem, early SSDs had simple controllers where the ATA spec relied on programmed-I/O (host CPU instructions moving data to/from the device and testing status), so the CPU was always directly involved in IO.
In the evolution of PC, there was a time when ATA direct host memory access by devices (DMA) was a significant, expensive advance. In the long run this cost was overcome by the enormous increases in logic density per unit cost of dedicated controllers. Today a $50 NVMe drive relies on a dedicated multi-core controller CPU hundreds of times more powerful than an 1980s IBM PC AT.
Trim in Computing (Wikipedia)
en.wikipedia.org
While the awareness by the host of deeply internal device mgmt details is as old as computing, the addition of ATA Trim came relatively late in the modular evolution of PC, first appearing in Windows 2007.
The idea of Trim has always been to give the drive hints (or early notice) to prepare memory for reuse.
MORE THOUGHTS ON DSM
To be very clear, implementation of Trim-related commands has always been optional! The specs have always said so.
The only reliable way for a drive to know the disposition of its host LBA space is for the host to tell it. And as a case in point, we have macOS APFS Spaceman, which is system code pertaining to APFS structures.
But the host doesn't have to issue discards per se in order to information the drive.
Even at the highest level of analysis, Trim commands work in several modes:
— Synchronous vs asynchronous: Does the OS storage manager have to wait for DSM to complete?
— NVMe and later-ATA-spec TRIM allows requests that affected extents be placed in a specific state, e.g. zeroed, vs indeterminate. There's an idea of "secure erase," etc.
— Discard works on LBA blocks, flash is reset by multi-block pages. What happens to discard of unaligned extents?
— What is the effect of discard on device command-queue? If discard is intermixed with other commands, what are the dependencies? Must the queue be flushed?
— If discard doesn't request a specific result state, is it OK for the drive to do nothing? e.g. interplay with secure erase functionality. IOW, can implementation options be partial?
We find many concerns which are complex and invite peculiar edge cases
All these concerns leads to a conundrum the basic value proposition of Trim:
THE DEBACLE OF MODERN SSD TRIM
If the host is going to pitch-in on the chore of flash readiness, when is a good time for the OS to tell the drive to prepare? Boot time? During mainline system operation? Never?
What happens to system performance if deleting files must generate synchronous discards (effectively writes) for the deleted file's related storage extents? Normally file deletion is accomplished by updating tiny amount of meta-data, to forget the associated storage extents.
What if discards blow up the drive command queue?
What about race conditions between discards that imply a specific storage state and command syncrony? Etc...
If the OS is going to be involved in flash storage maintenance, there are more or less advantageous times for it to conduct this chore depending on workloads, which is why the OS tends to relegate Trim to a maintain point, e.g., boot.
A survey of system admin literature reveals various approaches. Different OSes have their own ways. But this is most clearly seen in the Linux command structures which divide the chore of discard between volume mount options and an administrative command called fstrim(8). See Linux references at end.
Moreover, the ATA/NVMe specs keep evolving, the drive controllers keep getting more sophisticated, and different drives may treat DSM options differently.
Can we generalize a posture towards the administration of discard?
I think so. The progress of system design is rooted in the truism that good design modularizes concerns of subsystems to hide details in order to manage complexity; so don't expose the complexity of one subsystem to another without a clear theory of dependences that justifies the exposure. When a design gets modularity wrong, it becomes awkward and confusing.
SSD Trim being a case in point. But any measure, flash readiness for I/O must be regarded as a purely internal concern of a drive. There was a time when making the host aware of flash mgmt was appropriate. This was during the early development of SSDs, when it was impractical to use a dedicated controller and FTL was very simple.
Thanks to powerful dedicated chips we now find increasing modularity across every aspect of computer systems design. But this doesn't imply seamless interoperation. Modularity may manifest in surprising ways, such as Apple sealing its devices and spreading functionality across components that are considered separate modules in the PC market. There is no contradiction: Apple's closed designs give them the power to add value through vertical integration because they hide subsystem dependencies that might be visible and therefore inappropriate for PC parts.
That SSD Trim has become a point of friction for systems administration is in keeping with the poor design cut that is moderm Trim. In the NVM Express Commandset Spec, we find that the DSM sub-commandset allows the host to signal the drive with a wide range of hints for optimizing access, including caching hints. Again these are purely advisory, with the drive maker free to no implement and meet spec. But Apple may have other thoughts which result in foibles for hackintoshers.
Yet, even if Apple is a perfect student of PC modularoty, with no internal agenda, we should expect that market pressures will result in many drive vendors creating partial DSM implementations with awkward side-effects or failure modes. And finally, even if the implementations were somehow perfect, drive vendors tailor capabilities and behavior to market segments. So the permutations are a nightmare.
If there's any single conclusion that a user might draw from all this today, I think is "avoid discard."
Now I'll explain how a system can avoid the concern of discard.
THE MAGIC OF OVER PROVISIONING
First, I want to expose a common misunderstanding aver-provisioning:
There's an idea among Windows and Linux nerds that an SSD can be usefully over-provisioned be leaving "free-space" via the partition table (e.g., GPT layout). As mentioned in the OC documentation above, there's this idea of "leaving space where the controller can find it," without any sense of wonder over how exactly the drive is informed of this "free" space. How does the drive get information about volume layout if the OS doesn't tell it? It's possible the drive could know how to read partition layout: NOT!. The drive maker will have to maintain compatibility with system SW, but we find no traits of OS per drive filesystem compatibility listed by drive makers.
So how does this work? If just having space on a drive which is rarely touched helps the controller, there's almost always space that's rarely touched, unless the drive is full. So, from the drive perspective, what's special about about one area that goes untouched because it's outside any volume, versus area that just hasn't been touched yet? There isn't any obvious distinction.
For routine PC uses, a full drive and the need for high performance are mutually exclusive: if the drive is full, then write performance and wear are a minimal concern, storage is exhausted. Running a drive past full seems like a degenerate case. User-selectable headroom, might make sense for use-case where a drive is kept close to full with a very dynamic workload, which is not typical of consumer devices.
On Windows, some drives come with optional utilities that include "over-provisioning" controls. These controls can have awareness of Windows storage layouts. In such case, the utility might use this information to program the drive controller with proprietary layout specific parameters, e.g., the Magician utility can tell the drive certain area is headroom for performance using secret attributes in drive controller. Bit what does this have to do with OpenCore hackintosh, which has no such utility?
As far as I'm concerned, SSD over-provisioning secret sauce is at best academic and mostly irrelevant.
IOW, the OS is always supplying the drive the needed information for garbage collection through normal operation. Even without discard or explicit over-provisioning, the drive constantly receives information about flash reclamation from ordinary I/O: a write implies that the previously associated flash for its LBA extent is reclaimable.
There might be a scheduling advantage to Trim, in that signaling the opportunity for reclamation ahead of writes may allow hiding of latency in flash page setup. But a similar advantage can by arranged from pipelining writes and flash maintenance: when processing writes, the controller FTL sends the data for the requested LBAs to prepped flash pages, and notes the previously associated pages can be refreshed. Writes aren't delayed by flash maintenance latency which happens in the background. A slight over-provision in the drive (which can be hidden-from the user) ensures there's always free flash to ready to receive writes even when the drive is nearly full.
My conclusion is that discard and provisioning also should be a purely internal concerns for the drive logic, where drive traits may be marketed in manner stylized and tweaked for some general purposes, much as spinning hard drives are marketed in 2023.
SOME OPENCORE HISTORY ASSOCIATED WITH SSD INCOMPATIBILITY
Here is a selection of links which capture the history of SSD compatibility with Open Core:
Dortania — Anti-Hackintosh Buyers Guide — Storage
It's difficult to make sense of all these reports because of edge cases and lack of comprehensive study.
I use a SK Hynix Platinum P41 on an Asus Z590 Hero with 11th gen and Monetery through Ventura 13.3 and it's 100% perfectly reliable and fastest our of 4 drives I have used, including 980 Pro.
SSD hall of fame needs new entries #192
github.com
...cont.
Acidanthera NVMeFix.kext
github.com
LINUX COMMANDS FOR TRIM ADMINSTRATION
nvme-dsm (Ubuntu) man page
manpages.ubuntu.com
fstrim(8) man page
Linux fstab(8)
FURTHER READING
500 SSDs Listed by Brand, Model, Interface, Form Factor, Capacities, Controller, Cores, DRAM, HMB, NAND Brand, NAND Type, Layers, Categories, Notes, Product Page
docs.google.com
The Ultimate Guide to Apple’s Proprietary SSDs
(Higher level presentation that I would wish, but still a nice review of Mac-specific SSD tech.)
Choosing a Compatible SSD For a Boot Drive
Choosing a Compatible NVMe SSD for your macOS Boot Drive
Introduction to NVMe Choices If you have any previous hackintosh experience, you'll know that a limited number of hardware components have native support in macOS. You can't buy just any graphics card and expect it to work. Only certain Broadcom models of Wifi/BT cards have native support etc...
www.tonymacx86.com
...a poster (OP) asked if spontaneous boot failures he was seeing, without any other system changes, might be related to an incompatible drive?
The OP question wasn't quite appropriate to that thread, but it's a fair question in context, as the the thread originator, @trs96, offers views on Apple SSD design, hazards, trends in the consumer NVMe market, and helpful recommendations based on experience from the forums.
I took a fair amount of effort to assist OP, with many suggestions for troubleshooting, which I wrote in a way I hoped to be helpful to anyone with storage failures. Unfortunately for me, my effort was dumped down the memory hole when my posts were moderated out of existence due to being too far off-topic for that thread. That was fair, as I had only skimmed threads origin post, so my mistake. But I am not a moderator and I have no power to organize, threads, so I just offer help in context where I can. Frustrating.
In order to make my frustration productive, I re-read the origin post and recalled many other articles and experiences on NVMe SSDs and macOS.
I began to wonder about the idea of SSD "compatibility." Given the limited information available on drive operation and macOS, I think we can only discuss "incompatibility," as meaning applications of drives which haven't failed yet.
I noticed some patterns on incompatibility, as follows:
Three main issues:
A—Power (excessive power consumption in laptops)
B—Kernel panics due to command timeouts
C—Trim stalls at boot (and in my experience also during installation and snapshot mgmt).
In rare cases there are two other problem areas:
D—Total device failure (rare but I've seen it myself repeatedly, and mentioned elsewhere)
E—Problems with external NVMe hanging
"A & B" have never affected me, so I have no comments. I assume NVMeFix is helpful. I don't happen to use NVMeFix any more on my build (SK-Hynix P41 Platinum boot drvie and WD SN750 backup) with no ill effects. But I was using it with my failed Rocket4s.
"D & E" have both affected me directly and gave rise to creating this thread. I suffered complete disaster of a pair of new Sabrent Rocket 4s under Big Sur when I was first assembling my build, and later with Monterey I had problems with 980 Pro hanging with used with a Type-C ext enclosure. The ext-hanging caused me to go on a long review and rebuild of my USB config, but I was never able to attribute the problem to USB mapping. My problems with ext NVMe hanging abated since a later Monterey release and Ventura has always worked fine.
This leaves "C", for which I went through a major headache when an otherwise perfectly fine 980 Pro I'd purchased to resolve my Sabrent Rocket 4 disasters started taking 15 minutes to boot. It acted up after a Samsung firmware update under Monterey in early 2022. Early in my build I also used a 970 EVO without problems on Big Sur, but I needed a bigger drive. That 970 EVO predates a Samsung controller change to the model. I now use it every day as a USB-attached scratch drive. I've also worked with an WD Black SN750 which has always be perfectly reliable and speedy, but after an upgrade from z590 Comet Lake to Rocket Lake, PCI4, I wanted to feel what the new bus might offer, so I choose the SK-Hynix P41 Platinum based on study of reviews. it was the top performing drive in 2022.
As an aside, my conclusion re PCIe4 storage is it certainly gives the best Mac response I have seen. But it doesn't offer a meaningful usability advantage over PCIe3. Moreover, if you have Comet Lake dialed in, there's no value to upgrading to Rocket Lake. Advances for 12th and 13th generation are more significant steps forward.
Today on the forums, I see Samsung Trim stalls "C" are the most commonly reported drive issue. But my point of view may be mostly conformation bias—I don't know.
@trs96 is single-handedly winning the forums over to WD, which seems fine. WD has been consistently reliable for years, while SSD prices have fallen by half or more since 2021, so it's hard to go wrong. But there's evidence that many other choices that can work.
At the same time, I see no solid evidence that there's any firm idea of "macOS compatibility". What we have is mostly anecdotal evidence of incompatibility (this which includes my own stories) that at best amount to "for some reason, certain drives don't work well—or don't work at all."
While there are various lists (linked below). I can't come up with any clear pattern other than Samsung+Trim ("C") stall issues.
It gets confusing because I have seen posts which claim that a certain generation of Phison controller is no good, but then seen other posts which contradict this saying Phison good, Samsung Phoenix bad, etc.
The Compatibility thread above is agreeably preoccupied with WD. But it's justifications are not satisfying. And while Apple is clearly going its own way on everything, I see Trim as the lynch-pin for incompatible hack NVMe.
Trim is topic aggravation for me because in the late 2000s I was a regular on the now bygone Macintouch, and everyone there including the site owner was going cattywumpas over Apple's lack of support for Trim on 3rd party drives, as if it were some great betrayal of the Mac user community. But even supposedly expert opinion could not coherently explain what Trim did, why it was needed, and any downside for not supporting it. The best article of that age was written for Ars Technica, but its conclusions begged the question: Trim is needed or it would. I did a lot searching, and the best information I could come up with was academic papers justifying Trim for wear levelling, given contrived scenarios.
Old timers wlll recall "Trim enablers" and Apple ultimately offering the trimforce command, with a big warning "USE AT OWN RISK." Today, I think it's truly ironic that after Mac cognoscenti finally got the Trim support they always wanted, it's biting a new generation of enthusiasts in the backside. And still nobody really knows what Trim is good for or when to use it.
Obviously, I've got a bee in my bonnet on this stuff, so the rest of this post will be a compendium of annotated links pertaining to hazard "C", its workarounds, and my opinions on the general issue of SSD Trim. I hope this will provide a further helpful point of view on the situation from a particular perspective, and I welcome all thoughts and criticisms.
Off to the races...!
NVMe TRIM SPEC
In NVMe, Trim support is handled by an optional feature called "Dataset Management (DSM)."
You will find this referred to in the OpenCore documentation for "SetApfsTrimTimeout" quirk.
(Please read below the fold.)
NVM-Express-NVM-Command-Set-Specification (PDF)
Pg 28, section 3.2.3 — NVM Commandset: Dataset Management.
The DSM has an interesting range of attributes (opcodes):
Pg 31, Figure 42: Dataset Management – Context Attributes
Interestingly, Ubuntu (and maybe other Linux) expose NVME DSM operations as a user command nvme-dsm. (see LINUX section at end)
OPENCORE TRIM QUIRK
OpenCore Configuration Guide (as of 0.9.1) — SetApfsTrimTimeout.
Users with boot stalls can look for Spacemanger console boot log entries, accompanied by high drive activity for the duration of the stall. Try watching your mainboard's drive access LED.
The vocabulary is in the documentation is imprecis and doesn't exactly pertain to macOS. "Unmapped" and "deallocated" are figures of speech, not details about the design of Spaceman.
Moreover, the last sentence is a contradiction of the preceding sentence: the first says macOS has no state of "unmapped," it includes only "used or free," then it goes on to say all free space is "unmapped" at boot.
I read this as 'where other filesystems may track unmapped status, APFS does not.'
Meaning that Spaceman Trims everything marked as free without regard to previous status. The point about "fragmentation" seems incidental. It will never be explained how any particular number of DSM commands might lead to a slowdown.
So read the timeout option as "if it's not finished in 10 seconds" Triming is discontinued. The details are never explained.
...the time required for the drive to complete all the Spaceman DSM requests may far exceed the timeout.
Again, the terms "unmapped" and "deallocated" are used interchangeably in the documentation without a clear definition of either, so it's difficult to interpret these claims. In this OC documentation,"deallocate" is likely jargon. As the too-commonly mis-used term "garbage collection" in other literature.
In the literature, the drive operations are know as follows:
"Unmap" is the SCSI command set term.
"Trim" is the ATA command set term.
"DSM" is the NVMe command set term.
"Discard" is a Lunix term—I mention this because it's the only SW where we have visibility into the code.
I'm going to use discard from here on to mean the OSes, inc macOS, invocation of drive IO commands.
With this clarification, we can move on to observe that we can't easily study the specific discards done by Spaceman, nor do we have details on Samsung's implementation of DSM functionality (which is optional), so there's no way to know why Spaceman stalls.
One great reason that Samsung is slow is because it's DSM commands are actually implemented, whereas on other drives they may not be! But there are many possibilities.
APFS Fragmentation may or may not be relevant. This is never discussed.
As Spaceman doesn't track discard status, there's no basis for any idea of Spaceman Trim progress outside the current boot cycle. In the event of a timeout, there can be no resumption of interrupted Trim progress. So we can assume that Spaceman issues discards for all free space, which goes as far as it can under the time limit. And this interpretation fits the reported behavior perfectly.
Here, do not read "non-functional" as that the drive is rendered inoperative. Read it as once Trim cycles cause boot to take minutes, the system is effectively non-functional because re-booting hacks is routine to troubleshooting and minutes-long reboots are impractical.
At this point the patch is creating problems as much as it's working around them: DSM commands are issued for all free space, but why is this good? What is the "problem" exactly?
This next comment I find especially interesting:
The "found by the controller" simply makes no sense. The controller doesn't have awareness of the volume structure. That would be a compatibility nightmare. I discuss why below.
A little detour here... The article is in Russian, so here's the translation:
Can an SSD be used effectively without TRIM support? | Notes of an IT specialist
Можно ли эффективно использовать SSD без поддержки TRIM?
Вынесенный нами в заголовок вопрос довольно часто занимает умы системных администраторов. Действительно, что лучше, собрать из твердотельных дисков RAID массив, но потерять поддержку TRIM, или отказаться от отказоустойчивости в пользу высокой производительности? Ситуация усугубляется еще и тем...
It's a decent article. Please read it. Tldr Answer: Yes! Very much so.
As it mentioned in the OC documentation. Since Monterey, Only the value of zero (0) and -1 are respected.
My experience with Samsung 980 Pro, as of macOS 12 and 13, is that a timeout value of 0 is not reliable to disable discards:
—Spaceman may discard even if a timeout of 0 is supplied.
—If Spaceman is not allowed to complete, it will be run the next boot.
—If Spaceman is allowed to complete, it may not attempt discard again until some unknown threshold event. There is some trigger condition which will cause a Spaceman discard pass over all free space in the volume irrespective of the timeout. I believe trigger is related to system volume snapshot state changes. This means full discard pass(es) are run during macOS update installation. When the discard pass runs, the time required depends primarily on the amount of volume free space.
On the 980 Pro, the discard DSM operations count against wear, according to Samsung Magician.
MY PERSONAL OBSERVATIONS ON THE HISTORY OF TRIM (DSM, UNMAP)
The essence of Trim is the assumption that it is helpful to drive performance for the host to inform the drive of LBA allocation status. The root of this assumption are following facts about flash storage devices:
—The drive presents its store as an LBAs (logical block addresses, 0 to N-1) blocks for the advertised extent of the drive. Drive commands are issued for ranges of LBAs.
—The drive must handle LBA writes at a flash-block level, where flash blocks are large multiples (10s or more) of LBA blocks. Internally, the controler translates LBAs into page and cell addresses according to structure of flash and whatever proprietary juju the controller implements.
—In order to write to an LBA block, the portion of the flash block to be updated must be in a fresh state.
—When fresh, the write request for an LBA-block-sized area (which can be orders of magnitude smaller than the affected flash block) can be written just once, after which it must be refreshed. Other fresh regions of the same flash block can be written, at any time.
—Refreshing written blocks can only be done by refreshing the entire flash block which contains them. So if any LBA block within a flash black is to be rewritten, the entire flash block must first be refreshed.
—Flash blocks tolerate a limited number of refresh cycles (around 1000) after which the block becomes unreliable.
—These features of flash storage leads to a performance effect widely termed write-amplification: small writes from the host may produce orders of magnitude larger effects internal to the drive, affecting throughput and wear. A drive-internal control control stratum called the "flash translation layer (FTL)" determines how LBAs are mapped onto flash blocks. Using various levels of caching and allocation, the controller manages the wear and performance implications of write-amplification.
The history of Trim is with Compaq and Microsoft Windows. Flash was introduced in the mid 80s and SSDs used very simple controllers, basically directly exposing the flash structure to the storage driver. Microsoft extended the ATA commandset to add Trim, which allowed the driver to supply allocation hints to the drive.
Overview of ATA (Wikipedia)
Parallel ATA - Wikipedia
Compact flash was introduced in 1986, at the same time the 80386, which is the birth-point of the modern PC.
The design of flash memory and accommodations for Trim predates generations of PC storage interface design, but at the time Microsoft was adding ATA Trim, it was a common design approach to place complex device mgmt cores in SW to gain value from the PC's most-expensive component, the CPU. As a case in point, in the 90s, end-user cost of PC telephone modems was reduced by putting the modulator/demodulator function in a device driver, rather than include a dedicated component in the modem.
See Softmodem / Winmodem
Softmodem - Wikipedia
In a similar vein but occurring a decade later than Softmodem, early SSDs had simple controllers where the ATA spec relied on programmed-I/O (host CPU instructions moving data to/from the device and testing status), so the CPU was always directly involved in IO.
In the evolution of PC, there was a time when ATA direct host memory access by devices (DMA) was a significant, expensive advance. In the long run this cost was overcome by the enormous increases in logic density per unit cost of dedicated controllers. Today a $50 NVMe drive relies on a dedicated multi-core controller CPU hundreds of times more powerful than an 1980s IBM PC AT.
Trim in Computing (Wikipedia)
Trim (computing) - Wikipedia
While the awareness by the host of deeply internal device mgmt details is as old as computing, the addition of ATA Trim came relatively late in the modular evolution of PC, first appearing in Windows 2007.
The idea of Trim has always been to give the drive hints (or early notice) to prepare memory for reuse.
MORE THOUGHTS ON DSM
To be very clear, implementation of Trim-related commands has always been optional! The specs have always said so.
The only reliable way for a drive to know the disposition of its host LBA space is for the host to tell it. And as a case in point, we have macOS APFS Spaceman, which is system code pertaining to APFS structures.
But the host doesn't have to issue discards per se in order to information the drive.
Even at the highest level of analysis, Trim commands work in several modes:
— Synchronous vs asynchronous: Does the OS storage manager have to wait for DSM to complete?
— NVMe and later-ATA-spec TRIM allows requests that affected extents be placed in a specific state, e.g. zeroed, vs indeterminate. There's an idea of "secure erase," etc.
— Discard works on LBA blocks, flash is reset by multi-block pages. What happens to discard of unaligned extents?
— What is the effect of discard on device command-queue? If discard is intermixed with other commands, what are the dependencies? Must the queue be flushed?
— If discard doesn't request a specific result state, is it OK for the drive to do nothing? e.g. interplay with secure erase functionality. IOW, can implementation options be partial?
We find many concerns which are complex and invite peculiar edge cases
All these concerns leads to a conundrum the basic value proposition of Trim:
THE DEBACLE OF MODERN SSD TRIM
If the host is going to pitch-in on the chore of flash readiness, when is a good time for the OS to tell the drive to prepare? Boot time? During mainline system operation? Never?
What happens to system performance if deleting files must generate synchronous discards (effectively writes) for the deleted file's related storage extents? Normally file deletion is accomplished by updating tiny amount of meta-data, to forget the associated storage extents.
What if discards blow up the drive command queue?
What about race conditions between discards that imply a specific storage state and command syncrony? Etc...
If the OS is going to be involved in flash storage maintenance, there are more or less advantageous times for it to conduct this chore depending on workloads, which is why the OS tends to relegate Trim to a maintain point, e.g., boot.
A survey of system admin literature reveals various approaches. Different OSes have their own ways. But this is most clearly seen in the Linux command structures which divide the chore of discard between volume mount options and an administrative command called fstrim(8). See Linux references at end.
Moreover, the ATA/NVMe specs keep evolving, the drive controllers keep getting more sophisticated, and different drives may treat DSM options differently.
Can we generalize a posture towards the administration of discard?
I think so. The progress of system design is rooted in the truism that good design modularizes concerns of subsystems to hide details in order to manage complexity; so don't expose the complexity of one subsystem to another without a clear theory of dependences that justifies the exposure. When a design gets modularity wrong, it becomes awkward and confusing.
SSD Trim being a case in point. But any measure, flash readiness for I/O must be regarded as a purely internal concern of a drive. There was a time when making the host aware of flash mgmt was appropriate. This was during the early development of SSDs, when it was impractical to use a dedicated controller and FTL was very simple.
Thanks to powerful dedicated chips we now find increasing modularity across every aspect of computer systems design. But this doesn't imply seamless interoperation. Modularity may manifest in surprising ways, such as Apple sealing its devices and spreading functionality across components that are considered separate modules in the PC market. There is no contradiction: Apple's closed designs give them the power to add value through vertical integration because they hide subsystem dependencies that might be visible and therefore inappropriate for PC parts.
That SSD Trim has become a point of friction for systems administration is in keeping with the poor design cut that is moderm Trim. In the NVM Express Commandset Spec, we find that the DSM sub-commandset allows the host to signal the drive with a wide range of hints for optimizing access, including caching hints. Again these are purely advisory, with the drive maker free to no implement and meet spec. But Apple may have other thoughts which result in foibles for hackintoshers.
Yet, even if Apple is a perfect student of PC modularoty, with no internal agenda, we should expect that market pressures will result in many drive vendors creating partial DSM implementations with awkward side-effects or failure modes. And finally, even if the implementations were somehow perfect, drive vendors tailor capabilities and behavior to market segments. So the permutations are a nightmare.
If there's any single conclusion that a user might draw from all this today, I think is "avoid discard."
Now I'll explain how a system can avoid the concern of discard.
THE MAGIC OF OVER PROVISIONING
First, I want to expose a common misunderstanding aver-provisioning:
There's an idea among Windows and Linux nerds that an SSD can be usefully over-provisioned be leaving "free-space" via the partition table (e.g., GPT layout). As mentioned in the OC documentation above, there's this idea of "leaving space where the controller can find it," without any sense of wonder over how exactly the drive is informed of this "free" space. How does the drive get information about volume layout if the OS doesn't tell it? It's possible the drive could know how to read partition layout: NOT!. The drive maker will have to maintain compatibility with system SW, but we find no traits of OS per drive filesystem compatibility listed by drive makers.
So how does this work? If just having space on a drive which is rarely touched helps the controller, there's almost always space that's rarely touched, unless the drive is full. So, from the drive perspective, what's special about about one area that goes untouched because it's outside any volume, versus area that just hasn't been touched yet? There isn't any obvious distinction.
For routine PC uses, a full drive and the need for high performance are mutually exclusive: if the drive is full, then write performance and wear are a minimal concern, storage is exhausted. Running a drive past full seems like a degenerate case. User-selectable headroom, might make sense for use-case where a drive is kept close to full with a very dynamic workload, which is not typical of consumer devices.
On Windows, some drives come with optional utilities that include "over-provisioning" controls. These controls can have awareness of Windows storage layouts. In such case, the utility might use this information to program the drive controller with proprietary layout specific parameters, e.g., the Magician utility can tell the drive certain area is headroom for performance using secret attributes in drive controller. Bit what does this have to do with OpenCore hackintosh, which has no such utility?
As far as I'm concerned, SSD over-provisioning secret sauce is at best academic and mostly irrelevant.
IOW, the OS is always supplying the drive the needed information for garbage collection through normal operation. Even without discard or explicit over-provisioning, the drive constantly receives information about flash reclamation from ordinary I/O: a write implies that the previously associated flash for its LBA extent is reclaimable.
There might be a scheduling advantage to Trim, in that signaling the opportunity for reclamation ahead of writes may allow hiding of latency in flash page setup. But a similar advantage can by arranged from pipelining writes and flash maintenance: when processing writes, the controller FTL sends the data for the requested LBAs to prepped flash pages, and notes the previously associated pages can be refreshed. Writes aren't delayed by flash maintenance latency which happens in the background. A slight over-provision in the drive (which can be hidden-from the user) ensures there's always free flash to ready to receive writes even when the drive is nearly full.
My conclusion is that discard and provisioning also should be a purely internal concerns for the drive logic, where drive traits may be marketed in manner stylized and tweaked for some general purposes, much as spinning hard drives are marketed in 2023.
SOME OPENCORE HISTORY ASSOCIATED WITH SSD INCOMPATIBILITY
Here is a selection of links which capture the history of SSD compatibility with Open Core:
Dortania — Anti-Hackintosh Buyers Guide — Storage
It's difficult to make sense of all these reports because of edge cases and lack of comprehensive study.
I use a SK Hynix Platinum P41 on an Asus Z590 Hero with 11th gen and Monetery through Ventura 13.3 and it's 100% perfectly reliable and fastest our of 4 drives I have used, including 980 Pro.
SSD hall of fame needs new entries #192
SSD hall of fame needs new entries · Issue #192 · dortania/bugtracker
Guide Anti-Hackintosh Buyers Guide (link) After our discovery of a severe bug in the TRIM implementation of practically all Samsung SSDs we spent time investigating which SSDs are affected by all k...
github.com
...cont.
Acidanthera NVMeFix.kext
GitHub - acidanthera/NVMeFix
Contribute to acidanthera/NVMeFix development by creating an account on GitHub.
github.com
LINUX COMMANDS FOR TRIM ADMINSTRATION
nvme-dsm (Ubuntu) man page
Ubuntu Manpage: nvme-dsm - Send NVMe Data Set Management, return results
manpages.ubuntu.com
fstrim(8) man page
Linux fstab(8)
FURTHER READING
500 SSDs Listed by Brand, Model, Interface, Form Factor, Capacities, Controller, Cores, DRAM, HMB, NAND Brand, NAND Type, Layers, Categories, Notes, Product Page
SSDs - Google Drive
The Ultimate Guide to Apple’s Proprietary SSDs
(Higher level presentation that I would wish, but still a nice review of Mac-specific SSD tech.)
Last edited:
- Joined
- Apr 11, 2014
- Messages
- 204
- Motherboard
- Asus ProArt Z790-CREATOR WIFI
- CPU
- i9-13900KF
- Graphics
- RX 6900 XT
- Mac
Good read, thank you.
- Joined
- Mar 9, 2011
- Messages
- 107
- Motherboard
- ASRock Z370M-ITX/ac
- CPU
- I3-8350K
- Graphics
- RX 580
- Mac
- Mobile Phone
Ah, I had been looking for my post. I saw it had been moved (my bad) and accidentally (?) deleted.
@c-op-r, thanks for your responses. They were not in vain. I will start a thread in the right place and note what I found.
@c-op-r, thanks for your responses. They were not in vain. I will start a thread in the right place and note what I found.