- Joined
- Sep 14, 2011
- Messages
- 256
- Motherboard
- Z97X-UD5H rev. 1.0
- CPU
- 4690K
- Graphics
- EVGA NVIDIA GeForce GTX 650 (not Ti)
I posted on this problem before and discussed it in the chat but it's difficult to explain and I haven't seen anyone else run into it.
A task I have to run every couple of weeks involves importing about 10GB of data into MySQL and then transforming the structure through a long sequence of queries so that it can be used in a sphinx search engine. It's an SQL batch file with about 100 queries and about 40 of them take more than a minute—a few take 10 or 15. Total IO is maybe around 50GB read and write. Each query has been carefully optimized to minimize disk seeks so while the workload is heavy in filesystem IO it's actually CPU limited, not disk limited. (The task cannot exploit multicore since the data processing is serial by nature.) The DB uses MyISAM tables so it relies heavily on OS X's virtual memory for table caches. The batch job carefully allocates memory to mysqld for MyISAM key buffer.
The batch job took about 4 hours on my Late 2006 iMac. On the new hackintosh system (see .sig) it was much slower. The queries would run nice and fast for a while and then the performance would drop off horribly. The system would bog down and become very slow: web browsing was unusably slow, even typing in terminal/ssh, forget PhpStorm.
During these conditions dtrace would show that filesystem IO was ridiculously slow, the disk request queue was nearly empty but CPU was pegged on one core.
If I did a clean reboot, started the batch SQL job and left the system untouched then it would complete but slower than the old iMac. If I tried to do anything else with it after it had bogged down, it would grind to a practical halt and eventually I'd have to restart.
For all other work the system performed well except for the normal HD 3000 artefacts and system hangs in graphics engine.
I had a hunch that the slowdown was related to the system running out of free memory for new file buffers. This coincides with the moment when all the physical memory has been written to at least once and the kernel starts to recycle old file buffers starting with the least recently used.
To test the hypothesis I wrote a small program that recursively read from disk starting at a given node and looped back when all subdirs and files been read. I tried it on directories full of very large files, large files and a normal mixture of sizes.
With the large and very large files the performance crash was repeatable.
Most hackintoshes using HD 3000 do not actually allocate to frame buffers the amount of memory OS X thinks is allocated. These systems consistently experience video artefacts and hangs. I suspected this is related to my problem. Perhaps there is a conflict of some kind going on between the graphics software and the kernel over a portion of memory they both think they own, e.g. 512 - 480 - 2 MB, depending on BIOS config.
Recently I installed a video card (.sig) and disabled HD 3000.
The performance issues are now gone. The SQL batch job completes smoothly in about 2.5 hours and the system behaves very nicely throughout so I can continue normal work.
Given that HD 3000 is part of certain recommended builds, I think the founding fathers of this web site do us a favor if they could be a bit more specific about the conditions under which HD 3000 works properly. Many people have had problems with it that are quite uniform and predictable.
A task I have to run every couple of weeks involves importing about 10GB of data into MySQL and then transforming the structure through a long sequence of queries so that it can be used in a sphinx search engine. It's an SQL batch file with about 100 queries and about 40 of them take more than a minute—a few take 10 or 15. Total IO is maybe around 50GB read and write. Each query has been carefully optimized to minimize disk seeks so while the workload is heavy in filesystem IO it's actually CPU limited, not disk limited. (The task cannot exploit multicore since the data processing is serial by nature.) The DB uses MyISAM tables so it relies heavily on OS X's virtual memory for table caches. The batch job carefully allocates memory to mysqld for MyISAM key buffer.
The batch job took about 4 hours on my Late 2006 iMac. On the new hackintosh system (see .sig) it was much slower. The queries would run nice and fast for a while and then the performance would drop off horribly. The system would bog down and become very slow: web browsing was unusably slow, even typing in terminal/ssh, forget PhpStorm.
During these conditions dtrace would show that filesystem IO was ridiculously slow, the disk request queue was nearly empty but CPU was pegged on one core.
If I did a clean reboot, started the batch SQL job and left the system untouched then it would complete but slower than the old iMac. If I tried to do anything else with it after it had bogged down, it would grind to a practical halt and eventually I'd have to restart.
For all other work the system performed well except for the normal HD 3000 artefacts and system hangs in graphics engine.
I had a hunch that the slowdown was related to the system running out of free memory for new file buffers. This coincides with the moment when all the physical memory has been written to at least once and the kernel starts to recycle old file buffers starting with the least recently used.
To test the hypothesis I wrote a small program that recursively read from disk starting at a given node and looped back when all subdirs and files been read. I tried it on directories full of very large files, large files and a normal mixture of sizes.
With the large and very large files the performance crash was repeatable.
Most hackintoshes using HD 3000 do not actually allocate to frame buffers the amount of memory OS X thinks is allocated. These systems consistently experience video artefacts and hangs. I suspected this is related to my problem. Perhaps there is a conflict of some kind going on between the graphics software and the kernel over a portion of memory they both think they own, e.g. 512 - 480 - 2 MB, depending on BIOS config.
Recently I installed a video card (.sig) and disabled HD 3000.
The performance issues are now gone. The SQL batch job completes smoothly in about 2.5 hours and the system behaves very nicely throughout so I can continue normal work.
Given that HD 3000 is part of certain recommended builds, I think the founding fathers of this web site do us a favor if they could be a bit more specific about the conditions under which HD 3000 works properly. Many people have had problems with it that are quite uniform and predictable.
