Crashes always preceded by kernel[0]: zalloc did gc

[wonkywonky]

Kernel panics triggered by zalloc unable to alloc memory - kernel[0]: zalloc did gc

My Mountain Lion file server build has been rock solid (30d+ uptime easy between scheduled reboots) till recently (about 3 weeks ago?), when it started freezing randomly.

Servers logs indicate that crashes are always preceded by zalloc doing garbage collection. I'm curious if anyone else is encountering this issue; a few isolated cases came up in a Google search, and it

My server runs a H67 board w/ i5-2500K, 8GB RAM, an ATTO H308 HBA, ASM1061 mPCIE-SATA adapter, and a bunch of disks in various ZFS configurations (using ZEVO Community Edition).

The only thing I can think of that I changed recently that may have coincided with the start of these problems was the addition of the ASM1061 mPCIE-SATA board. However, I believe I've encountered the same crash after removing the adapter. (I say "I believe" because I haven't been able to dedicate the time it takes to do proper, rigorous testing).

I've tested and swapped out RAM and swapped out the H308 HBA for a spare. I've been using ZEVO CE v1.1.1 on my desktop and file server for many many months with zero issues.

Any thoughts?

 

[ppohio]

I am having the same problem with my machine, different hardware. Mine is a pretty vanilla build CustoMac Pro from July.

Recently started getting the hangs about every 24 hours. I have had a crazy work schedule so I have jus thad to work through it, but will try to see if I can figure anything out this weekend.

Based on this old thread at InsanelyMac it seems to be related to Power Management.

Edit: The only thing that I can think of that I changed since building the machine was connecting my UPS to it via USB. I unplugged it to see if it will make any difference, but I doubt it. My most recent hangs have come in the middle of the night when the machine is idle, and they are just shy of 48 hours apart. Previous Console message was 90 minutes before I get the kernel[0]: zalloc did gc

 

[wonkywonky]

I moved the system to an ASRock Z77E-ITX + i3-2105 to rule out issues with the Realtek 8111E, ASM1061 PCIE->SATA adapter, motherboard hardware. Using samisnake's patched BIOS, native power management enabled and using MacPro3,1 system definition.

Crashed again after 2 days of uptime.

Checking my server logs, I see that memory usage by kernel_task keeps increasing until finally kernel memory allocation fails and the KP occurs. It sure looks like there's a leak somewhere.

 

[ppohio]

I think you've found the problem, wonky...

I am seeing the same thing on my system. Rebooted last night and 6 hours later of just idling, the machine goes from using 700mb in kernel_task at boot to 2gb in the morning. I have nearly identical software on my Macbook Pro, I am letting that just sit for a while after reboot to see what kernel_task does on that machine for a comparison. But looking at my history in iStat menus, my pattern looks to be about the same as yours, it just builds up until it crashes.

What is some other information that might be helpful to tracking this leak down? I am running such a vanilla machine that I am at a loss for what could be causing the issue, because I would think that everyone who built a CustoMac from the last three months would be experiencing the same problem, but I guess since they aren't the problem could be traced to a rogue/improper kernel extension?

 

[wonkywonky]

More historical data:

From my server:

From my desktop:

For those reading who are not using iStat, blue/red/grey = wired/active/inactive.

Incidentally, my desktop experienced this problem for the first time yesterday: I noticed that kernel_task was holding on to 6+GB of memory at one point and my desktop finally KPed around 7GB allocated.

Looking at the 30 and 90 day logs for both machines, it sure seems like something changed about a month ago.

 

[wonkywonky]

I think you've found the problem, wonky...
What is some other information that might be helpful to tracking this leak down? I am running such a vanilla machine that I am at a loss for what could be causing the issue, because I would think that everyone who built a CustoMac from the last three months would be experiencing the same problem, but I guess since they aren't the problem could be traced to a rogue/improper kernel extension?

Like you, I don't really have the time to spend on this right now. As a stopgap measure, I'm scheduling reboots every 48h on my server. Oh and I tried to give myself more breathing room by replacing the 8GB stick I removed from it to bring the total onboard RAM to 16GB. On the desktop, I keep an eye out for kernel_task.

We could start comparing hardware + kexts + applications installed and see if something pops out. In particular, do you remember adding anything new about a month ago? Maybe it was the move from 10.8.3 to 10.8.4...?

 

[ppohio]

Looks like I am on the 48 hour reboot cycle as well for the time being! Frustrating but we'll figure it out.

Pretty sure it wasn't a move from 10.8.3 to 10.8.4 because I have only been running a Hackintosh on 10.8.4. I also built a nearly identical system for a friend of mine a couple weeks before this and he isn't experiencing any of the same problems.

My Macbook Pro today never went above 600mb in kernel_task, while my Hackintosh is now at nearly 5gb after less than 24 hours uptime. Brutal.

Below is my multibeast setup, might be a good place to start for comparison:

 

[ppohio]

Hey Wonky -

Just a heads up, last night I booted off a drive running my initial install, basically just OS X post multi beast, almost no other software except for Chrome and iStat Menus. The machine idled overnight and most of the day today while I was in meetings, and kernel_task was stable at about a gigabyte (my Macbook Pro has 8 gigs of ram compared to 16 on this machine, and the MBP uses ~550mb ram and is stable around that number, so a gig sounds about right). So it does seem that the problem was introduced after the initial setup with Multibeast.

So my next step was to compare .kexts and see what was different on my usual boot system that is leaking. Here's what I have, maybe you can compare when you have a chance and see if anything matches up, but after googling a bit I am less optimistic about it being one of these.

ATTOiSCSI.kext <- I think this is from the Drobo
AvidDX.kext <- Avid specific, and I have run Avid on other systems without this issue
DROBOTTBT.kext <- Drobo specific and I set up my Drobo after I started having crashes
LittleSnitch.kext <- Pretty sure it isn't this or tons of folks would have this problem
PACESupportFamily.kext <- I think this is Avid related for the licensing
Sentinel.kext <- Pretty sure this is Avid related too
TrustedDATASCSIDriver.kext <- I believe this is Drobo based on Googling

 

[sman591]

I'm running in to the same issues as well. So far it's happened two times, both after leaving my hackintosh on for multiple days at a time and idling in the middle of the night before getting "zalloc did gc" and a bunch of other related console messages
It also takes out my entire ethernet network when it happens..... really gotta find out what causing it

I've got almost identical hardware as ppohio, with the GA-Z77x-UD5H, i7 3770k, and GTX 660SC

 

[ppohio]

Sman! Welcome to the thread, glad to have you.


Can you check for the ballooning mem usage in kernel_task throughout the day? Wonky and I are using iStat Menus but you can check it in Activity Monitor as well (under System).


Last night I again booted off my backup clean install of the OS and checked memory usage - not only did kernel_task stay more or less constant around a gig, I also noticed that it actually went DOWN a bit (could dozen megs) instead of just constantly rising. To note, on both setups it starts around ~756mb, then rises to around a gig. On my problem setup, it just keeps going until it KPs. On my other setup it just stays there.


We should probably also start comparing applications - I am working today but tomorrow I am going to try to find a couple hours to go through the software I've installed and see if I can isolate anything.