It seems there is a couple of resolutions for this problem. To me, it's a driver issue, and it'd be the best to resolve it within the driver. For example, VoodooHDA does not have this problem, yet it has other problem in my case, though...
AppleALC 1.4.3 has a fix (workaround actually to disable HDMI/DP audio to avoid the kernel panic), but there isn't a release yet. So I have built it manually with the HDMI/DP audio being disabled always. For this, please find the attached kext. In my case, this has addressed the kernel panic problem.
FYI, I have attached the code changes I have made based on the 1.4.3 code base. Also, I am asking the AppleALC maintainer to use a boot argument so that it'd be easier to use it.