I never ever use safari. It is not even on my desktop, but I get the freezes, so Safari is very unlikelly the root of the problem, since I never launch it.
My freezes usually happen when I'm doing something on the trackpad more actively. (either on finder, double swiping on launchpad, browsing on chrome, etc...) So if you're browsing a lot that may make sense that it crashes more on safari.
I may plug a mouse and start using it instead of the trackpad to see if I can get it to crash.
Anyone had a freeze while using a mouse?