Handoff was working from iPhone to Mac but not vice versa. Subsequently I logged out/logged back into iCloud on both and toggled Bluetooth etc and got it working from Mac to iPhone but not vice versa! So it's worked both ways, just not at the same time!
If I open the Bluetooth settings on both devices then I can see the iPhone on the Mac's pairing list but the Mac doesn't appear on the iPhone's pairing list. Not sure what's going on there but it seems a lot of people are having issues with Bluetooth on Yosemite going by the threads on the official Apple forums.
Update: After much jiggery-pokery on both devices I think it was the combination of resetting the network settings on the iPhone (which also resets all Bluetooth settings) and re-logging into iCloud that fixed it as I now have it working bidirectionally. It still doesn't feel bulletproof given how much messing around was required to get it going on 2 devices with fairly "clean" builds but I can rest knowing that it at least works.