What do you mean by this exactly? I removed the Displays.prefPane file and all that happened after a reboot was I had no preference pane for displays lol.
That was not what you wanted to remove as you found out. It is the file /Library/Preferences/com.apple.windowserver.plist.
How do you switch ? Make sure to add nvda_drv=1 to yout boot arguments.