Interesting.. I gave up on this after couple days of troubleshooting, without any luck.
Today I tried again, making the USB, I didn't change any BIOS settings from before, but this time. I was just simply able to boot straight to the installer, no boot-flags or anything, just installed.. then...