Yes, the Xbox is made of PC parts, yes the controller is just a USB device. Connect red to red, black to black, green to green, white to white, and leave yellow alone. Done.
I think that Linux may be able to use this as a standard HID by now, as far as Linux is concerned it is just a weird...