The framebuffer and the 3d-renderer are separate pieces of hardware. TI has written an open-source KMS-driver for the framebuffer (and got it included in linux 3.3's staging-area, but without any support for using the SGX...), and the SGX-driver is a user-space library, that doesn't require X to be used, though it does need the framebuffer-driver to send it commands. With the KMS-driver, wayland should work, though it won't have 3D out-of-the-box, but that should be sorted out by the time Wayland becomes useful.