It's sadly not open-source, nor is it based off of any portal/raycasting engine I've ever come across.
Your best hope would be to attempt to reverse engineer the game from the datafiles and a disassembly of the game executable, something which is not a lot of fun and takes entirely too much time to justify.
Asking LucasArts to liberate the source would be a good idea, though I seriously doubt they'll follow through on it.