I haven't checked the code yet, might be fun to do a pass on it.
there's a trick I used a long time ago on GBA, you can actually read 16bit values as unaligned 32bit values:
ldr value, [palette, pixel, lsl #1]
its "wrong" but it works, I tried it out on Pandora, it still works...