Run `xfwm` from terminal without trailing & and don't close the terminal. Watch out for any messages that will appear in the terminal. If there are any - they might be useful. Paste them here. When it segfaults, see if there is any message, then run it again in the same terminal.
Given my experience with sawfish wm, I doubt that xorg upgrade will fix that. It's possible that xfwm has some ARM related bug. I recommend running sawfish instead of xfwm. At least I will be using it, and I'll iron out any ARM bugs in sawfish.