MicroB (N900 browser) did smooth zooming and was running on an ancient version of Gecko (equavilent to Firefox 3.something).
It's also open source IIRC, and I'm not sure how it did the smooth zooming, but it might be able to be looked at, given they used Gecko, allbeit an old version of it.