Some more thoughts about that:
The OMAP5432 can get really warm, as we already know. Sadly, we don't know the TDP value (which basically says how hot it gets).
The RK3568 is the SoC used in the Anbernic RG503. It doesn't have ANY cooling. Not even a small heatpad. It just sits on the board, directly on top of the battery.
So we can assume that the RK3568 doesn't run hot. It might not be a huge speed increase compared to the OMAP5432, but it's a lot more efficient and cooler, which is a huge step forward.
Now, the RK3588 is not widely used yet. I don't know anything about the Linux Support or how hot it runs, but I still want to take it into consideration.
About the price:
The RK3568 currently is about 25 EUR.
The OMAP5432 is about 50 EUR.
The RK3588 is about 120 EUR.
Of course, that might all change if the global parts shortage ends at some time (and it looks like it 's getting better end of 2022 already).
The RK3568 has a TDP of 5W, the RK3588 6W.
We don't know about the OMAP5, but the OMAP3 used in the Pandora was 0.68W
So it needs a bit of testing, but seeing the RK3568 being used without any heatsink at all in a commercially available device, running Linux as well is a good sign.
At that price point, I think the RK3568 is looking really good. When considering the dimensions as well, I think the RK3588 is just too big. Not exactly sure which package you went with for the OMAP5, but looks like it'd be 17mm, which means the RK3568 is worth trying, RK3588 is probably much too big.
I wouldn't recommend using Anbernic's lack of cooling as a sign that this chip can be used fine without any cooling. These kinds of budget devices typically cut some corners in this domain and that results in decreased performance. Either the chip is getting too hot and throttling, or they just limit the chip in software as its cheaper than finding an appropriate cooling solution.
In terms of some realities of heat dissipation with regards to RK3588 and RK3568, the RK3568 will definitely be easier to keep cool. However, it is easier to keep cool because it has no high performance cores, so we take a hit there (the A15's in the OMAP5 are previous generation high performance cores). The RK3568 has proven to perform quite well for the general purpose tasks, but its going to be making more use of all those cores in order to do it, since each one is less performant and, importantly, does not support out-of-order execution. I would expect
slightly better general purpose performance than the OMAP5 and roughly the same or maybe even a little bit worse for emulation. No reasonable PS2 emulation on this SOC, but up to it should be ok.
The RK3588 will perform significantly better, up to Gamecube (it can even do some Wii and Switch emulation). However, heat will be a serious challenge with this. In reality, in order to deal with this we would probably have to leave off most of the A76 cores and probably just run with 1, maaaaybe 2. At full blast (so when doing GC, Wii, Switch) this chip actually will need more like 10W with all cores on. I also have some concerns regarding the current power supply capabilities of the existing circuit, especially surrounding the battery charger in early boot phases. We're barely able to get the OMAP5 going due to limitations in early boot with the input current limit on the BQ charger chip. We need the SOC to be running in a limited power state early on in order to open up the gates and allow more current in, which the RK3588 may need too much to get kicked off for this. ARMv8's secure boot sequence introduces some limitations here which could require us to make changes to the main board to get this to work. The RK3568 also poses risk for this, but the risk is less.
Then of course there is the issue of OS. These days, mobile cores for higher performance graphics are really targeted at Android. In particular, these chips we're discussing are targeted at EdgeAI purposes on Linux. As such, the acceleration libraries provided by RK (this is almost always closed-source) reflect this. Android has kind of taken over as the target platform for emulation, and so the emulators themselves are better optimized for this platform. Similarly, manufacturers of chips such as these focus their efforts on optimizing their graphics acceleration libraries for Android and spend less time on the Linux versions. One may argue that Android runs the Linux kernel, so it should be the same, but it is not. There are separate OpenGL/Vulkan userspace libraries for Linux and Android as Android has specific requirements and additional features which need to be supported. You can see this in particular with the RK3588, which can handle emulation
very well under Android, but actually quite poorly under Linux (see the end of the video I posted earlier). This means that regardless of which chip we use, we really must begin considering Android as a platform when targeting emulation. Linux is clearly much better for general purpose behavior, but for emulation and web browsing (which both heavily depend on 2D and 3D acceleration) Android is likely to have significantly better performance.
Also, it doesn't really seem to me that we would make good use of any of the features of the RK3568 over the RK3566, so maybe that is a better target as its cheaper.