It seems like opening up the device improved the noise.
While typing this text the GPD is about 1 meter away in its dock and I almost don't hear it.
It's not annoying any more and while typing I don't hear anything.
It seems like it stays that way not as long as I don't load new web pages.

I have an other issue now.
On Windows when using the Thunderbolt dock my mouse pointer frequently freezes and then moves in some direction for a while.
After that I get back control.

I haven no idea how to fix that.
It's working fine on Linux.
It's likely I have removed/blocked some important driver while cleaning my Windows.
(Seems like Linux is the much better choice for desktop work. I'm glad I installed it.)
Maybe I'll finally have to give Discord my phone number to finally be allowed to post messages in the GPD server.