I'm not expert in the field, but from what I know main fear regarding GC is it's unpredictability. That is, if your app does a lot of voodoo with memory (many creations/destructions), GC might decide to clean it up at any point during app's operation which might result in noticeable stall for a fraction of a second. This might be annoying in a game and quite hard to track if you are unaware about it. GC might be called manually to make it predictable easy - so in practice it should rarely (never) be an issue. Also IMHO there is a myth regarding GC from master programmers who are good enough to manage memory themselves and any automatic memory management simply is a disadvantage (and potential risk) in their eyes.
I'm not too worried about it myself. If you are experienced C++ developer, you know there are plenty of possible hard to track memory issues in C++. Safety of C#, especially for smaller projects, looks very attractive to me.