I don't personally understand the AltGr issue for some people, but then it's only recently I've had a keyboard with such a key labelled on it, and as far as I can tell it's not mapped to anything on my current key mappings, so it's safe to say I've never used it.
But going by what wikepedia says, it's designed to enable you to type the extra symbols which are printed on the keyboard. So it suggests to me that as long as things are printed there, that's how you get them. And since most of the keyboard layouts have symbols in different shifted and non-shifted positions to any common full-sized layout, if I can cope with those in unusual places I'd suggest users of AltGr symbols can also learn.
But as I say, I've never learned or needed to learn to use it, living in an anglophone country (I use the compose key on the rare occasions I need to type a foreign word), so perhaps my reasoning is flawed.