Symbols are on two modifers anyhow, you are just not lumping thirdlevelshift and alternate graphics in with Fn. Meaning you can actually do thirdlevelshift, and alternate graphics, like on a real keyboard.
And yes, with Ctrl Alt and Shift AltGr on shoulders you are getting both shift type modifiers.
R alt is something only found on US keyboards, putting symbols that belong on other buttons on letters has just as little to do with that as anything else.
There are many ways to technically have more Alts and AltGrs. L and R alt is two, out of two needed alts, but there was two of that anyhow.
With the keycode being different for the two AltGrs you could pass the same Thirdlevelshift and hope, which is not as good.
The best way however is to set it up with something entirely different and unencumbered like BTN_MODE 0x13c. Which all things considered is far less of a hack than having two Fns, that mostly do things Fn should not do, and where the things it should, doesn't work, along with a number of other things that are problematic and limiting.
And... Sorry to be blunt, but you have just shown, again, that despite using an AltGr based keyboard in your daily life, you don't actually understand how it works.
I need a keyboard map for use in this example. The statements, however, should be applicable to any of the keyboard maps made.
http://www.keyboard-layout-editor.com/#/layouts/b766912420e62d3618bb682251e50f5f
Any AltGr keyboard requires a given number of base keys. Think the unshifted-unFn top level layer of the keyboard in front of you now - regardless of which kind or flavor it is. Those are the base keys. Press the 4 key and it receives 4. For the purposes of this discussion, 4 is the base key.
Shift does not modify the base keys - those are still present, but the computer receives two signals to 'use' them. The base key Rshift is held down and the base key 4 is pressed which signals as $.
Fn works somewhat differently. Instead of returning a modifier of Fn is being held and 4 was pressed, Fn actually tells the computer, "I have two base keyboard maps and I am activating the second set of base keys." With Fn enabled on the above (or any) layout, the Fn layer symbol is returned as a raw base level symbol - in this case it is €. Not, "Fn is being held and 4 was pressed', but € as if € was a base letter all it's own.
In this way, Fn expands (doubles) the number of 'base' keys present on the keyboard by creating a 2nd 'Fn layer' virtual keyboard.
AltGr works just like shift. It -requires- base keys in the keyboard definition to all exist in order to have a base key to modify. The computer doesn't see the base key, but sees that 'I am in AltGr active mode and AltGr + 4 is pressed.' If the base key that AltGr is expecting is on the Fn layer, that is OK as the Fn layer is seen as a different keyboard.
The Pyra does not have enough physical keys to give a complete base key set for any standard keyboard definition. AltGr simply will not have enough base targets to function without a Fn layer/logic present.
To make things a step more difficult, all of the broken symbol pairs create a set of quasi-base keys that aren't actually base keys - they're something of our own invention. If the keyboard definition used in Debian is US English, the AltGr version of that keyboard would be US International. US International under AltGr expects it to look just like this - with ALL of those base keys present AND no additional base keys beyond those:
http://www.starr.net/is/type/intlchart.html
Remember, Fn creates a 2nd logical set of base keys. This is VERY useful if you want to use AltGr mode on a Pyra, because the Pyra itself does not have enough physical keys for all of the base keys needed. It MUST use a Fn layer keyboard logic in order for those additional 'base keys' to exist.
Now, I also mentioned symbol pairs above. Symbols mapped OFF of their base, i.e. giving { it's own key on the Fn layer instead of it being shift+[ results in a base key being present that AltGr won't recognize as one of it's modifiable bases. It may not be a big deal that AltGr+Fn+{ does nothing. But that does not escape the reality that AltGr+Shift+key IS a very true reality.
Look at the US International standard keyboard again. Look closely at the 4 key.
http://www.starr.net/is/type/intlchart.html
You will notice that it has 4 markings on the US International Standard AltGr layout.
normal 4 = 4
Shift + 4 = $
AltGr + 4 = ¤
AltGr+Shift+4 = £
That type of usage of two modifiers to create 4 symbol outputs per key is going to look very familiar to many of you. Every additional modifier key doubles the number of results a given non-modifier key can return. In the case of the Pyra, each key can produce values with any combination of held presses of LShift, Rshift, LCtrl, RCtrl, LAlt, RAlt(AltGr), LFn, RFn, LSuper, RSuper(LidSwitch). That means every non modifier key has 1 * 2 * 2 * 2 * 2 * 2 * 2 * 2 * 2 * 2 * 2 = 2^10 = 1024 potential logical combinations/outcomes PER non modifier key.
So, to summarize, AltGr ONLY exists when it is activated. When AltGr is activated, the RIGHT Alt key changes definition to AltGr. AltGr works JUST LIKE SHIFT does and the computer sees a series of co-events during it's use. "AltGr is being held and 4 is being pressed, that resolves to ¤." Additionally, it is true that if, "AltGr is being held AND Shift is being held and 4 is being pressed, that resolves to £."
Fn works entirely differently from AltGr and Allows a full 2nd keyboard 'base map' (Hardware based or software driver based) which Shift and AltGr can further use as a base map to give conditional output. Fn simply doubles the number of non-modifier base keys available.
I hope that helps your understanding of how and why Fn and AltGr are not only different but how each necessarily must function differently from the other.
AltGr and Shift both require a base key to operate on. Fn creates additional base keys.