The story of how to bring back _ and : on Debian 12 Bookworm (Download provided)

[Askarus]

The long journey of a key press

I have finally got to the root of the issue of why _ and : do not work on Bookworm.
Since we all want to learn something I did a bit of a writeup about what is going on deep behind layers of key codes and keysyms, and what was the issue and how to fix it.

Let's start with the journey of our dear electrons once a key gets pressed.

1. Key gets pressed
2. The kernel does something and invokes a keycode (see keyboard matrix, however that is not relevant here)
3. pyrainput/funkeymonkey catches that keycode and translates it to a different key code [Pyra specific]
4. xkb translates the keycode into a keysym

Why do we need pyrainput (3)?

Pyrainput does a few things

1. Hides all the input devices and creates 3 new ones (nubs, gamepad, keyboard)
   This is needed because oiginally each nub, the gamepad and the keyboard are all different input devices.
   With funkeymonkey running, the rest of the OS does not see the original input devices any more, but only the one we defined
2. Handle the easy switching between gamepad and mouse/keyboard mode for the gamepad section
3. Hide FN and assign a separate keycode for each orange(FN) key.
   That is required because with only X alone we would not be able to to CTRL+ALT+F6 to switch terminal.
   So with pyrainput, each fn key has its dedicated keycode.
   Remember, keycodes are the lowest value of the key press chain

Why do we need xkb (4)?

A keycode alone does not do much. xkb is part of X11.

• xkb has it's own version of keycodes. They are called keysyms.
• A keysym specifies what symbol a keycode is assigned to.
  KEY_1 for example is assigned to "1".
• Keycodes use a US QWERTY keyboard as base.
  Kexsyms will then define the layout according to the keyboard set in the window manager
  For example on a QWERTZ keyboard KEY_Y would be assigned to "Z"

All easy, right? So what's the problem?

The issue of missing keycodes?
Remember how above we specified that a US keyboard is being used as a base for keycodes, the lowest layer?
On a US keyboard there is no key defined for ?_+{}:".
Also remember how every orange(FN) key gets its own key code?
You see the problem?
Right, pyrainput must translate FN+D on the Pyra keyboard to a keycode for _ that does not exist in the Linux keycode definition header.

And this is the utterly stupid part here.
We have hundreds of keycodes like KEY_SOUND, KEY_VIDEOPHONE, KEY_SPREADSHEET, KEY_SPELLCHECK.
You can find the list here.
Can you imagine how there exists a dedicated keycode for anything you can and can not think of except for a hand full of ASCII codes?

How do we get out of this?

The solution to this is as easy as it sounds.
Let pyrainput translate FN+D to KEY_SOUND and let xkb translate KEY_SOUND to _.
That is exactly what we do and it works, except that it doesn't.


An other chain of events: xkb symbol layers.

So what we did was to define our oen Pyra xkb layout that translates all the keycodes we get to the keysyms we need.
That works super well for those keycodes that are defined like SF;[]=.
However when translating KEY_SOUND to _ that has no effect.

To understand that we must understand that a xkb layout consists out of layers of symbol definitions.
For our pyra that is

pc+pyra(pyra)+inet(evdev)

Do you spot the problem? inet(evdev) loads after pyra and overwrites it.
As fate wants, pretty much all of the weird non ASCII codes are defined in that inet layout.

Can we get rid of that? Unfortunately not.
To understand that, we must understand an other topic regarding xkb.
There are rules, in our case evdev, that define the order in which symbol files are being loaded.
And how fate wants once again that rule file contains the nice letters

Do not edit this file, it has been automatically generated

That is right, no easy editing of that.


How to get out of this mess?

Ther are 2 very simple solutions that work.

1. Edit the inet keysum definition file and comment out the ones we need to use for the Pyra [We use that one currently]
2. Run a command after xkb initialized its symbols
   setxkbmap -symbols pyra
   This command makes so that our pyra layout hets highest priority and loads its keysym definitions after inet

An other solution would be to test all the keycodes, check if there are some not defined in /usr/share/X11/symbols/inet and use those for _ and :.
This solution however is super tedious to do and shares the biggest disadvantage with 1 (see below).
This was the first attempt I trued and for that I improved pyrainput so that users now can set their own keycodes via a config file.
That however will be an other topic and we can fix the current issue luckily without touching pyrainput at all.

Why _ and : stopped working in Bookworm?

It is very simple, since we manually edit /usr/share/X11/symbols/inet, this is a very fragile foundation we built our house on.
That file is different between Debian versions.
The keycode we use for _ has no keysym defined in /usr/share/X11/symbols/inet. So that keycode is free to use.
In Buster and Bookworm however the keycode we use is defined in that file.
Issue in the chain and poof, ther fors your _

The fix was as simple as commenting out 2 lines in /usr/share/X11/symbols/inet and out _ and : are back again.

I do not like this fix at all, since it is very fragile and once thgat file changes, we are in trouble again.

What are better ways to handle this?

1. Define a xkb rule to load the pyra symbols after the inet symbols and fix the core of the issue
2. Issue a command later than the xkb initialization and load the pyra symbols again to overwrite the initial xkb layout

1. xkb rules

• Advantage
  We want this, no questions asked. This would be the clean way to handl the issue
• Disadvantage
  I could not get this to work.
  I spent so many hours and tried pretty much everything the internet and ChatGPT (man that was trash) suggested and it did not work.
  The issue is that we may not edit manually the evdev rules and neither editingthat nor any other solutions I tried changed the symbol load order.
  Also I have pretty much no experience in that and do not know how to debug tis properly.

2. Issue a command after xkb to load the pyra symbols

• Advantage
  - Easy to do (systemd or .bashrc)
  - Way more future proof that nwhat we have now since we do not need to edit /usr/share/X11/symbols/inet any more
• Disadvantage
  - Not as clean as 1, a bit more of a hack
  - Needs to add a systemd service
  - Obfuscates what's going on
  -> If one is not aware of that manual overwrite and prints the xkb layout, then it will be very unclear why the pyra symbols have precedence over the inet symbols and why that even works.

How to procede?

Until the dev team has decided on this matter I publish a .zip file attached to this post that executes solution (1).
The file contains an edited /usr/share/X11/symbols/pyra and /usr/share/X11/symbols/inet file as well as an install script.
Unzip and execute the installer as sudo
sudo ./install.sh
This will copy the files you need in place and comment out all the keysyms we need and bring back your _ and : on Bookworm.

Even wit hthat in place I strongly vote for solution (2) and to add a systemd service to the mix that loads the pyra symbols manually and saves us from debugging this in the future again.
also we need to make sure that this will be documented properly and easy accessible somewhere.
I spent many days debugging this, starting with pyrainput and working my way through the xkb stack.

What about the headlessterminal?

You might have noticed that in the terminal, without X11 running, many of the orange (FN) letters do not work.
It is all those letters that do not have keycodes and rely on xkb to translate them into keysyms.
so are we lost?
No

It is a bit of extra work to do, but according to the all mighty Internet, we need to use a different mechanism to get all the specia; characters to work in the terminal.
That mechanism is localectl and most sources say that it's required to define xkb for X11 and locales for the terminal.
I have not looked into it and do not know if localectl can handle our keycodes.
However this is also something in my list since some people (mainly aTc) request this.

Unfortunately there is no built in solution to handle X11 and a headless terminal with just one edit.

Let me conclude with a personal message

Spoiler

Take out your Pyra

Give it some love

Get this project rolling once again

UPDATE - 2024 - 03 - 10?

I spent quite a bit of time trying to get solution (2) to work.
So I tried to make a systemd service that reloads the pyra symbols manually after xorg got initialized.
Can this even work?
I dpecified DISPLAY=0 but still the log says that it can not find the display.

So can anyone help here?
Can systemd be used to do a setxkbmap -symbols pyra after xorg initializes?

I have manually started my service while the display was running and that did not work.
Also I have specified to run as a regular user.

 

[Linux-SWAT]

Now get the DSP to work ! :^)
Well done. Thanks for the explanations.

 

[Askarus]

Now get the DSP to work ! :^)
Well done. Thanks for the explanations.

DBPs are one of the lesser things I care about.
I will probably not use that at all, or very little at best.

After this topic I will focus on debugging LTE and why it is so bad.
We need to figure out if this is a hardware or software issue.

There are a few things open and DBPs are on my list.
However it is nothing I personally care about.

 

[spud42]

DSP not DBP, the Audio issue..Digital Signal Processor...

 

[Askarus]

DSP not DBP, the Audio issue..Digital Signal Processor...

Ah, right.
Anyways, I remember that someone said that DBPs are broken,too on Bookworm. So checking this IS on my list

I will not touch the audio system, since there are already people at it who have more experience than I do with such subjects

Believe me, there is enough for me to do to keep me occupied full time for a few months if I want.
Even if we do not account in building the physical Pyra units themself

 

[Askarus]

Update:

I spent quite a bit of time trying to get solution (2) to work.
So I tried to make a systemd service that reloads the pyra symbols manually after xorg got initialized.
Can this even work?
I dpecified DISPLAY=0 but still the log says that it can not find the display.

So can anyone help here?
Can systemd be used to do a setxkbmap -symbols pyra after xorg initializes?

I have manually started my service while the display was running and that did not work.
Also I have specified to run as a regular user.

Here is my service.
It is stored as /etc/systemd/system/pyrakeymap.service

Spoiler

[Unit]
Description=Loads the Pyra keymap manually to overwrite inet(evdev)
After=graphical.target
Requires=graphical.target

[Service]
type=oneshot
ExecStart=/usr/bin/setxkbmap -symbols pyra
RemainAfterExit=yes
Environment=DISPLAY=0
User=askarus

[Install]
WantedBy=multi-user.target

Please help if you know what I am doing wrong here and if you know if what I want to do can be achieved via systemd

I have spent quite a lot of time on this already and I do not feel like I want to go through more trial and error.
We have a working solution and if there is no easy way to system wide reload the pyra symbols after xkb initializes or to overwrite the symbol order, I will just stick with the modified ines symbols.

 

[PowerGod]

I don't know if this could make the difference, buy the examples I see for the "Environment" setting uses quotes:

Environment="FOO=foo"

This page shows an example of overriding system settings on a Fedora distro, but since uses Systemd too I suppose it could be similar:

Setting Environment Variables for systemd Services | Baeldung on Linux

Explore different ways to set environment variables for daemons managed by systemd.

www.baeldung.com

 

[pimaster]

Whenever I've been messing with the DISPLAY variable, I've had to put :0 instead of just 0.

 

[netcat]

When you login to your pyra via ssh (terminal) do : and _ work?

Linux run-level 5 is gui mode but it might be fiddly.

 

[Askarus]

Whenever I've been messing with the DISPLAY variable, I've had to put :0 instead of just 0.

I have seen that and also tried ":0"

Thanks for the hints @PowerGod and @pimaster
I will try that.

When you login to your pyra via ssh (terminal) do : and _ work?

No idea.
I'd have to try that at some point.
You might need to loadkeys us or something if your external keyboard dies not work

 

[Askarus]

I have made a minor modification to the pyra layout.
It is not required to update however it makes the file more robust.
Updated file is attached to the first post.

Changelog:

Added line:
key <POWR> { [ XF86PowerOff ] };

Reason:
This line is redundant and covered by inet already. However the power button breaks if I put setxkbmap -symbols pyra into the autostart of MATE or XFCE.
I experimented with that solution as an alternative to systemd which I still can not get to work.
Adding the power button definition to the pyra keys fixed the issue.

Updates
I made some progress with the systemd part.
I have now a service that works when I start it manually. However I could not figure out how to make it work so that it overwrites the symbols properly.

The service looks like that

Spoiler

[Unit]
Description=pyra keymap
After=graphical-session.target

[Service]
Type=oneshot
ExecStart= /usr/bin/setxkbmap -symbols pyra

[Install]
WantedBy=graphical-session.target

I put it into /etc/systemd/user/. That way it picks up the system environment properly.

After that I came up with the autostart solution mentioned earlier. With the power button fix provided earlier that works.

 

[Askarus]

Okay, finally I got bothered by this once more.

I finally found a solution to this issue that solves the problem in a clean way.
No touching of system packages, no systemd hacks or whatever.

I finally figured out how to get the xkb symbol name for any keycode I want.

This new method requires an update to pyrainput because we need to assign new keycodes.

Here is how the whole procedure looks like

# Set new keycode via pyrainput (not published yet. needs to merge my Merge Request)
- /etc/pyrainput.cfg
-PYRA_KEY_FN_APOSTROPHE = 196
- PYRA_KEY_FN_D = 197

# Reload Pyrainpyt daemon to read new keycodes
sudo systemctl reload pyrainput

# get the actual keycode
# The offset of 8 between the one set in pyrainput and the one you need for further steps
# This is not required but I better be sure
sudo xev
- press key
- remember keycode

# Get keysym name for xkb
# key <I347> for example
# If this results in multiple lines, take the one feom evdev
# For 204 and 205 it should be ALT and META
cd /usr/share/X11/xkb/keycodes/
grep -d recurse KEYCODE ./

# open /usr/share/X11/xkb/symbols/pyra
- look for 'underscore' and 'colon'
- set underscore to 'key <ALT>'
- set colon to 'key <META>'

# reload Pyra symbols
setxkbmap -symbols pyra

# Now you should have your : and _ back

 

[Askarus]

So, finally I updated pyrainput to contain the new keycodes.
This is finally a clean fix that does not modify any system files.
Only pyrainput and our Pyra keysym definitions needed to be updated.

If you already use the old fix, it is save to manually replace the files. There is a README in the .zip with instructions.
If the old fix still works for you, then you probably do not need to do any action at all. This new fix will go into Wally's testing OS and if it works it'll go into Bookworm stable pretty soon after anyways.

 

[johnicboom]

In case the info is still useful, /usr/lib/funkeymonkey/ was the right place for me. Thanks for working this out!