Help translating "quoted-printable" to emoji

[PowerGod]

Hi, I have a phone that doesn't show emoji, but as far as I tell to people to not send me those things, they still does...

On the phone itself I can only see little squares, but I can save the messages in .vmg format and look at them from the PC.

The content of a message with just emoji appears like this:

TEXT;CHARSET=UTF-8;ENCODING=QUOTED-PRINTABLE:=ED=A0=BD=ED=B8=82=ED=A0=BD=ED=B8=82

I had a look at these lists:
https://apps.timwhitlock.info/emoji/tables/unicode
https://unicode.org/emoji/charts/full-emoji-list.html

but I can't find combinations corresponding to the previous codes.

Is there some kind of translation to do first ?

What am I missing ?

Thanks

EDIT: Since I can't find emoji in my PC fonts too, I suppose I'll need to download all those images and then associate them to a code... once it is properly converted...

 

[kuru]

Korean?

=BD=ED
U+BDED
뷭

 

[Hellfleck]

those are hex codes seperated by the = sign, you can enter them here https://www.branah.com/unicode-converter under utf-8 as \xED\xA0\xBD\xED\xB8\x82\xED\xA0\xBD\xED\xB8\x82 for example

2 emojis, the board wont't let me write them so these : https://emojipedia.org/face-with-tears-of-joy/

 

[levi]

Ah, so it's two 1f602's then? Translated to UTF-8 as ed-a0-bd-ed. I guess I didn't recognise that immediately as Unicode topped out at 16 bits the last time I learned UTF-8, and that code's 17 bits long. Wikipedia's UTF-8 page doesn't seem to explain how to convert a 17-bit code with the top two bits set, but that translator seems to show that's what should happen.

 

[PowerGod]

@Hellfleck : thanks, this helped a lot

Ah, so it's two 1f602's then?

Yeah, I too didn't know how many of those codes were needed for an emoji

With my old phone I was using MyPhoneExplorer to manage SMS, but it seems unable to deal with this format.... and the programs I found on the internet to view vmg files are really basic (text only)...

At some point I will do an utility to read/manage vmg files, with support for emoticons, it will be very useful.

 

[levi]

Oddly enough, this site reckons the UTF8 for u+1f602 should be F0-9F-98-82, which also translates to \x1f602 in the translatior linked by Hellfleck. I don't really understand what's happening with UTF8 at this end of Unicode, but provided you can find a way to convert the numbers you need to unicode characters programatically, I guess you're sorted.

 

[PowerGod]

Yeah, that's the plan

 

[levi]

FWIW in python 3:

>>> "smiley".encode(encoding='utf-8')
b'\xf0\x9f\x98\x82'
>>> bytes(b'\xf0\x9f\x98\x82').decode(encoding='utf-8')
'smiley'
>>> bytes(b'\xed\xa0\xbd\xed').decode(encoding='utf-8')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xed in position 0: invalid continuation byte

In other words, python's decoder reckons the '\xed' format utf-8 string is wrong, so python's built in decoder won't help you. I've no idea if and how the 0xed and the 0xf0 encodings are related.

Edit: This board silently eats the smiley of 'crying smiley face' so I've had to replace it with the word 'smiley' in the code example up there, unfortunately. To reproduce it, when you see the word 'smiley' instead press shift+ctrl U then 1f602 and a space character to enter the smiley in a bash terminal at least.

 

[___]

there is a quopri library for python that can decode quoted printables. It's part of the standard libs: https://docs.python.org/2/library/quopri.html (python2)

 

[PowerGod]

Cool, could be useful, anyway, in the worst case I can just make a database with images associated directly to the quoted-printable strings.

I think I'll just generate HTML files with the content of the messages, so I can include images without dealing with a customized text-box object.

 

[levi]

there is a quopri library for python that can decode quoted printables. It's part of the standard libs: https://docs.python.org/2/library/quopri.html (python2)

I don't think the data's in base64 format though. If I treat those hex codes as ASCII, absolutely none of them are 8-bit clean, so can't be base64/MIME encoded or Q-encoded, which I think is what that library's for.

 

[Cloudef]

That's not valid UTF8, but UTF16 encoded as UTF8. Ironically since this mistake is so widespread, there's derivative standard where this is valid https://en.wikipedia.org/wiki/UTF-8#CESU-8

 

[levi]

I don't think it's CESU encoded. That message consists of two SMP1 characters, and is 8 bytes long. Wikipedia reckons CESU encoding of non-BMP characters gives you six bytes per character, while this is four.

If it is UTF-16 at any point, I can't work out how it's encoded. Wikipedia only has a definition for codepoints up to 0x10nnn while this is 0x1fnnn codepoint. My python gives a six byte encoding for UTF-16, but a different one to the one we're seeing in this .vmg file.

 

[Cloudef]

It's UTF16 surrogate pairs, that got encoded into UTF8, thus being 6 bytes long in total. It's exactly what CESU-8 describes.

Here is output from ftfy for example:
utf8:

 └╼ ] printf '\xED\xA0\xBD\xED\xB8\x82' | ftfy -e utf8     
ftfy error:
This input couldn't be decoded as 'utf8'. We got the following error:

    'utf-8' codec can't decode byte 0xed in position 0: invalid continuation byte

ftfy works best when its input is in a known encoding. You can use `ftfy -g`
to guess, if you're desperate. Otherwise, give the encoding name with the
`-e` option, such as `ftfy -e latin-1`.

cesu8:

 └╼ ] printf '\xED\xA0\xBD\xED\xB8\x82' | ftfy -e cesu8
(the smiley but gets stripped by this board)

 └╼ ] printf '\xED\xA0\xBD\xED\xB8\x82' | ftfy -e cesu8 | xxd  
00000000: f09f 9882                                ....

 

[levi]

Oh right, somewhere I dropped the final two bytes when I was experimenting with python. It is 6 bytes per character indeed, my fault sorry. I don't quite understand why you can't decode it as UTF-8 - shouldn't it then just degrade to its two surrogate pairs?

Perhaps it's because as wikipedia says: 'Isolated surrogate code points have no general interpretation; consequently, no character code charts or names lists are provided for this range.'

So a UTF-8 decoder errors because the surrogates when considered as individual glyphs are invalid. Tricky stuff!

 

[Cloudef]

Yeah, and this is mostly widespread because of UTF16 environments. UTF16 is considered a broken encoding and should not be used anywhere anymore. There's even a site aimed for windows developers to shed light on the UTF16 (and other encodings) issues http://utf8everywhere.org/