MIME types

[sindbad]

(I doubt they use libpmagic live; magic is not really slow, but its never been really fast either; most, including KDE, just go by filename IIRC. But I coudl be wrong, as these things change.. they mght have made a backfiller, that goes by name and for unknown types then goes in backgrond to magic the files and see. But we digress even too much for us
KDE doesn't go for extension first. I think it uses magic first and caches all results, but I'm not sure.

Anyway, a PXML builder would probably be of dubious usefulness, but a PND builder might even be useful for mac/linux devs. Especially if it's made with Qt and is native everywhere (including gnome and mac).

 

[dflemstr]

Hmm, I might actually make a Qt version of this (if noone else does before me ), since a PXML writer has to be written sometime.
I haven't actually done any Qt before (sue me), but I've heard that it's dead simple and with the signal-socket system that it has, this could be done in a day. Oh well, I have to find time for it first...

BTW: MIME is a full standard (full list of registered types: www.iana.org/assignments/media-types/) and is being used in GNOME and XFCE aswell. It's actually news to me that any linux tool goes by file extension; never seen such a tool and used it. Well, you learn new things every day

 

[lulzfish]

" I might actually make a Qt version of this (if noone else does before me wink.gif )"

Well, I'm not really planning to do anything else during Computer Organization class.
I might as well try.

 

[dflemstr]

" I might actually make a Qt version of this (if noone else does before me wink.gif )"

Well, I'm not really planning to do anything else during Computer Organization class.
I might as well try.

Sure
Just try to, if possible, use a separate class/header/whatever for the writing and to use TinyXML for it; then the code can be reused

It's not that writing a writer for the new format is difficult, I'm just lazy at the moment, and kinda busy... Otherwise I would have done it in the libpnd patch.

 

[Squidge]

Right, but why would you store raw linux executables on a Mac or Windows machine? PNDs, yes, but ELF-binaries?

You wouldn't (unless of course your a developer), but it's always handy to know that when you are listing a directory, that you know which are executable. Since the 'x' bit and mime types are not portable, I think we should use something that is. The only thing we have however is the filename, so the filename should indicate this to us (Unless of course the file is in a directory that is exclusively for executable files, such as /bin or /usr/bin, for example, then it can be assumed.)

We should however, imho, follow the linux (or rather GNU) convention and go by MIME types primarily, since the Pandora uses a linux-based OS.

Sure it does, but there's a big difference here. One Linux OS is running on a full blown PC, and the other is running on a portable device which uses nand & sd cards. Since mimetype has to open every file to determine its type, how much longer will this take on nand/sd cards compared to a typical harddisk? How long would it take if you mounted the Pandora over nfs to another linux machine over Wifi?

I think in the end it comes down to speed and performance. Sure, we could open each file to determine the contents, but I don't think we should need to do that when we can have all the information we need from just the filename. It just seems like a "Nice idea, but not an absolute necessity" option to me. Kinda what Microsoft have done with Vista, and what they continue to do with it's successor - sure, it can look fancy fading your windows in and out when you minimize/maximize them, but I'd rather it didn't so I can't get on with what I want to do faster. They have the same annoying feature with fading menus and various other bits.

 

[Kramy]

Pick a file extension and use it. There's no reason not to. Following Linux conventions is nice, except that in this case it doesn't make sense.

It took me a little while to realize Linux didn't have an executable extension. When I first got my GP2X, I was confused by that. I had no clue how to find info on commands/executables available.

Eventually I stumbled across "man ls" on a webpage, and figured out the rest from there.

 

[streak]

do we really need to make things fancy? What for? Somehow extension, solving any problem with executable elf files. I say NO for MIME.

Why some orthodox linux users trying to complicate some things that should be easy as damn hammer . If you want do add some "geekness" to OS, thats allright, but dont force other people to use it [just make you-customized distro and use it ]. Many people wants simplicity, like I and [ i think ] Squidge. I dont wanna mess with linux commands. OpenPandora isnt desktop PC, even if panda have own keyboard. Main aim for pandora devs should be making it as easy as it should be and newbie frendly, not as geekest as it could be... :/

Remember that a lot of people will never use linux, because linux users wants to make it elite-geek thing.

 

[Tor]

It took me a little while to realize Linux didn't have an executable extension. When I first got my GP2X, I was confused by that. I had no clue how to find info on commands/executables available.

Eventually I stumbled across "man ls" on a webpage, and figured out the rest from there.

Just press a letter or two, then hit <tab> a couple of times and bash will list all commands starting with that/those letter(s). With tcsh set up correctly you only have to hit <tab> once, but with any of those shells you don't really have to 'ls /usr/bin' (it could be a lot of work if you $PATH contains a dozen directories) to find the actual name of the command you only remember was 'b'-something.

 

[dflemstr]

do we really need to make things fancy? What for? Somehow extension, solving any problem with executable elf files. I say NO for MIME.

Well, I actually find the windows way of doing things quite confusing. I change the filename from "somethingorother.jpg" to "somethingorother.txt" and BAM, it becomes a text file all of a sudden?
The contents are all the same, why shouldn't it show as a JPG file, still?

I maybe make this sound 'geeky' or 'complicated' when it really isn't. I only want an executable file to show as an executable file even if it has a strange name, that's it.

And sure, I have nothing against introducing an extension; however, a linux executable file should still be recognized as one, even if it hasn't got the ".pex" or something else as extension. If I happen to rename all my files to some other extension by accident, it should still be possible to open them.
And if some dev forgets to follow the extension convention, as is usual in the Linux world, I'd like to still be able to execute it.
Sorry, but that's only my opinion.

Anyways, can someone make a new thread about this? I thought that using MIME-types would be the obvious way to do things, since that's the way everyone in the OSS world does things by now, but this apparaently isn't the case.
I would be more than happy to discuss this in another thread than the "Pxml maker" thread, however!

 

[streak]

do we really need to make things fancy? What for? Somehow extension, solving any problem with executable elf files. I say NO for MIME.

Well, I actually find the windows way of doing things quite confusing. I change the filename from "somethingorother.jpg" to "somethingorother.txt" and BAM, it becomes a text file all of a sudden?
The contents are all the same, why shouldn't it show as a JPG file, still?

I maybe make this sound 'geeky' or 'complicated' when it really isn't. I only want an executable file to show as an executable file even if it has a strange name, that's it.

And sure, I have nothing against introducing an extension; however, a linux executable file should still be recognized as one, even if it hasn't got the ".pex" or something else as extension. If I happen to rename all my files to some other extension by accident, it should still be possible to open them.
And if some dev forgets to follow the extension convention, as is usual in the Linux world, I'd like to still be able to execute it.
Sorry, but that's only my opinion.

Anyways, can someone make a new thread about this? I thought that using MIME-types would be the obvious way to do things, since that's the way everyone in the OSS world does things by now, but this apparaently isn't the case.

I would be more than happy to discuss this in another thread than the "Pxml maker" thread, however!

We still could mix these 2 things together, aswell: MIME types checking + Extension, but i still have a felling that MIME will slow system performance, and this is the last thing i would like to see on pandora: snail-speed OS

BTW. If we speak about mime recognition things, only known files will be recognised by system , what will happen if inside of file will hide some new zip-archive , but your system havent got it in the "database", for example "file1111" with KGB format inside [KGB Archiver], system will treat this format as "data", and if you have an extension it will be easy to trace the extension in internet. Or even if you know the extension , it will not be mistery for you whats inside the file... and finally how MIME recognition helps in this situation?

 

[Squidge]

Well, I actually find the windows way of doing things quite confusing. I change the filename from "somethingorother.jpg" to "somethingorother.txt" and BAM, it becomes a text file all of a sudden?
The contents are all the same, why shouldn't it show as a JPG file, still?
More to the point, why would you ever want to do that? If a file was a jpeg, and was marked as such (by a file extension, for example), then it should never change. To convert it to another file format, load that file into a converter program, and write out the new format.
QUOTE

I maybe make this sound 'geeky' or 'complicated' when it really isn't. I only want an executable file to show as an executable file even if it has a strange name, that's it.
That's no problem. We can just check the 'x' bit. It's not what you said before however.

Anyways, can someone make a new thread about this?

Done!

 

[dflemstr]

We still could mix these 2 things together, aswell: MIME types checking + Extension, but i still have a felling that MIME will slow system performance, and this is the last thing i would like to see on pandora: snail-speed OS

Linux uses MIME for everything, and Linux certainly isn't slow.
Or I should say: GNU/Linux, since Linux isn't an OS.

But sure, sounds sensible. This is what KDE does, as I said before.

BTW. If we speak about mime recognition things, only known files will be recognised by system , what will happen if inside of file will hide some new zip-archive , but your system havent got it in the "database", for example "file1111" with KGB format inside [KGB Archiver], system will treat this format as "data", and if you have an extension it will be easy to trace the extension in internet. Or even if you know the extension , it will not be mistery for you whats inside the file... and finally how MIME recognition helps in this situation?

MIME types can be added just like extension associations. The common database for types is quite up-to-date, I haven't yet seen a file with an "unknown" mime type (which means that the system didn't understand it), and I use really weird file types.

And, a problem with extensions is that they don't tell you, either, what type of file the file really is. For example, what is a ".bin"-file? A ".raw"-file? etc. Could be anything, a binary, an executable, a raw image, a screendump, a GPU RAM dump or whatever. Thus, I don't really see your argument.

--------------------------------------
Now: I won't be arguing for MIME types any more (clarification: by arguing myself; I will respond to other's arguemnts, however), since it isn't that vitally important to me; I just wanted to make it clear that extensions aren't the norm in the Linux world.
Have you ever seen a configuration file that ended with ".txt" in Linux? I haven't. Different situations need different naming conventions, and files shouldn't be forced to have a specific suffix. It can help, yes, but it shouldn't be forced.
The x-bit has it's use, but only for executables. The x-bit's purpose, though, isn't to mark something as executable, but to protect executables from being run before first getting +x.

MIME types have been used for ages in the Linux world, and trust me, there have been fierce discussions about them in the past. MIME types turned out to be the better alternative in most cases, however.

Anyways, if someone really wants to discuss this in depth, maybe thereby starting a flame war, then do so; I will participate until it becomes too cumbersome to constantly post arguments and/or until it simply becomes too unpleasant to carry on.


Interesting fact (as maybe my final words):
The new PackageKit technology, currently integrated in Fedora (and of course Gentoo etc, for those who are into that sorta thing), has the capabilities to automatically download and install an application for an unknown MIME type. Just double-click on the file, and if none of your programs are able to open it, a new one is installed that can.
That wouldn't work with extensions since they are ambiguous.

 

[Squidge]

Hey, I'm not shooting down your mime types idea for Pandora. There are many great things you can use it for.

The only thing I disagree with you on is that we dump extensions for mime types. If you are happy to have both, then thats great. Of course, any individual developer is free to do what they like - there data files could be .dat, .bin, .raw etc or extensionless, I really don't care, and I doub't other people care much either.

However, executables should be clearly marked as such so that OSs that don't have access to the 'x' bit and don't want to drag the file down to determine the type from the contents can easily do so. The same can be said about other files that people expect others to use. For example, if someone uploads loads of icons to the Pandora file archive, the files should have extensions to easy state the type of contents across every OS. Some might not need it, but it would be a benefit for those that do. If you include a jpeg with your game however, feel free to leave off the extension, as its then a datafile that belongs to your app, rather than something you intend other to use directly.

BTW, could you extend your "Linux uses mime types for everything" comment ? I thought it was more of a KDE thing than a Linux thing ? I know Linux will run an executable regardless of file extension, but thats more of a casual check of the file header ("does first line contain '#!'" and "are the first four bytes 7F 45 4C 46) than a mime type as such. If I said a file that started with "sausages" was the mime type 'executable', I doubt it would still try to execute it, as it would have no parser.

 

[sindbad]

BTW, could you extend your "Linux uses mime types for everything" comment ? I thought it was more of a KDE thing than a Linux thing ? I know Linux will run an executable regardless of file extension, but thats more of a casual check of the file header ("does first line contain '#!'" and "are the first four bytes 7F 45 4C 46) than a mime type as such. If I said a file that started with "sausages" was the mime type 'executable', I doubt it would still try to execute it, as it would have no parser.
At least all major desktop environments decide after libmagic first, extension second. Many other tools do the same thing, so it's the de-facto standard.

 

[Kramy]

It took me a little while to realize Linux didn't have an executable extension. When I first got my GP2X, I was confused by that. I had no clue how to find info on commands/executables available.

Eventually I stumbled across "man ls" on a webpage, and figured out the rest from there.

Just press a letter or two, then hit <tab> a couple of times and bash will list all commands starting with that/those letter(s). With tcsh set up correctly you only have to hit <tab> once, but with any of those shells you don't really have to 'ls /usr/bin' (it could be a lot of work if you $PATH contains a dozen directories) to find the actual name of the command you only remember was 'b'-something.

My GP2X appears to be missing the tab key.

You (like many others) appear to be assuming that because we're going to be using a linux handheld, we also have a linux desktop.

But for a lot of us (probably over 20%) this isn't the case.

I have since tried out Ubuntu, so now I know a few bash commands and what most of the /bin/ executables do.

And sure, I have nothing against introducing an extension; however, a linux executable file should still be recognized as one, even if it hasn't got the ".pex" or something else as extension. If I happen to rename all my files to some other extension by accident, it should still be possible to open them.
And if some dev forgets to follow the extension convention, as is usual in the Linux world, I'd like to still be able to execute it.
Sorry, but that's only my opinion.

I remember a windows exploit where people would distribute an exe file labelled "blah.txt", with a text document icon.

I would really hate it if I clicked on "Happy_Pic.jpg" and rather than launching GIMP or a photo viewer, it steals all my passwords from Firefox, because of MIME type handling.

Maybe display a warning if an executable without an executable extension tries to run off an SD card?

Linux uses MIME for everything, and Linux certainly isn't slow.
Or I should say: GNU/Linux, since Linux isn't an OS.

1) Some distros are slow. Ubuntu certainly is.
2) We have a really low number of IOs available per second. Anywhere from 10-25 would be a good estimate.

Interesting fact (as maybe my final words):
The new PackageKit technology, currently integrated in Fedora (and of course Gentoo etc, for those who are into that sorta thing), has the capabilities to automatically download and install an application for an unknown MIME type. Just double-click on the file, and if none of your programs are able to open it, a new one is installed that can.
That wouldn't work with extensions since they are ambiguous.

:blink:

I don't like that at all. Maybe you cut that description down? I really hate it when my computer does anything important without asking/suggesting, first.

Reminds me of the XP device manager always trying to connect to the internet to search for drivers for unknown devices. :rolleyes:

 

[Squidge]

I don't like that at all. Maybe you cut that description down? I really hate it when my computer does anything important without asking/suggesting, first.
I would assume you would get a dialog box showing words to the effect of "No installed application is registered to handle the file type image/xyz, do you want to check if there is an application available?" and if the user selects "Yes", then give a list. Certainly don't do it all automatically however, that would be major WTF.

Also, I would really be against a filename such as "readme.txt" being an executable that runs without asking upon double clicking - that could cause a fair amount of confusion and surprise. I know that can happen on Windows (unless configured differently), but at least it has an appropriate icon to show that the file was actually executable.

 

[lulzfish]

I don't like that at all. Maybe you cut that description down? I really hate it when my computer does anything important without asking/suggesting, first.
I would assume you would get a dialog box showing words to the effect of "No installed application is registered to handle the file type image/xyz, do you want to check if there is an application available?" and if the user selects "Yes", then give a list. Certainly don't do it all automatically however, that would be major WTF.

Also, I would really be against a filename such as "readme.txt" being an executable that runs without asking upon double clicking - that could cause a fair amount of confusion and surprise. I know that can happen on Windows (unless configured differently), but at least it has an appropriate icon to show that the file was actually executable.

Ubuntu has a special icon for executables, and when you double-click an executable text file like a script, it will open a dialog box that says, "This is an executable text file. Do you want to run it or open it?"

 

[Tor]

Well, I for one will not care too much about what you do with extensions vs. mime/libmagic, except this: Whatever you choose to do, please leave out extensions for executables! Most files in Linux actually use extensions, e.g. .png, .jpg, .ps, .pdf, .tar, .tar.gz and so on. The files that almost never have any extension are executable files, that is how you recognize them when there are no other means.

 

[skeezix]

Keep context..

The de facto for Pandora will be pnd files or PXML.xml directory based apps, as far as the user is concerned. Theres not a lot of reason to be tossing around single files that are compiled binaries with no packaging. The primary users of that I woudl expect to be devs.. compile somethign and run it, and too lazy to rename one line in a template PXML.xml file. Or send the file to someone else as an alpha/beta without doing 'make pndfile' or something. So whatever.

Those people probably also know how to say "./foo" to run it; the extension won't reallymatter. The only value it could add would be to put it in the libpnd autodiscovery code so the launcher could find "./foo.pnx" as an executable, without having to magic every file on an SD card.

So if we really want, I can drop that in pretty easily without breaking anything. I'm just ot sure if its worth it

jeff

 

[skeezix]

I should add..

One concern is.. we should never have a case, on SD, where >1 app is in one dir that are unrelated. ie: Say we add file extension .pnx as autodiscovery, and then you drop 5 of them in one dir. It might be hard to put them into the pnd_run.sh to invoke them, but if we did .. (since we want the working-directorywrites to always go to the same SD appdata directory).. couldbe really messy I imagine.

Definately should be left to devs only, and thus not really worth it.

jeff

ie: Consider..

Application directory 'foo' with 'PXML.xml' in it, and executable 'foo'; works fine, since app is all contained in a directory, and the libpnd/pnd_run.sh process ensures that any writes back to it go to the same SD card, in the /appdata directory (whatever we call it.)

Likewise for a .pnd bundled app.

But if you just have somedir/bar.pxe and somedir/glorpy.pxe .. how would we mount that to union it to the appdata? And if we did, then both those would have to use the same appdata. Then you go and move them to another dir and ..... a different appdata again, or same one? We'd have to go by bin name since we have no unique-id to go by?

Starting to ge really messy. Stick to PXML.xml and .pnd apps.