8.3 renaming of ST disk images

[tapsel]

I tested castaway yesterday, which is excellent, as I already said.

But although you can't display more than 12 or so disk images in it's menu in this beta, I've already had
problems today to remember which game is on which disk. If you download FOF or Automation
menus for example, it's a real pain to rename them first before putting them on your SMC (with pclink
hack would get something like XY90A~15.STT). And if there are several games on the disk, you have
to rename the file to something like BUBBL&RI.STT (for Bubble Bobble + Rainbow Island).

This is, as such, not a problem of the castaway emu of course. But I thought, perhaps there is a DB or
something (TOSEC?) similar to the MAME library, which could be implemented into castaway to
recognize the disk images in the start menu?

If not , it will get very tricky and time consuming, once you want to fill a SMC with more than just a
hand full of ST games...

 

[skeezix]

Yea, me too. I was thinking of makign a filename -> game mapper.. ie: If it starts with A, and has 097 in the title, show Automation 97, Xenon", instead. But the filenames coudl be who knows what of course :/

So it also occurs to me to have it build a database of md5 hashes of the disk images, and then list them by that. Woudl be a little work, and slow when it finds new disk images, but shoudl work nciely.

Does this sort of database already exist I can poach from?

Of course, now with saved disk images, it woudl wreck that too, but I doubt too many will save their games..

jeff

 

[tapsel]

You have already thought-out the problem more than I did.

Basically a DB with md5 hashes would be a nice idea, but I doubt that people will
pass on savegames for a more elaborated naming system...

It sounds highly sophisticated to make, and I doubt that it's even possible (I have no idea
about GP32 dev. and emulation)... But I just know that some MAME frontends build up
some sort of game library on every startup...

Would it be possible to integrate some mechanism, which checks for new images on every startup,
tries to match the new files with the "Known-image-DB" (md5 checksum of all known
Menus/Compilations/whatever) and adds them to your "personal file library" either with the
common file name (if no match was found), or with the disk description (e.g.: "Bubble Bobble,
Rainbow Islands + Docs") from the "Known-image-DB" (If the file matched)? After that, the file could
be identified via filename, or another static value, which would make saving possible, without fucking up
the whole naming system.

This would, even if possible, cause alot of other problems l s'pose...

 

[rcx21000]

Couldn't you make a little program that has the database in it and you run all the images through the prog and it will rename them to whatever it is in the database, so in the emu you can just say that if the filename is this then display this untill it gets through the whole database and if it isn't in the database it'll just show the filename (and if when you run it in the program and it isn't in the database it'll leave it alone and a "report generator" would come up to report the incompleteness of the database )

 

[tapsel]

@rcx21000

Hmm... Similar idea, just with renaming the files on-the-fly instead of referencing them with an additional
game library, if I got it right...? Wouldn't this make it necessary to match all the files with the (immense)
disk image database on every startup? I'd imagine this being very slow... Or how would you recognize
already renamed files?

I thought about an internal game library since this would let castaway recognize already checked files
at startup, and even if it had to look over the whole library f or every file (let's say max. 200 files on
128MB SMC) to check if it's a new one, this would be a lot faster than to iterate through the whole
image database (literally thousands of entries).

But I have to stress again that I have absolutely no idea about emu developent and just basic DB knowledge, so
if I'm completely off track, please don't throw bricks at me!

 

[Arutha]

I'm guessing rcx21000 is suggesting an external program to rename the roms/disk images (in this case, on your PC prior to uploading). The emu then just compares filenames, hopefully quicker than doing an md5 comparison on the GP32.

If that's not what you meant, its another possibility.

For me, I would rather have the ability to use zip files implemented before this, but that may not be a common request .

 

[rcx21000]

Yes, that is what I was suggesting, but wouldn't getting MD5 and comparing be even slower or am I missing something. Oh, I just had another idea, just have a prog on the comp go through the database and make a smaller database that says file #1 is called this file #2 is this...

 

[tapsel]

You're most probably right, Arutha! *doh*

But this wouldn't provide a solution for the 8.3 limitation problem, which is the most
annoying thing in my eyes.

What's the point with zip file implementation? Just saving space or is there another
advantage?

Another question... If you have 8.3 formatted zip files, but the zip files have arbitrary
filenames, can GP32 read and print out this filename? This would solve the problem...
or is this a limitation of the GP32 file system?

 

[Arutha]

Okay, what I was meaning specifically is a two step process:

1) On your PC you've downloaded 1, 5 or 100 .ST files... you run them through a program on your PC and it compares them to a database (MD5 or other comparison to know what the file is) and renames the files to a set name. You upload the renamed files to your GP32

2) On the GP32, the castaway emu would have to be programmed to know that a specific filename has XXX games on it (i.e. A_400 would show up as "Golden Axe & Edd the Duck" in the file list)


I am by no means an expert on the best way to do something like this and have no idea what other better ways exist... this method is simply trying to keep the amount of comparison and the database size on the GP32 to a minimum.

As for zip files, I know that you CAN have a zip file named in an 8.3 format that contains a longer filename (I have several zips like this for the MSX emu). I believe Ryleh also has setup the MSX emu to compare the MSX roms/disk images to some standard... if he happens to read this maybe he'll give his advice. Since this is possible, perhaps with zip support the filename of the zipped file (i.e. GoldnAxEddDuck.ST rather than A_400.zip) could be displayed. However, I do not know how easy/hard to program this would be, or how much time it would take to run on the GP32 (unzipping 50+ files to get their names might take too long to be feasible).

I actually just wanted zip for space reasons... I know I can buy more SMCs, but up to know I've been able to keep pretty much every game I want on one SMC (with another for MP3s)... its nice not having to switch to play a different system! I was fully expecting to add at least one more when the Genesis/MAME/SNES emus are running BTW, but I love having lots all on one.

Last note, sorry for the long post... while I was typing this I did a quick zip check... when using cracked (and already compressed) .ST files, the zip for space reasons isn't worth it, you don't gain much. I do not have any original images to test, those would probably be worth it.

 

[skeezix]

You're most probably right, Arutha! *doh*

But this wouldn't provide a solution for the 8.3 limitation problem, which is the most
annoying thing in my eyes.

What's the point with zip file implementation? Just saving space or is there another
advantage?

Another question... If you have 8.3 formatted zip files, but the zip files have arbitrary
filenames, can GP32 read and print out this filename? This would solve the problem...
or is this a limitation of the GP32 file system?

Real problem is the 8.;3 limitation of course..

"md5" is very slow; not sure how slow it'd be on the GP32, but not fast, thats for sure. I'd envision something like this:

Have all your disk images on the desktop; run the database-utility-thingy on them and it'd spit out mydatabase.dat or something, which would have the filename and the descrioption in it, like this:

A_097.STT Dungeon Master, OIDs

Then when Castaway/GP sees the directory, it looks up the description in the datafile if present, and shows it beside the filename. Easy and not too slow.

I could create this easy and pretty fast.

The hard part is building up a master database of md5 to descriptions.. the utility woudl read this file, and then md5 hash all the disk images (on the PC side). Whwen it finds a match in the database, it puts the descrioption into your .dat file for the handheld, so the handheld has almost no work to do.

But we need this database.

So does a md5+description database of ST disks already exist?

Hell, I can make the md5 hashes.. if someone can give me a file like this:

A_097.ST Description...

Then I can md5 hash A_097.ST automagically with a tool I'll make, and it'll spit out the md5 hash and the description, and then we're 3/4 done..

so give me a few volunteers for manual labour and I'll make it happen

jeff

 

[Matty Jetman]

Hiya,

Right this does seem a good idea but it instantly looks like a lot of hard work.
You have a database of the compilaion disks but what happens if you rename them wrong? Or download a wrongly named file? Or some of the games on the comps don't work??

One way round this would be to set up a ftp/website containing all the renamed comps that match the database.
Perhaps we could add our own ST disk images....
I have noticed that some games are missing from all the comps as well.....

ST had the 8.3 limitation as well!!!!

Adios,
Matt

 

[tapsel]

I think the datfile is a very good idea.

But I'm somewhat confused anyway... why stick to the md5 hashes at all, when the files have to be named
specifically? If they'd have another name, they'd also have another md5 hash, so we could use the filename
in the first place, right? :huh:

But I come back to these TOSEC project (the old school emulation project) I mentioned before.
Yesterday I found 4 so-called TOSEC compilations in edonkey, every file about 500MB in size. I didn't
fully understand what the prospects of this project are (just google for it, you will find the project page.
I'm still at work, so I can't check it out right now), since I just didn't care, but as far as i understood, they
tried to rename every single ST (and other systems) disk images with a special naming system. I finished
downloading one of these files and it contained several hundred ST images from about 4 groups like
"ripped off" or "automation". I think since most of the ST images floating around in the net are from such
groups, and these groups probably stick to the TOSEC naming convention (I'm not sure). most of them
may already be named this way. So if we can add this TOSEC library , or just convert a TOSEC list
(filename, description) to a database, we have already covered thousands of very common disk images.

EDIT

Sorry, the post above was a mental snapshot. Like most of the time, I got things confused.

So, browsing http://tosec.org cleared things up a little.

The TOSEC crew renames emulation related files, as I said. But the comp. groups, of course,
don't care much about the TOSEC naming concept.

But!

Most of the ST compilations are covered by the TOSEC libraries, and if you search the databases
on their webpage (info -> dbsearch), you will find them with name, size and a CRC32 checksum!

Checksums like MD5 or CRC32, of course, stay the same when you change the filename. So if we
could use the TOSEC DB, we could identify our disk images by generating the CRC32 checksum
and mapping them with the DB, as skeezix said.

 

[skeezix]

Nono, the trick is to use MD5 since then you do not care about filename at all. I'd only need the filename once, up front, to buildf the first database.

ie: For you non coders : )A "hash" is just a one-way-function, which means it is lossy and doesn't go backwards. So you get a 800k ST disk, and it turns into a 30 digit number say. A checksum. That number is "reasonably unique". If you rename the disk 30 tiumes it doesn't matter.. the content is the same, and the checksum (hash) woudl come out the same every time. The liklihood of 2 disks coming out to the same numhber woudl be low, but could happen. But not frequently, and so for any one perosn to have it happen to them would be low.

So you get all these disk images. You run the converter tool on them, and it spits out a file which has the md5 number, and the games description. No filenames in it.

Do you'd get mygames.dat and copy it to your GP32.

So.. options. Once the convertyer has figured out what disks you really have (nomatter what you name them), it would write out a datafile... filename, md5, description.

Castawya/GP runs, finds the filenames. For each filename, it looks it up in the .dat file. Easy and fast. Then, any that did not match, it md5s them and looks them up in whatevers ledft of the .dat file, to see if they mnatch but you renamed them. (or even skip this step for speed).

All this depends on having a master database that maps on your pc, the md5 to description, though, so that you can run the converter to spit out the file.

The hard part is us building this master database

jeff

 

[tapsel]

Of course you are right about the checksums. I'm a coder too, so I should've known it
better. I was a little confused.

Check my last post... I edited it before I read your answer.

So the TOSEC DB might be the thing we are looking for. It's nearly complete, it has
an elaborated naming concept, and I think the db data is even available for download.
What I don't know though, is if it has a description field for the compilation contents.
The TOSEC naming system may be a good thing to store/identify files on PC-side but it's
not very usable for the castaway menu, since these names are 1. very long and 2. don't
provide you with the information what the disk image actually contains.

But "at least" this would spare us from building up a DB from scratch and generating all
the checksums. And since you'll probably find txt lists from all comp. groups with something like
"disk #, description", you could even fill in the missing description data with simple
scripts.

 

[rcx21000]

I don't exactly understand what you want to do but it does sound something like this which would be the most efficient way of doing it (I think):

1) PC Utillity renames the images to 1, then 2, then 3...
2) PC Utility goes through all disk dumps and gets their MD5 and compares them to the database
3) PC Utility makes a mini-database with only the images that you have like this:

1Attack of the Googlies
2Mad Man 2
3A_097

(sorry but I don't know about any games, so I am almost sure that the names don't exist )
(the ones labeled 1 and 2 are known, but the last one isn't so it's original name is put there)

4) You copy all the images + the mini-database to GP:\GPMM\ATARIST
5) When the emu loads, it checks what the file #1 is on the mini-database then replaces the name, then #2.......

This way should be faster than putting the whole list of games on the GP32 and only the comp has to do MD5 (comps always get the hardest work )

 

[stickofjoy]

there's a list of all the compact disc menus and contents on JOE'S FTP. it would take a little bit of formatting, but if we decide on some rules it might be possible to script it or if we pick one each we could format them into some sort of order?

:lol:

 

[skeezix]

It occurs to me I could even go for a cheap and easy hack; a filename -> description mapping, that each person creates themselves. After awhile, everyone gives me their files, I merge them, and then we have a file to include in the distribution. The filenames in practice do not change much. Automation is almost always A_###.ST for instance.

So I could make it take a file with "filename<TAB>Description", and then after a couple weeks I do the big merge. That'd be cheap, but easy

jeff

 

[Flack]

Hey Skeezix,

Here's an idea that would be more work for end users, but less work for you. Have a file that the user creates himsef with the translation. In other words:

AFILEO~1.XXX, A File Description One
AFILET~1.XXX, A File Description Two
etc.

Then, the user can just copy this file to their GP32 under the Castaway directory. Castaway would then check the list and convert any found 8.3 filenames to long file names, and leave any not found as short filenames.

I just see you guys chasing your tails for a long, long time, trying to find all possible disk images and all possible name variations, etc.

Note: this doesn't disclude making a "master list" someday, that could be distributed with Castaway. Users would simply overwrite (or merge) their own list.

 

[tapsel]

Well... having a simple tool for it would still be l33t though.

You load a list of disk images into a simple converter utility, and enter a description for the
content of every file using the GUI. The application reads file after file from your FS, calculates the
CRC32 checksum, gets something like "55B8364A", and copies the original disk image to a local
ATARIST folder with the new filename "55B8364A.STT" (it's a 8 bit checksum, so we could just
as well use it), leaving the original disk image unchanged. Then it also writes an entry to the
datfile, e.g.: "5B8364A;My presonal description". The datfile is written to the ATARIST folder after the
conversion, you upload the whole folder to your GP32. Castaway maps filenames to datfile, voila!

This way, once we have a "master list" with CRC32 checksums (I'll stick to the TOSEC DB, since in my
eyes it would be perfect), we could integrate this DB to the utility. When you load a file, the utility
calculates the CRC32: "55B8364A", checks the internal DB, finds an entry (according to the
TOSEC DB this would be: "55B8364A, 'Automation Menu Disk 389 (19xx)(Automation)[a].st',
'Prince of Persia') and returns "Prince of Persia" as a content description proposal. You can edit
the description in the GUI if you don't like it, then let the utility make the rest.

 

[stickofjoy]

just found this big list, may be of help?

http://stgameslist.atari.org

matt