Command line help needed (All files within foldertree)


DrasticNerd

Script Kiddie
Joined
Jul 11, 2019
Messages
122
I don't see what's so scary about converting between a command line output or even relatively crusty html and xml. XML is just a text format with a tightly defined schema. You could write a program in perl, python, java almost any language if you wanted to do this repeatedly and relatively safely, or you could even do the conversion manually in a text editor for a one off.
ah I think you have misunderstood my comment. Converting between file types (XML, HTML etc.) isn't a problem.

The conversion I was referring to is converting the output from a generic checksum tool into a dat file.

Have you ever opened up a dat file in a text editor to look at its contents? If yes then maybe you will understand why it is going to be onerous to write this conversion code entirely from scratch (seeing as there is no foundation I can build off. Editing dat files for rom audit programs is too niche). It's not that I am incapable of writing this converter - I know that I have the skills to do it. It's the length of time that it will take to get it perfect.

If you are not familiar with dat files, you can see a small sample I attached to my thread here
 

DrasticNerd

Script Kiddie
Joined
Jul 11, 2019
Messages
122
No dat file there - just an archive with an xml file.
Thanks @Djoga'Ro That is the file I meant.

I use the term "dat file" to refer to any file designed to contain rom checksum information (i.e. for use in a rom audit program), regardless of the file extension of the file in question.
 

Djoga'Ro

moonstruck
Joined
Apr 3, 2016
Messages
2,098
The file extension is just part of the name - you can change it, it wouldn't change the content's format.
So the question is, will you know or can you recognise the format of every file you wish to manipulate and will there only be formats you'll know to handle. Unpacking and packing archives can be done with cli tools or via libraries, for a lot of archive formats that is, not all of course.
 

DrasticNerd

Script Kiddie
Joined
Jul 11, 2019
Messages
122
Yeah, but regardless of the extension, it contains text that is valid XML. You've said converting to and from that isn't a problem already.
Sorry. I am having difficulty explaining.

It isn't that difficult to convert a text-based file to another text-based file type. That is if the contents don't need to adhere to a strict structure.

My problem is that I will need to convert a report output by a generic hash tool into a XML dat file. From previous experience (converting a different file format into a XML dat file) I know that it will be ardous.

(Which is why I was hoping that DatUtil would be the answer. Only to hit a roadblock when it appears that the tool can not search below one level of subfolders! :-|| Can not understand why it was ever programmed that way.)


So the question is, will you know or can you recognise the format of every file you wish to manipulate and will there only be formats you'll know to handle. Unpacking and packing archives can be done with cli tools or via libraries, for a lot of archive formats that is, not all of course.
Not sure what you mean by manipulating files? I have no intention of unpacking any files.

I point the tool at the game folders and run it to record the checksums (and filename + paths) of every file inside each game folder.

Neither the tool (nor I) know what types of files will be inside each game folder and it actually doesn't matter. I only need to record their existence, location and checksums. The hard part will be converting the contents of the generic report into a dat file.
 

Djoga'Ro

moonstruck
Joined
Apr 3, 2016
Messages
2,098
It's been a while, but iirc you mentioned something like you could have output files lik eyou need them almost, just one kind of hash missing. In that sense, manipulate := add the missing values.
 

FBnil

my lonely NES is skilling me
Joined
Dec 14, 2012
Messages
4,346
Location
Yurp
If the tool can not look into sub-sub directories, why not create lots of symbolic links to fool the tool that the files are there?

So original dir:
  • C:\ExampleOfFolderContainingGames\Game1\Imafile.dll = DETECTED
  • C:\ExampleOfFolderContainingGames\Game1\SubfolderOfGame1\Imafiletoo.dll = NOT DETECTED

After transforming:
  • C:\ExampleOfFolderContainingGames\Game1\Imafile.dll = DETECTED
  • C:\ExampleOfFolderContainingGames\Game1\SubfolderOfGame1___Imafiletoo.dll = DETECTED
  • C:\ExampleOfFolderContainingGames\Game1\SubfolderOfGame1\Imafiletoo.dll = NOT DETECTED

Windows has junctions which you can set, and after the replacing all / with ___ (and then in your xml you can replace them back to / )



and then to clean up, search for all junctions and delete these (just dir /s /b |findstr ___ and delete all files in that output; make sure you have no files with triple underscore BEFORE you create the junctions)

 

DrasticNerd

Script Kiddie
Joined
Jul 11, 2019
Messages
122
If the tool can not look into sub-sub directories, why not create lots of symbolic links to fool the tool that the files are there?

Thanks FBnil. That is quite an original approach to the problem I have. (That is not sarcastic. I am sincerely impressed with the way you have thought outside the box)


The catch is - and the fault is mine, not you - is that I'm, ah, highly allergic to symbolic links!

(I first heard about sym links + junction links a few years back. I read up on them and proceeded to go wild with creating them all over my PC. It ended up with me accidently deleting lots of my own files when trying to remove some of the links. Since then I have refused to have anything to do with them!)
 

DrasticNerd

Script Kiddie
Joined
Jul 11, 2019
Messages
122
It's been a while, but iirc you mentioned something like you could have output files lik eyou need them almost, just one kind of hash missing. In that sense, manipulate := add the missing values.

Would that it were that simple!

Off the top of my head the only one I can think of that has a missing hash is QuickSFV. (It does all subfolders but it only hashes MD5)

As such, it is inferior to HashMyFiles (Which can do all subfolders, can do CRC32 + MD5 or SHA for that matter)

Ideally I would prefer to work with DatUtil as it does all of the right output - not just hashes but recognising each folder in the first layer as being unique games and the "rom dat" compatibility (which is arguably the most important aspect as creating rom dats is my primary goal for the projects I have in mind)

However, as far as I can make out, DatUtil can not process more than one level of subfolders. (I find it hard to take in how such a design flaw could have been introduced. Unless the creator thought this was a feature - as it was originally designed for MAME roms which don't have a complex foldertree structure AFAIK)

As the program was last updated around 10 years or so ago and as the source provided on the website is, I'm informed, incomplete, I don't see a way to get DatUtil working.

Which leaves me with running HMF and writing code to convert its output into "rom dat compatible" output. I have done similar things before - for instance, I have converted SmokeMonsters lists into rom dat files (for an example, see my GBC translated rom dats in this thread here ) but the source data was tab delimited text and it already had all files grouped for each game - which a generic hash tool can not do.


...I'm now thinking of shelving this project for the foreseeable future. (Too little benefit for the amount of effort involved + I'm now working on other interesting hobby projects...)




Afterthought - if anybody reading this in future has the full source code to DatUtil then should you ever decide to release an updated version (no depth restriction), PLMK TY!

Afterthought 2 - Or if anybody decides to create their own alternative to DatUtil then I would be highly interested in testing your release!
 

PowerGod

Forum Addict!
Joined
Jun 20, 2011
Messages
4,018
I still can't understand why you find this so difficult to obtain, but anyway it's also not clear to me in what way the subfolders should be included in the list...
as an example, this is a piece of one of the xml you posted:
Code:
<datafile>
    <machine name="Arle no Bouken - Mahou no Jewel (Japan) (SGB Enhanced) (GB Compatible) [T-Eng Jazz]">
        <description>English translation by Jazz version (2.1)</description>
        <rom name="Arle no Bouken - Mahou no Jewel (Japan) (SGB Enhanced).gbc" size="1048576" crc="9a14a37a" md5="d2bb101d6b203d8b29cc0db5944eb244"/>
    </machine>

Now, let's say that "machine name" is the main folder of the game, and "rom name" is one of the files in the first folder, how should the subfolders and relative files be included in this format ?
Should every subfolder become a "machine name" ? Or maybe the "rom name" will become "subfolder+'\'+name" ? Or you don't even want subfolder names reported here, but just the entire list of files/subfiles like if they where all in the main folder ?
 

FBnil

my lonely NES is skilling me
Joined
Dec 14, 2012
Messages
4,346
Location
Yurp
Same here: I still do not understand your structure. I failed to produce "extra games" by giving it subdirectories. It just fails to see a game there. Can you show me a structure of such zipfile or directory structure?
And do you need the directories in the rom name?
If we use the -a (append to output), then a flatfile will be created, that with little work (skipping all datafile/?xml/!DOCTYPE tags and parsing all game data, then prepending and post pending that simple xml header around it) and you get a valid xml. But the games do not reflect where they are. This can be then edited in a second-pass to be added, like so (working code for now with some perl, will try to CMD it):


I've created a directory ./roms/bombermans/bomberman_world/ and ./roms/bombermans/neobomber/ then just gave the datutil 2 rounds, with one directory each time, which then produced:


Code:
        <game name="bomberman_world">
                <description>bomberman_world</description>
                <rom name="bombermans/bombrman/bbm-p0.65" size="131072" crc="0a20afcc"/>
                <rom name="bombermans/bombrman/bbm-p1.62" size="131072" crc="982bd166"/>
        </game>
        <game name="neobomber">
                <description>neobomber</description>
                <rom name="bombermans/neobomber/093-c1.bin" size="4194304" crc="d1f328f8"/>
                <rom name="bombermans/neobomber/093-c2.bin" size="4194304" crc="82c49540"/>
                <rom name="bombermans/neobomber/093-c3.bin" size="524288" crc="e37578c5"/>
                <rom name="bombermans/neobomber/093-c4.bin" size="524288" crc="59826783"/>
                <rom name="bombermans/neobomber/093-m1.bin" size="131072" crc="e81e780b"/>
                <rom name="bombermans/neobomber/093-p1.bin" size="1048576" crc="a1a71d0d"/>
                <rom name="bombermans/neobomber/093-s1.bin" size="131072" crc="4b3fa119"/>
                <rom name="bombermans/neobomber/093-v1.bin" size="4194304" crc="02abd4b0"/>
                <rom name="bombermans/neobomber/093-v2.bin" size="2097152" crc="a92b8b3d"/>
        </game>

is that what you mean? Else, if you just want a flatfile without the location directory structure in the rom name, that second pass does not even to be done. Please explain more clearly what structure you will need.
 
Top