Command line help needed (All files within foldertree)

[DrasticNerd]

I'm tinkering with a little hobby project and I've run into a commandline problem I cant solve.

My goal is to checksum all files within a specific foldertree and output the result formated as a xml dat file.

Rather than use a generic checksum tool and then go through the headache of converting its output to the desired format (I may end up going down that route if all else fails), I looked around for a dat creation program.

The only solution I could find was "DatUtil"

This looked like the answer I was hoping for until I tried running it. When I run it on a folder - it only processes the files within the first layer of subfolders of that folder.

EXAMPLE:
say the top level folder is C:\ExampleOfFolderContainingGames\

Suppose I use commandline:

datutil.exe -f genericxml -a "output.xml" -v "C:\ExampleOfFolderContainingGames"

and:

• C:\ExampleOfFolderContainingGames\Game1\Imafile.dll = DETECTED
• C:\ExampleOfFolderContainingGames\Game1\SubfolderOfGame1\Imafiletoo.dll = NOT DETECTED

I tried to be clever and wrote a bat file to recurse the foldertree but recursing the commandline used is useless. (Because the recursion continues inside an existing game folder, the tool then treats that game folder as the directory containing the game folders. So all of the subfolders within the game folder are output as separate games!)

Is there a clever commandline trick that can be used to process all files within 'C:\ExampleOfFolderContainingGames' ?

 

[levi]

Oh, windows

 

[Djoga'Ro]

Oh, xml

Since the tool expects a directory and not a file list, it would be up to the tool to walk the paths. I don't see no recursive option in the readme.
But maybe the options to merge -i and/or -I (uppercase i) could be used. In your script walk the directories bottom up. Within a path of the same game first do the thing for the current dir. then merge the results in the one-level-down dirs (which you did in earlier iterations of the loop) into the result of your current dir. But from a games top-level dir you don't write an intermediate result into the then current dir, but into your collection top dir; when processing the first game, with -o, and after that/when processing each next game with -a to append.
I don't really know, the concepts at play here, so I might be wrong entirely. ?

 

[PowerGod]

What king of "games" do you have in that foldertree ?
As far as I see that tool is only for MAME-like ROMS, and those are really dependent to the emulator/version used to play them... so, seems to me that the best way would be to do the contrary, create first the XML lists of supported games from the emulator(s) you need (or get the correct version of DAT files from some sites), and with that(those) DAT, check your game folder with tools like RomCenter or CLRMamePro.
Its is better to move your roms in different folder then, to keep separate ROMS for every version of the emulator.

I was working for some tools to deal with directories full of ROMS and STUFF you don't even know how them are compressed or what them contain, but are for *nix, and anyway to do console things I suggest you to install Cygwin on Windows, it is the definive tool.

The "ARCHIVE UNMESSER" was made because I had folders with subfolders with archives in many formats, and those archives not only sometimes had a wrong extension, but also they contained sub-archives inside them...
So the tool, using recursively 7z just extracts EVERYTHING from what that can recognize as an archive, and recompress "every single file" as ZIP in a single directory, the result then can be scanned with ADVSCAN (using the DAT you want) so that it will merge/remove all the duplicates and not supported ones by the emulator you want to use.

ADVSCAN can put the removed roms in a folder that can be rescanned again with other DAT files, so you can get from there the roms managed by other DAT files.

PowerGod / aMESSofROMS · GitLab

A collection of CLI tools to manage ROMS starting from an EXTREME MESSY CONDITION. These tools, if combined, will become a single command to have a clean folder...

dev.pyra-handheld.com

I made a little guide for the ADVSCAN years ago, but you can also use other tools to do the same.

[WIP] Advance Projects and ROM management guide

== WORK IN PROGRESS == I'm trying to create a complete guide, easy to read but very explanatory, to describe how to use and manage ROM SETS for MAME-like emulators, but also I want to point out why you have to do this type of things. I never found a guide like this, I just found little guides...

pyra-handheld.com

 

[DrasticNerd]

Oh, windows

lol. forgive me

Oh, xml

hmm well it doesn't have to be XML. Any output that is a dat file format that is accepted by rom manager programs (e.g. ClrMamePro, RomCenter, Romulus) would be FBM. (I currently prefer Romulus for ease of use but IIRC the major rom programs accept most of the standard dat formats.)(Exceptions apply ofc for niche or locked dat formats such as Cowerings GoodTools and AdvanceScene)

I am only looking at XML output for this project because I have experience with XML dat files from a separate (completed) project of mine. (This other project reads downloaded dat files and applies automated editing - e.g. custom merges, splits, purges, renames etc. and outputs custom dat files for me to use...)(Yes, I take rom naming seriously. It's one of my nerdy interests!)

I don't see no recursive option in the readme.

Well no I couldn't either. But I wasn't expecting a recursive method in the tool.

I'm hoping there is some windows commandline magic I could use instead. IDK, something like (warning: made up example following) " C:\YourPath\ *.* /f " - a generic advanced command which means 'process everything in this foldertree'

But maybe the options to merge -i and/or -I (uppercase i) could be used. In your script walk the directories bottom up. Within a path of the same game first do the thing for the current dir. then merge the results in the one-level-down dirs (which you did in earlier iterations of the loop) into the result of your current dir. But from a games top-level dir you don't write an intermediate result into the then current dir, but into your collection top dir; when processing the first game, with -o, and after that/when processing each next game with -a to append.

I should have mentioned I did try the -o and -a options but it didn't work like I wanted. I wrote a bat to recurse through all subfolders and apply the command line each time but it treated each lower level folder as being a new game name. (Which was obvious in hindsight - it is starting the command afresh on a different folder. So it treats the first level of subfolders from the toplevel folder as each folder = 1 new game)

As far as I see that tool is only for MAME-like ROMS,

I agree that the tool was clearly designed for MAME first and foremost. Processing files to record their checksums isn't MAME exclusive however. (What I mean is that there is nothing stopping you from running DatUtil on, say, translated SNES roms.)

From my observation the only major difference between MAME dats and non-MAME dats is that MAME dats record additional information for child roms, clones etc. (Technically even that isn't exclusive to MAME dats as Gruby dats do this too - they group region variants of games emulated by ScummVM into child versions of a parent game!)

In any case, DatUtil doesn't claim to be MAME exclusive. It does allow creation of non-MAME dat files.

The problem I am running into - and IDK if this is an oversight or an intentional "feature" - is that the tool appears to only process the first level of subfolders

i.e. you can scan a directory which contains subfolders of games (each folder is a separate game)(first subdir level) but it doesn't appear to support scanning the contents of any subfolders in that game folder (second level of subdir)

the best way would be to do the contrary, create first the XML lists of supported games from the emulator(s) you need (or get the correct version of DAT files from some sites), and with that(those) DAT, check your game folder with tools like RomCenter or CLRMamePro.

That's the advice I would give somebody getting into emulation too.

It doesn't work when you are working with games that are not recorded in any of the publicly available dat files. Which is why I am looking into scanning files to create my own dat files.

 

[ElPoco]

I don't think there is any generic recursive option for Windows shell, but you should be able to make a batch file (or command) that will do it.

Windows Batch File Looping Through Directories to Process Files?

I need to write/use a batch file that processes some imagery for me. I have one folder full of nested folders, inside each of these nested folders is one more folder that contains a number of TIF i...

stackoverflow.com

 

[PowerGod]

@DrasticNerd : then the thing could be a little more complex... anyway I remember some kind of program that had a database to recognize all the PC games in a folder and it was able to find out even the version/region of the games using checksum... I'll do a little research and report if I find it again...

Anyway, this can go in every file of every subdirectory, just put the instructions you need in the "echo %%f" part

for /f "delims=" %%f in ('dir /b/s') do (echo %%f)

EDIT:
I found the program to catalogue the software, it's called DataCrow and after scanning a folder recursively is able to check the files online, put them in the local DB and from there you can EXPORT an XML with the data you need, there's also the MD5 HASH, but I don't see the CRC... anyway could be a great base for a list.

Data Crow - Cataloger

EDIT:
in the options you can choose what kind of check you want to use and there's a long list, CRC included, but you can use only one type of check, so only the active one will be exported.

 

[FBnil]

@DrasticNerd Why not use Powershell? You can create an empty XML object and inject all the values as you go, then save it.

You can use Get-Item to get the files using a pattern

Or use Get-ChildItem

Get-ChildItem C:\ExampleOfFolderContainingGames\ -Filter *.dll -Recurse | select -ExpandProperty FullName | ForEach-Object { Write-Host $_ }

Get full path of the files in PowerShell

I need to get all the files including the files present in the subfolders that belong to a particular type. I am doing something like this, using Get-ChildItem: Get-ChildItem "C:\windows\System32" -

stackoverflow.com

If you do not like PowerShell, then did you try dir?

dir /B /S C:\ExampleOfFolderContainingGames\*.dll |findstr something

The other thing is that do not understand. do you want md5sum.exe the subdirectory dll files or not (only the toplevel ones, inside each game)?

 

[DrasticNerd]

@ElPoco - Thanks for the suggestion. I had already made a bat file to recursively run the command on each folder and the results are no good. (It treats the immediate level of subfolders as new games. So recursing each folder level in the tree is not the way to go)

@PowerGod -

• Thanks for the suggestion. A "for" would would run the tool command on each directory. Whereas I need the command to process the entire foldertree. If that is even possible.
• Data Crow - Interesting. The problem is I need CRC32 as well as MD5 for my project. (I had previously looked at using QuickSFV for my project. While fast, it only outputs MD5 checksums so that killed off that line of enquiry.)
• (Yes, I'm aware of the existence of dat files for PC games - vastly incomplete ofc - but I'm not interested in dat files for PC games.)(BTW I didn't say my project was processing PC games... )

@FBnil -

• Yes, I do PowerShell a little. (I am writing my own sets of tools for a game program that allows powershell scripts as add-ins)
• Thanks for the suggestion. A "ForEach-Object" would would loop the tool command. Whereas I need the command to process the entire foldertree. If that is even possible.
• "do you want md5sum.exe the subdirectory dll files or not (only the toplevel ones, inside each game)?" My problem is that DatUtil appears to be only processing the input folder, its immediate subfolders and the files in the roots of those subfolders. Whereas I want a full foldertree. I want to return ALL files in ALL subdirectories of C:\ExampleOfFolderContainingGames\

Post automatically merged: Aug 23, 2021

I feel as if I'm clutching at straws by now. I'm beginning to reach the conclusion that there isn't any advanced commandline magic that will do what I am looking for.

That being the case, there are only two possible approaches left.

1. Fix the DatUtil tool
2. Find a tool that will process an entire foldertree and output their CRC32 and MD5 checksums.

I want to avoid option 2 as it will be one major PITA to write code to convert whatever output the tool produces into a XML dat format.

Option 1 would be doable if I could compile and knew C (or C+ - I'm not sure which one it is)

The source code is inside the download here

Spoiler: copy of source code

/* --------------------------------------------------------------------------
* DatUtil - Written by Logiqx (http://www.logiqx.com/)
*
* A simple little utility for converting, cleansing and comparing ROM manager
* dats (e.g. ClrMamePro, RomCenter).
* -------------------------------------------------------------------------- */

/* --- Version information --- */

#define DATUTIL_VERSION "v2.46"
#define DATUTIL_DATE "13 April 2009"


/* --- The standard includes --- */

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/stat.h>


/* --- Dat library includes --- */

#include "datlib/src/datlib.h"
#include "datlib/src/macro.h"


/* --- Misc library includes --- */

#include "datlib/src/misc/getopt.h"


/* --- DatUtil definitions and macros --- */

#include "datutil.h"


/* --- Is DatLib debugging enabled? --- */

extern int datlib_debug;


/* --------------------------------------------------------------------------
* The main program
* -------------------------------------------------------------------------- */

int main(int argc, char **argv)
{
/* --- For getopt function --- */

extern char *optarg;
extern int optind;
char c;

struct stat buf;
struct dat *dat=0, *blend=0;
struct options *options=0;

int test=0, incorporate=0, errflg=0;

/* --- Display version information --- */

printf("===============================================================================\n");
printf("DatUtil %s (%s) - using ", DATUTIL_VERSION, DATUTIL_DATE);
display_datlib_version();
printf("Written by Logiqx (http://www.logiqx.com/)\n");
printf("===============================================================================\n");

/* --- Allocate memory for user options --- */

STRUCT_CALLOC(options, 1, sizeof(struct options))

/* --- Get the options specified on the command line --- */

while (!errflg && (c = getopt(argc, argv, "f:qkjo:a:tA:V:C:R:F:T:E:H:U:O:M:Z:N:g:cG:S:!rlsXDp:i:I:xmnvd?")) != EOF)
switch (c)
{
/* --- Saving --- */
case 'f':
LOWER(optarg);
options->save_format=optarg;
break;
case 'q':
options->options|=OPTION_ALWAYS_QUOTE;
break;
case 'k':
options->options|=OPTION_KEEP_FULL_DETAILS;
break;
case 'j':
options->options|=OPTION_NEBULA_JUKEBOX;
break;
case 'o':
options->save_name=optarg;
options->save_mode="w";
break;
case 'a':
options->save_name=optarg;
options->save_mode="a";
break;
case 't':
test++;
break;
/* --- Header text --- */
case 'A':
options->header.author=optarg;
break;
case 'V':
options->header.version=optarg;
break;
case 'C':
options->header.category=optarg;
break;
case 'R':
options->header.name=optarg;
break;
case 'F':
options->header.description=optarg;
break;
case 'T':
options->header.date=optarg;
break;
case 'E':
options->header.email=optarg;
break;
case 'H':
options->header.homepage=optarg;
break;
case 'U':
options->header.url=optarg;
break;
case 'O':
options->header.comment=optarg;
break;
case 'M':
options->clrmamepro.forcemerging=optarg;
break;
case 'P':
case 'Z':
options->clrmamepro.forcepacking=optarg;
break;
case 'N':
options->clrmamepro.forcenodump=optarg;
break;
/* --- Game selection --- */
case 'g':
options->options|=OPTION_GAME_SELECTION;
options->game_selection=optarg;
break;
case 'c':
options->options|=OPTION_CLONE_SELECTION;
break;
case 'G':
options->options|=OPTION_SOURCEFILE_SELECTION;
options->sourcefile_selection=optarg;
break;
case 'S':
options->options|=OPTION_SUBSTRING_SELECTION;
options->substring_selection=optarg;
break;
case '!':
options->options|=OPTION_INVERT_SELECTION;
break;
case 'r':
options->options|=OPTION_REMOVE_CLONES;
break;
/* --- Cleansing --- */
case 'l':
options->options|=OPTION_LOWER_CASE;
break;
case 's':
options->options|=OPTION_SORT_GAMES_BY_PARENT;
break;
case 'X':
options->options|=OPTION_FIX_MERGING_OFF;
break;
case 'D':
options->options|=OPTION_REMOVE_DUPLICATES_OFF;
break;
case 'p':
if (strstr(optarg, "rom"))
options->prune_roms++;
if (strstr(optarg, "disk"))
options->prune_disks++;
if (strstr(optarg, "sample"))
options->prune_samples++;
break;
/* --- MD5/SHA1 --- */
case 'i':
options->fn=optarg;
break;
case 'I':
options->fn=optarg;
incorporate++;
break;
/* --- MD5/SHA1 --- */
case 'x':
options->options|=OPTION_EXTENDED_CHECKSUMS;
break;
case 'm':
options->options|=OPTION_MD5_CHECKSUMS;
break;
/* --- Directory scan --- */
case 'n':
options->options|=OPTION_NO_INSPECT_FILES;
break;
/* --- Information --- */
case 'v':
options->options|=OPTION_VERBOSE_LOGGING;
break;
case 'd':
options->options|=OPTION_SHOW_DEBUG_INFO;
break;
case '?':
errflg++; /* User wants help! */
break;
}

/* --- The user must specify either one or two dat files --- */

if (argc-optind!=1)
errflg++;

/* --- Display the help page if required --- */

if (errflg)
{
printf("The best tool for converting between different ROM manager data file formats!\n");
printf("For instructions on use, read the documentation that's been provided. Here is\n");
printf("a brief summary of the options... far greater detail is in the documentation.\n\n");

printf("Saving [-f <output format>] [-q] [-k] [-j] [-o|a <output file>] [-t]\n");
printf("ClrMamePro + [-R <refname>] [-F <fullname>] [-C <category>]\n");
printf(" RomCenter [-V <version>] [-T <date>] [-O <comment>]\n");
printf(" [-A <author>] [-E <email>] [-H <homepage>] [-U <url>]\n");
printf(" [-M <merging>] [-P <packing>] [-N <nodump>]\n");
printf("Game Selection [-g <game names> [-c]] [-G <sourcefile names>]\n");
printf(" [-S <substring of description>] [-!] [-r]\n");
printf("Cleansing [-l] [-s] [-X] [-D] [-p[rom|disk|sample]]\n");
printf("Blending [-i <info file>] [-I <incorporate file>]\n");
printf("MD5/SHA1 [-x] [-m]\n");
printf("Information [-v] [-d]\n\n");

printf("Output formats and aliases:\n\n");

printf("MAME ListInfo listinfo info li clrmamepro clrmame cmpro cmp cm\n");
printf("MAME ListXML listxml lx\n");
printf("Generic XML genericxml generic gx\n");
printf("RomCenter 2 romcenter2 rc2 romcenter rc\n");
printf("Tab Delimited delimited tab text\n");
printf("AAE RomList aaeromlist aae romlist\n");
printf("MAME GameList gamelist gl\n");
printf("TitleList titlelist titles\n");
printf("RC2 Sublist sublist sub rcsublist rcsub\n");
printf("Map File mapfile map\n");
printf("HyperSpin Database hyperspindb hyperspin hsdb\n\n");
}

if (!errflg)
{
/* --- Always show a summary after loading --- */

options->options|=OPTION_SHOW_SUMMARY;

/* --- If using an 'blend' dat, load it into memory --- */

if (options->fn)
{
if ((blend=init_dat(options))==0)
errflg++;
else
{
printf("\n");
if (incorporate)
options->incorporate=blend;
else
options->info=blend;
}
}

/* --- User must have specified an input name --- */

options->fn=argv[optind];

/* --- Log name for DatUtil --- */

options->log_fn="datutil.log";

/* --- If the user didn't specify an output name, use datutil.dat --- */

if (!options->save_name)
options->save_name="datutil.dat";

/* --- If the user didn't specify an output mode, use overwrite --- */

if (!(options->save_mode))
options->save_mode="w";

/* --- Initialise the main dat --- */

if (!errflg && (dat=init_dat(options))==0)
errflg++;

/* --- Save dat from memory --- */

if (!errflg)
{
if (test==0)
errflg=save_dat(dat);
else
printf("No data file has been saved since 'test mode' is being used.\n");
}

/* --- Release dat(s) from memory --- */

dat=free_dat(dat);

if (blend)
blend=free_dat(blend);

/* --- Delete the log file if it is empty --- */

if (stat(options->log_fn, &buf)==0 && buf.st_size==0)
unlink(options->log_fn);
}

if (datlib_debug)
{
printf("%-16s: ", "DatUtil.main");
printf("Freeing memory of options...\n");
}

FREE(options);

return(errflg);
}

I can't make sense of the source code. Where is the code limiting the subdirectory levels?

 

[PowerGod]

You'll need also to look at the code of these libraries that are used:

/* --- Dat library includes --- */
#include "datlib/src/datlib.h"
#include "datlib/src/macro.h"


/* --- Misc library includes --- */

#include "datlib/src/misc/getopt.h"


/* --- DatUtil definitions and macros --- */

#include "datutil.h"

 

[Djoga'Ro]

I can't make sense of the source code. Where is the code limiting the subdirectory levels?

/* --- Initialise the main dat --- */

if (!errflg && (dat=init_dat(options))==0)
errflg++;

/* --- Save dat from memory --- */

if (!errflg)
{
if (test==0)
errflg=save_dat(dat);
else
printf("No data file has been saved since 'test mode' is being used.\n");
}

It looks like all info about the job is in 'options' and init_dat is doing the work. So look at

#include "datlib/src/datlib.h"

or, if it has only the function declarations (as it should be), look for the corresponding datlib.c (or datlib.cpp maybe, if this is C++, though the code rather looks like plain C).


What about?:

HashMyFiles: Calculate MD5/SHA1/CRC32 hash of files

HashMyFiles is small utility that allows you to calculate the MD5 and SHA1 hashes of one or more files in your system.

www.nirsoft.net

Though I don't get, why someone would keep the source code something like that to themselves.

How's your programming at all or your object-oriented programming? With Visual Studio Code (or what the free edition's called) and utilizing some open source libraries for md5, crc32 generation and xml reading/writing it should be an easy project in C#. Or Visual Basic even.
xml export libraries typically just express a given data structure in xml. So the critical part would be, defining the data structure elements the right way for the xml lib to do the right thing, but that's to be expected to be well documented, and foremost to match a certain target structure, if a particular xml format is to be made.
Everything else about it should be a walk in the park.

 

[Djoga'Ro]

Find a tool that will process an entire foldertree and output their CRC32 and MD5 checksums.

I just realized, that might not be all you want - that said tool still needs to gather more information about the files content. ?

Still, if programming's an option, you could use a tool that does all you need, but maybe lacks crc32ing or md5ing. Then you write something, that reads the xml file, generates for all data files mentioned there the missing checksum/hash, adds that value and writes out a new xml (or overwrites the old one).

 

[levi]

HashMyFiles: Calculate MD5/SHA1/CRC32 hash of files

HashMyFiles is small utility that allows you to calculate the MD5 and SHA1 hashes of one or more files in your system.

www.nirsoft.net

Though I don't get, why someone would keep the source code something like that to themselves.

Because CRC32/MD5 and even SHA1 have easily computable clashes. They're still useful if you control all of the data, and there's nobody who gains anything by engineering in a hash clash. But if it were open source, someone could edit it and add SHA2 and SHA3 support, which would modify a large part of the project, and may make it effectively theirs if you need in future to relicense it (mainly in order to sell it).

 

[Djoga'Ro]

Because CRC32/MD5 and even SHA1 have easily computable clashes. They're still useful if you control all of the data, and there's nobody who gains anything by engineering in a hash clash. But if it were open source, someone could edit it and add SHA2 and SHA3 support, which would modify a large part of the project, and may make it effectively theirs if you need in future to relicense it (mainly in order to sell it).

Sorry, I'm trying, but I don't get your meaning.
I presume those checksum/hash algorithms in question are fully disclosed and that - since they seem to be widely used - there are good open source implementations around. And walking some directory tree, feeding files to those algorithms, accumulating information in memory and writing information out in one or the other format ain't black magic.

Maybe they wrote the juicy bits themselves, which could easily mean lots of manhours and/or clever ideas incorporated, and might be deserving protection - especially if it outperforms others in one dimension or more -, but such things I'd rather expect to come in form of dedicated libraries.

 

[HelenF]

Though I don't get, why someone would keep the source code something like that to themselves.

It's very common for these small Windows utilities to be closed source. A lot of the authors probably haven't thought about it that deeply. I got the impression there's a bit of "this is my project and I don't want people modfying it" feeling around. And sometimes if you ask someone to open-source something, they say they didn't write it with that in mind and they're ashamed of the mess.

 

[levi]

Sorry, I'm trying, but I don't get your meaning.
I presume those checksum/hash algorithms in question are fully disclosed and that - since they seem to be widely used - there are good open source implementations around.

Checksum protocols are generally written by academics, and everything they write including the reference implementation is open source by default. It's then generally tidied and prettified by the folks at OpenSSL and others, again open source but this time by intent. There's nothing stopping you reimplementing that and selling it, or even giving it away for free like this but without the source, but in it's original forms it's all open source all the way down.

That's not really the point of what I wrote. What I wrote was a slightly tongue in cheek treatise on open source politics. It's not altogether surprising you didn't get it at first inspection.

 

[Djoga'Ro]

It's very common for these small Windows utilities to be closed source. A lot of the authors probably haven't thought about it that deeply. I got the impression there's a bit of "this is my project and I don't want people modfying it" feeling around. And sometimes if you ask someone to open-source something, they say they didn't write it with that in mind and they're ashamed of the mess.

Thanks, haven't even thought of that, though I'd be among the first to understand. Then again, I'd probably need to pretty something up, I already showed to the world in whatever form, and show them also how pretty I can code - given enough time.

That's not really the point of what I wrote. What I wrote was a slightly tongue in cheek treatise on open source politics. It's not altogether surprising you didn't get it at first inspection.

Still not getting it.

 

[DrasticNerd]

You'll need also to look at the code of these libraries that are used:

/* --- Dat library includes --- */
#include "datlib/src/datlib.h"
#include "datlib/src/macro.h"


/* --- Misc library includes --- */

#include "datlib/src/misc/getopt.h"


/* --- DatUtil definitions and macros --- */

#include "datutil.h"

The source folder in the DatUtil download only has two files. The other file is empty.

It looks like all info about the job is in 'options' and init_dat is doing the work. So look at

or, if it has only the function declarations (as it should be), look for the corresponding datlib.c (or datlib.cpp maybe, if this is C++, though the code rather looks like plain C).

The source folder in the DatUtil download only has two files. The other file is empty.

What about?:

HashMyFiles: Calculate MD5/SHA1/CRC32 hash of files

HashMyFiles is small utility that allows you to calculate the MD5 and SHA1 hashes of one or more files in your system.

www.nirsoft.net

Though I don't get, why someone would keep the source code something like that to themselves.

Thanks. I do use HashMyFiles so I was keeping this as a possibility for my last option. (I could run HMF on the folder tree then convert the HTML output. That would be a major PITA to convert the HTML output to the XML dat util format.)

How's your programming at all or your object-oriented programming? With Visual Studio Code (or what the free edition's called) and utilizing some open source libraries for md5, crc32 generation and xml reading/writing it should be an easy project in C#. Or Visual Basic even.

I humbly admit that my programming power level isn't up to anywhere near the typical users of these forums. (I'm of the belief that most Pandora/Pyra owners are white collar I.T. professionals with computer science degrees!)

I am a complete failure at OOP sadly (Every time I have tried to learn a language that is object-oriented, I eventually had to quit due to the intense headaches I started getting!)

I just realized, that might not be all you want - that said tool still needs to gather more information about the files content. ?

No, not really. This is going to be generic as possible so the raw information being sought is the filename, path (relative to the folder for that game), filesize, CRC32 and MD5.

Ideally the tool would output in rom dat formats (like DatUtil does). But I'm not aware of any other tools like this.

Failing that, I would have to run a generic file checksum tool (such as HashMyFiles) then convert its output to the desired format. My programming skills are capable of doing that but it would be an ardous task to say the least. Which is why I am hoping that somebody who knows C will volunteer to fix the subdir limits imposed in DatUtil. But from the helpful feedback above, it appears to my limited understanding that the source code provided is incomplete?

 

[Djoga'Ro]

it appears to my limited understanding that the source code provided is incomplete?

From what you said, I'd guess you've got the code just for the executable, which doesn't do anything interesting - just pre-checking the command line inputs, putting those into some struct meant to hold such info, passing that struct to a function in the library (where the actual payload processing happens, including what to do with the input - the path you give being one of those inputs) and than writes the results of that to a file.

I thought there was an option to output xml, too, with HashMyFiles. ?

OOP is in no way a necessity - I'm not actually convinced that's always the right paradigm, if even often at all. I just think, if you need sth. on Windows and were familiar with OOP already, grabbing VS and making a .Net app in C# would be an easy and convinient way to go. Doing it in some scripting language would be viable, also. As long as there are libraries to do the heavy lifting one needs done like the hashing and the xml im-/exporting in this case.

 

[levi]

I don't see what's so scary about converting between a command line output or even relatively crusty html and xml. XML is just a text format with a tightly defined schema. You could write a program in perl, python, java almost any language if you wanted to do this repeatedly and relatively safely, or you could even do the conversion manually in a text editor for a one off.