[SCRIPT] CLI script to get NO-INTRO DAT lists


PowerGod

Forum Addict!
Joined
Jun 20, 2011
Messages
4,417
EDIT: tl;dr the script is in the second post.


I'm trying to get DAT files from the NO-INTRO site using command line...


The site needs both GET and POST requests, the GET for going in the specific "Standard DAT" page, and the POST to select the system for the list.

This is the "Standard DAT" page:
Code:
https://datomatic.no-intro.org/index.php?page=download&op=dat

A command like this is returning the "Standard DAT" page, but not for the system I specified:
Code:
wget --keep-session-cookies --no-check-certificate --post-data="sel_s=Sega+-+Game+Gear"  "https://datomatic.no-intro.org/index.php?page=download&op=dat"  -O result.html

The post-data string should be correct, I checked both from the browser logs and also fiddler what the browser was sending after selecting another system...

After that I should also send another POST command because "Prepare" has to be pressed to get the file, but I'll look at this after the first issue is solved.

Do you know where could be the problem ?

EDIT:

Somehow I solved the first issue like this:
Code:
wget \
--load-cookies cookies.txt \
--save-cookies cookies.txt \
--keep-session-cookies \
--no-check-certificate \
--user-agent="Mozilla/5.0 (Windows NT 6.3; Win6>4; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.132 Safari/537.36" \
--post-data="sel_s=Sega+-+Game+Gear"  \
"https://datomatic.no-intro.org/index.php?page=download&op=dat"  \
-O result.html
Now, "result.html" contains the form to select the data to list for, in this case, the Game Gear.

The second request to compile the form and to get to the download page works too:
Code:
wget \
--load-cookies cookies.txt \
--save-cookies cookies.txt \
--keep-session-cookies \
--no-check-certificate \
--user-agent="Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.132 Safari/537.36" \
--post-data="inc_complete=0&inc_unl=1&inc_pirate=1&inc_physical=0&special1_filter=all_specials1&language_filter=all_languages&region_filter=all_regions&prepare_2=Prepare"  \
"https://datomatic.no-intro.org/index.php?page=download&op=dat"  \
-O result.html

Now, this is the point where I am stuck...

With the browser I see that after confirming the form in the previous page, the address changes and doesn't have anymore the same GET parameters as before:
Code:
https://datomatic.no-intro.org/index.php?page=manager&download=9113

To download correctly the file, I need to instruct WGET to catch those "page and download" parameters, but I don't know how to do it...

EDIT:

I found a way to extract the address from the log file, but i don't like it very much... anyway it works...

I added the parameter " --append-output=LOG_FILE", and then
Code:
DOWNLOAD_URL=$(grep "^--" LOG_FILE | tail -n1 | sed 's/.*http/http/')

So I can use this URL in the last wget request.
 
Last edited:
The script is pretty functional now, if someone is interested I put it here:


Code:
$ ./nointro-DAT.sh

=======================
NO-INTRO LISTED SYSTEMS
=======================

Atari - 2600
Atari - 5200
Atari - 7800
Atari - Jaguar
Atari - Lynx
Atari - ST
Bandai - WonderSwan
Bandai - WonderSwan Color
Benesse - Pocket Challenge V2
Benesse - Pocket Challenge W
Casio - Loopy
Casio - PV-1000
Coleco - ColecoVision
Commodore - 64
Commodore - 64 (PP)
Commodore - 64 (Tapes)
Commodore - Amiga
Commodore - Plus-4
Commodore - VIC-20
Emerson - Arcadia 2001
Entex - Adventure Vision
Epoch - Super Cassette Vision
Fairchild - Channel F
Fukutake Publishing - StudyBox
Funtech - Super Acan
GamePark - GP32
GCE - Vectrex
Hartung - Game Master
iQue - iQue
Konami - Picno
LeapFrog - LeapPad
LeapFrog - Leapster Learning Game System
LeapFrog - My First LeapPad
Magnavox - Odyssey2
Mattel - Intellivision
Microsoft - MSX
Microsoft - MSX2
Microsoft - XBOX 360 (Digital)
Microsoft - XBOX 360 (Title Updates) (Discontinued)
Mobile - J2ME
Mobile - Palm OS
Mobile - Symbian
NEC - PC Engine - TurboGrafx 16
NEC - PC Engine SuperGrafx
Nintendo - e-Reader
Nintendo - Family Computer Disk System
Nintendo - Game Boy
Nintendo - Game Boy Advance
Nintendo - Game Boy Advance (Multiboot)
Nintendo - Game Boy Color
Nintendo - New Nintendo 3DS
Nintendo - New Nintendo 3DS (Digital)
Nintendo - Nintendo 3DS
Nintendo - Nintendo 3DS (Digital)
Nintendo - Nintendo 3DS (Digital) (CDN)
Nintendo - Nintendo 3DS (Digital) (Updates and DLC)
Nintendo - Nintendo 64
Nintendo - Nintendo 64DD
Nintendo - Nintendo DS
Nintendo - Nintendo DS (Download Play)
Nintendo - Nintendo DSi
Nintendo - Nintendo DSi (Digital)
Nintendo - Nintendo Entertainment System
Nintendo - Pokemon Mini
Nintendo - Satellaview
Nintendo - Sufami Turbo
Nintendo - Super Nintendo Entertainment System
Nintendo - Virtual Boy
Nintendo - Wii (Digital)
Nintendo - Wii (Digital) (CDN)
Nintendo - Wii (Digital) (Split DLC)
Nintendo - Wii U (Digital)
Nintendo - Wii U (Digital) (CDN)
Ouya - Ouya
Philips - Videopac+
RCA - Studio II
Sega - 32X
Sega - Beena
Sega - Game Gear
Sega - Master System - Mark III
Sega - Mega Drive - Genesis
Sega - PICO
Sega - SG-1000
Sinclair - ZX Spectrum +3
SNK - Neo Geo Pocket
SNK - Neo Geo Pocket Color
Sony - PlayStation (PS one Classics) (PSN)
Sony - PlayStation 3 (PSN) (Decrypted)
Sony - PlayStation 3 (PSN) (Encrypted)
Sony - PlayStation Portable
Sony - PlayStation Portable (PSN) (Decrypted)
Sony - PlayStation Portable (PSN) (Encrypted)
Sony - PlayStation Portable (PSX2PSP)
Sony - PlayStation Portable (UMD Music)
Sony - PlayStation Portable (UMD Video)
Sony - PlayStation Vita
Sony - PlayStation Vita (PSN) (Decrypted)
Sony - PlayStation Vita (PSN) (Encrypted)
Tiger - Game.com
Toshiba - Visicom
VTech - CreatiVision
VTech - V.Smile
Watara - Supervision
Zeebo - Zeebo

-------
Use the name of a system (a perfect match is not required) to download the relative DAT:
./nointro-DAT.sh "game gear"

Code:
$ ./nointro-DAT.sh "game gear"
===
DOWNLOADING "Sega - Game Gear" DAT LIST
===                                    

$ ls
'Sega - Game Gear (20200302-200516).zip'
 
Last edited:
Small warning: It seems to move and rename your files so you might want to clone your Games folder before you give it a go.
 
Small warning: It seems to move and rename your files so you might want to clone your Games folder before you give it a go.

This scripts (nointro-DAT.sh) just downloads the DAT list, don't rename anything (except maybe if you download twice the same list).

Maybe you are speaking about the other one in the git, "aunmess.sh", that will indeed rename and move EVERYTHING, it is intended to solve situations where ROMS collections are so messy that you don't even think to try solving by hand.
(And were even tools like clrMamePro/RomCenter don't have even nearly a chance to success.)
So consider that script the "chemotherapy" for ROM folders.

In some future they will be connected somehow, to become a single tool, but the entire chain is not ready yet.
 
Last edited:
Back
Top