[Online Game Stores] How to aggregate a list of all the owned games ?

[PowerGod]

Hi, in these times there are a lot of online game stores, most of them are constantly giving free or high discounted games, and at some point it could become difficult to track all the games you own, and in what store...

Every store has its unique way to show a list of all the owned games, sometimes those are only shown in a page with simple text that you can copy/paste, sometimes them can be exported in some format, but sometimes the list is visibile only on a specific client software that has no visible options to let you grab it...

I'll try to put here all the possible methods I found to grab the list of owned games from every online store.

(AGGREGATOR) GOG Galaxy

Spoiler

With this client is possible to aggregate all the games of most games stores in a single spot, so it makes you able to use a global search function.

It has anyway some issue:

1. It uses a pre-configured games database, so while a game can be added easily because its "page" is already present, not all the games (or particular editions of the same game) can be linked, and in some cases games are recognized as different ones showing a totally wrong page.
2. The integrations to the other stores sometimes could fail to synchronize correctly (or become obsolete if not official), removing from the list the games of that store
3. There is no way to export the list of games, so the only useful thing is the integrated search function.

(LIST) Epic Games

Spoiler

You can get a page of all your orders from this address, after the login.

https://www.epicgames.com/account/transactions?historyType=PAYMENT_HISTORY

The page is not meant to be exported, and you need to click the [SHOW MORE] button different times to have the complete list, but the text could be copied and managed later.

(LIST) Steam

Spoiler

From the Client or Site, in your account details there are two options, the list of what you bought and the list of all your game licenses (that contains even free and gifted games).

https://store.steampowered.com/account/

Both lists are plain text, so those can be copied and managed with a text editor.

(LIST) GOG

Spoiler

From the Site, in the Order history section you can see the list of all your games.

https://www.gog.com/en/account/settings/orders

The list is split in many pages, and while you can copy the text to put it in some editor, you'll have to repeat the process for every single page.

---------------------------------------------
ALTERNATIVE USING GOG APIs
---------------------------------------------

This solution will use both a Browser and a console, because the login session could need a CAPTCHA verification, but once we got the list of owned games it is possible to get all their info without an account login.

From the Browser login to the site as always, then:

• 1) https://embed.gog.com/user/data/games
  
  You'll get something like this:
  

     {"owned":[1,2,3,5,6,1073977251,1075740587,1084263831,1086484900,etc...

• 2) Filter from that list only the numbers to obtain something like this:
  

      1073977251
      1075740587
      1084263831

  
  From VIM you can just use this command to obtain the same:
  

  %s/\(\[\|,\)\([0-9]\+\)/\r\2\r/ge | v/^[0-9]/ :d | noh

• 3) For every ID create a link like this (the last words are options to show or not more stuff from the product) and save this list in a file called "mygogids.txt":
  

      https://api.gog.com/products/1073977251?expand=downloads,expanded_dlcs,related_products,changelog
      https://api.gog.com/products/1075740587?expand=downloads,expanded_dlcs,related_products,changelog
      https://api.gog.com/products/1084263831?expand=downloads,expanded_dlcs,related_products,changelog

  
  From VIM you can just use this command to obtain the same:
  

  %s/\(\d\+\)/https:\/\/api.gog.com\/products\/\1?expand=downloads,expanded_dlcs,related_products,changelog/

• 4) Use WGET to retrieve all the information for the games and put all of them in the same file
  

  wget -i mygogids.txt -Omygoggames.txt

• 5) Open "mygoggames.txt" and filter all the titles as you want
  
  From VIM you can just use this command to obtain an ordered list of all the titles/DLCs (consider it BETA anyway):
  

  %s/"title":"/\r/ge | %s/",.*//e | sort u | g/^{/:d | noh

  
  Example:
  

  198X
  2020 Celebration Pack
  A Plague Tale: Innocence
  A Plague Tale: Innocence - Coats of Arms DLC
  A.D. 2044

(LIST) IndieGala

Spoiler

From the Site, in the Library section you can see the list of all your games.

https://www.indiegala.com/library

The list is split in many pages, and while you can copy the text to put it in some editor, you'll have to repeat the process for every single page.

Please update me if you found a faster way to get these lists, and also add what kind of methods are needed for other stores too.

 

[levi]

In many modern browsers, you can get an inspector up by pressing ctrl+shift+i (at least in firefox). You can then drill down to see where the content it stored, although these days with modern sites that's not quite as easy as it used to be. But with judicious use of the way you can hover over a div of html and see on the page where you are or if you're off the bottom of the scrolled area, I was able to determine that you can pull out posts on this site by looking for the div classed 'bbWrapper'. That still has some html cruft around it, so passing it through html2txt might be a useful tactic, or you might be better off with a more manual process using a text editor or sed.

That doesn't help with sites with mutlitple pages. I'd check in that case if the site defined a reader view (usually indicated by a page appearing in your address bar), which sometimes helps.

 

[PowerGod]

Added a solution for GOG using APIs.

 

[gpsqueeek]

Hi there, I use Lutris on Linux, I get GOG and Humble Bundle together (It also has Steam, which I do not use). Works fine for me but I do not have a lot of games, though, not sure it would be something like that you are looking for...?

 

[PowerGod]

Hi there, I use Lutris on Linux, I get GOG and Humble Bundle together (It also has Steam, which I do not use). Works fine for me but I do not have a lot of games, though, not sure it would be something like that you are looking for...?

My main purpose is to have a text list, so I can choose in witch way to manage it.
I have too many games in too many stores, so at least I want a fast way to check for duplicates, or what I actually already have instead of buying something again in another store.

I suppose Lutrix uses APIs of the managed Stores to get lists, right ?
Those lists, once obtained, are put in a DB of some sort or maybe there are some kind of exports in other formats ?
As I see from the site, Lutrix have a DB of configurations for games, what happens to the games that are not on that DB ? Them are not managed/shown on the client ?

 

[gpsqueeek]

My main purpose is to have a text list, so I can choose in witch way to manage it.
I have too many games in too many stores, so at least I want a fast way to check for duplicates, or what I actually already have instead of buying something again in another store.

I suppose Lutrix uses APIs of the managed Stores to get lists, right ?
Those lists, once obtained, are put in a DB of some sort or maybe there are some kind of exports in other formats ?
As I see from the site, Lutrix have a DB of configurations for games, what happens to the games that are not on that DB ? Them are not managed/shown on the client ?

I have no idea about the technical implementation but I can see the available games even if they are not installed (they are greyed out in the list)

 

[PowerGod]

I found a more "simple" way to get the list of game from the Epic Store...

There is still the need to go to the account and payment history, but instead of clicking the MORE button over and over to show the entire length of the page, it is possible to open another TAB in the browser and using this command:

https://www.epicgames.com/account/v2/payment/ajaxGetOrderHistory?&page=1&locale=en

It will return some lines of JSON code containing like 10 orders with dates and name of the games... now just changing the "page=" number on the link you can get the next 10 lines, and so on... once you got the text of all those pages, it is possible to manage it with a text editor...

I'm still playing with bash to see if I can automate this from a terminal... but there's always the issue of the initial cookie login that seems to need a browser...

EDIT:

There's a way to use chromium-like cookies with wget !!
They must be modified in a Netscape (or probably Firefox) format to be read, and there's a chrome extension called "Get cookies.txt" that does just this !!

So now there's only the need to login once from the browser on the Epic Store, export on a txt file the session cookies (and with that extension is just 2 clicks), and finally point the cookie file to wget.

wget --load-cookies=epicgames.com_cookies.txt "https://www.epicgames.com/account/v2/payment/ajaxGetOrderHistory?&page=1&locale=en" -Omyepicgameslist.txt

Now the only part missing is a good automation to get and aggregate all the pages...

EDIT2:

nope... not working... the link needs also some kind of timestamp that I was thinking optional... but instead seems essential to generate the right requested page...

https://www.epicgames.com/account/v2/payment/ajaxGetOrderHistory?page=11&lastCreatedAt=2021-04-29T16%3A37%3A04.276Z&locale=en

that "lastCreatedAt=" seems to contain the date present in the last order of the previous page... so this will make things really complex...

EDIT3:

I'm almost there... I was able to obtain a long list of things, but it's difficult to compare them with the games listed on the client... at least with the list view the client can tell me the total games I have (excluding DLCs)... but now the difficult (if not impossible) part is to find what orders are related to DLCs and what are related to the base games... I can't find anything that could point out that state....

Anyway this is the script that will get and aggregate all the JSON of every pages in the account orders:

#!/bin/bash
#
# Script to get the list of games owned on the Epic Store
# v.0.2a - PowerGod (2022)
#
# REQUIREMENTS:
#  - a previous connection from a Browser to the Epic Store account
#  - export of the session cookies (in a Netscape format) in the working folder
#  - correct the "locale=" in "pageSiteDefault" variable
#  - change the "lastPageList" variable to limit the downloaded pages 
#    (the script will terminate automatically if it find an empty list page)
#
# TODO:
#  - use the namespace to find the relations from base game and DLC ???
#

cookieDefaultName=epicgames.com_cookies.txt
listDefaultName=myepicgameslist
listFinalName=$listDefaultName.txt
wgetLog=wget-log.tmp
pageSiteDefault=https://www.epicgames.com/account/v2/payment/ajaxGetOrderHistory?locale=it
lastTimeStamp=2999-12-31T01:01:01.001Z
lastPageList=100
lastStart=0
OrdersTotal=0

rm $wgetLog

echo ===============
echo EPIC STORE LIST
echo ===============

for (( i = 1; i <= $lastPageList; i++ )) 
do 
    # UrlEncoding TimeStamp
    lastTimeStamp=${lastTimeStamp//:/%3A}
    # Downloading the page with the list starting from page 0
    echo "Downloading PAGE $i (of $lastPageList MAX)"
    # The first page request must be "neutral", the next pages instead
    # will require more parameters and a date to start with
    if [ "$i" == "1" ]
    then
        echo "$pageSiteDefault"
        wget --load-cookies=$cookieDefaultName "$pageSiteDefault" -O$listDefaultName-p$i.tmp -a$wgetLog
        # Getting the total orders from the first page
        OrdersTotal=$(sed 's/"total":/\n"total":/g' $listDefaultName-p$i.tmp | grep '^"total' | sed 's/^"total":\([0-9]\+\).*/\1/' | sort -nr | grep '^[0-9]\+' -m1)
    else
        echo "$pageSiteDefault&page=$(( i - 1 ))&lastCreatedAt=$lastTimeStamp"
        wget --load-cookies=$cookieDefaultName "$pageSiteDefault&page=$(( i - 1 ))&lastCreatedAt=$lastTimeStamp" -O$listDefaultName-p$i.tmp -a$wgetLog
    fi
    # Generating the last TimeStamp from the downloaded page:
    #  - getting all "createdAtMillis" milliseconds
    #  - removing the last 3 digits    (that seems to give issues with the conversion)
    #  - ordering the list to get only the older one
    #  - converting the result to the wanted DateTime format UTC (still not UrlEncoded)
    lastTimeStamp=
    lastTimeStamp=$(date --date='@'$(sed -r 's/"createdAtMillis":/\n/g' $listDefaultName-p$i.tmp | sed -r 's/^([0-9]+).*/\1/g' | sed 's/[0-9]\{3\}$//' | sort -n | grep '^[0-9]\+' -m1) +%Y-%m-%dT%T.001Z -u)
    #echo $lastTimeStamp
    # Getting the first order number in the page
    lastStart=$(sed 's/"start":/\n"start":/g' $listDefaultName-p$i.tmp | grep '^"start' | sed 's/^"start":\([0-9]\+\).*/\1/' | sort -nr | grep '^[0-9]\+' -m1)
    echo "- ORDERS FROM $lastStart of $OrdersTotal"
    # if TimeStamp empty or lastStart surpasses total orders, most probably we are at the last page
    if [ "$lastTimeStamp" == "" ] || [ $lastStart -ge $OrdersTotal ]
    then 
        echo LAST PAGE $i HAD NO ENTRIES OR SURPASSED THE TOTAL ORDER COUNTER, EXITING SCRIPT
        i=$(( lastPageList + 1 ))
    fi
done

# Merging all the pages in a single file
cat $listDefaultName-p*.tmp > $listFinalName
# Extracting and formatting only useful data
# DESCRIPTION / NAMESPACE / OFFERID
# and sorting by DESCRIPTION
sed -e 's/"description":"\([^"]*\)"/\n<DESC> \1\t/g' $listFinalName | sed -e 's/^\(.\+\t\).*"offerId":"\([^"]*\)"/\1\2\t/g' | sed -e 's/^\(.\+\t\).*"namespace":"\([^"]*\)".*/\1\2/g' | grep "^<DESC>" | sed -e 's/^<DESC> //g' | sed -e 's/^\([^\t]*\t\)\([^\t]*\)\t\(.*\)/\1\3\t\2/g' | sort > $listFinalName

# (to find out what products are aggregated, like base games with DLCs, a SORT for NAMESPACE will be required after the script)

rm $listDefaultName-p*.tmp

echo ===============
echo Epic Store List updated in "$listFinalName"
echo ===============

 

[PowerGod]

Damn... I'm so close to get it... but still something is missing...

From the JSON downloaded before I found out that the "namespace" hash can be used to find out all the stuff related to the same game, so I can obtain a list like this, sorted by "namespace" hash and "offerId" hash (that is the single product Id):

Total War: WARHAMMER - Assembly Kit    0c321ac6d5fe49139c153253b60702f4    0f8a1b035e2a416bb0ee10b522590e82
Total War: WARHAMMER - Wurrzag    0c321ac6d5fe49139c153253b60702f4    10d73f7c4ba14883a796cdca8891bec5
Total War: WARHAMMER - Grey Wizard    0c321ac6d5fe49139c153253b60702f4    59841368649549a5b3aa36510895383b
Total War: WARHAMMER - Bretonnia    0c321ac6d5fe49139c153253b60702f4    7dcd7b53544c4b66bde5efbdc8789c3d
Total War: WARHAMMER - Jade Wizard    0c321ac6d5fe49139c153253b60702f4    aa3caef38b7c46f68b10665eb401b19e
Total War: WARHAMMER - Isabella von Carstein    0c321ac6d5fe49139c153253b60702f4    adb4782630b94ef483aca95c77de0ced
Total War: WARHAMMER    0c321ac6d5fe49139c153253b60702f4    bb3ce2b4668b4b22aae989f4508f0b71
Total War: WARHAMMER - Grombrindal The White Dwarf    0c321ac6d5fe49139c153253b60702f4    ed5421d3433c4775afc5fae130718a0b
Metallic Equipment Set    0dc18227b3f14162968d8e7969e454ef    3e8b3d42d46249ebbe4e370278972b0e
Prototype Arsenal Set    0dc18227b3f14162968d8e7969e454ef    e9c3e34d31364233805244cc5a9a7325
DAEMON X MACHINA    0dc18227b3f14162968d8e7969e454ef    f6d665930cc34c34ac69cfe52906a723

Still there is nothing that can be used to understand what product is a DLC and what is the base game...

In the example before, the item with description "Total War: WARHAMMER" (without " - something else") is easily identifiable as the base game...
there is anyway another game that follow another logic, "DAEMON X MACHINA" is the base game, while "Prototype Arsenal Set" and "Metallic Equipment Set" are DLCs...
with this second example there is no simple way to "remove DLCs from the list" automatically... it is possible to keep just one of the lines containing the same namespace, but doing it is also possible to lose the main description of the game and getting just one of the DLC lines...

maybe there are around other APIs where I could get more info on the single products...

 

[PowerGod]

After some digging seems like the Epic Store still doesn't have documented Web API to query the store... this maybe is also the reason why they still don't have a Web Store interface...

Also I found out other two things:
- the namespace of "Borderlands 2" is different for every DLC... so there is absolutely no visible connections with the main game...
- some games I have in library did not generate the related order when I added them... so from the account information page not all the owned games are listed, but only the ones that generated an order...