It seems having only a couple of http retries is not enough? These are the settings that I am using now, just giving GOG much more breathing room with a minute long http_retry_delay (instead of hitting GOG every few seconds if it decides to not want to co-operate for a moment), but I also increased the retry count very high because it seems something like 3-5 is not necessarily enough, as I can see from the log below? (I increased the fetch delay which I guess makes the whole update process slower, but hoping it helps GOG also to co-operate.)
# HTTP request settings
HTTP_FETCH_DELAY = 3 # in seconds
HTTP_RETRY_DELAY = 60 # in seconds #If you reduce this such that the wait between the first and third try is less than 10 seconds, you're gonna have a bad time with the 503 error.
HTTP_RETRY_COUNT = 99
HTTP_TIMEOUT = 60
From the log. What is the difference between 500 Server Error and "Connection aborted"? 500 is a generic error that something went wrong (reason unknown, but not meaning the server is unreachable?), while "connection aborted" means the server actively refused the connection? Does it seem below that gogrepoc was able to get the needed data, after enough retries; ie. no title/file was missed?
07:40:33 | (2195 / 2583) fetching game details for the_humans_3...
07:40:43 | request failed: 500 Server Error: Internal Server Error for url:
https://www.gog.com/account/gameDetails/1275264927.json (99 retries left) -- will retry in 60s...
07:41:50 | request failed: 500 Server Error: Internal Server Error for url:
https://www.gog.com/account/gameDetails/1275264927.json (98 retries left) -- will retry in 60s...
07:42:59 | request failed: 500 Server Error: Internal Server Error for url:
https://www.gog.com/account/gameDetails/1275264927.json (97 retries left) -- will retry in 60s...
07:44:06 | request failed: 500 Server Error: Internal Server Error for url:
https://www.gog.com/account/gameDetails/1275264927.json (96 retries left) -- will retry in 60s...
07:45:13 | request failed: 500 Server Error: Internal Server Error for url:
https://www.gog.com/account/gameDetails/1275264927.json (95 retries left) -- will retry in 60s...
07:46:39 | request failed: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')) (99 retries left) -- will retry in 60s...
07:47:59 | request failed: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')) (98 retries left) -- will retry in 60s...
07:48:59 | (2196 / 2583) fetching game details for the_i_of_the_dragon...
07:49:13 | request failed: 500 Server Error: Internal Server Error for url:
https://www.gog.com/account/gameDetails/2000520059.json (99 retries left) -- will retry in 60s...
07:50:23 | request failed: 500 Server Error: Internal Server Error for url:
https://www.gog.com/account/gameDetails/2000520059.json (98 retries left) -- will retry in 60s...
07:51:35 | request failed: 500 Server Error: Internal Server Error for url:
https://www.gog.com/account/gameDetails/2000520059.json (97 retries left) -- will retry in 60s...
07:52:43 | request failed: 500 Server Error: Internal Server Error for url:
https://www.gog.com/account/gameDetails/2000520059.json (96 retries left) -- will retry in 60s...
07:53:56 | request failed: 500 Server Error: Internal Server Error for url:
https://www.gog.com/account/gameDetails/2000520059.json (95 retries left) -- will retry in 60s...
07:55:09 | request failed: 500 Server Error: Internal Server Error for url:
https://www.gog.com/account/gameDetails/2000520059.json (94 retries left) -- will retry in 60s...
07:56:41 | request failed: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')) (99 retries left) -- will retry in 60s...
07:58:00 | request failed: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')) (98 retries left) -- will retry in 60s...
07:59:01 | (2197 / 2583) fetching game details for the_immortal...
07:59:06 | (2198 / 2583) fetching game details for the_incredible_adventures_of_van_helsing_complete_pack...
07:59:10 | (2199 / 2583) fetching game details for the_incredible_adventures_of_van_helsing_final_cut_game...
07:59:18 | request failed: 500 Server Error: Internal Server Error for url:
https://www.gog.com/account/gameDetails/1448013298.json (99 retries left) -- will retry in 60s...
08:00:31 | (2200 / 2583) fetching game details for the_incredible_adventures_of_van_helsing_ii_complete_pack...
08:00:36 | saving manifest...
08:00:39 | saved manifest
08:00:39 | saving resume manifest...
08:00:40 | saved resume manifest
08:00:40 | (2201 / 2583) fetching game details for the_incredible_adventures_of_van_helsing_iii...