Could you let me know if it still gives errors after you cllean up the full disk ? I've had previous reports of the Logging module misbehaving with remote drives and since I notice you're using a server, it may also be that. If it is, I'd appreciate any errors you get , so I can try and work out exactly what's happening.

It doesn't use temp for the logging, right now. I deleted a file on my desktop worth 6 gigs, did the clean, deleted a few files, and that was enough to fix it. Just got your message after doing all that. I need a properly larger HD to dedicate to gogrepo, honestly. I have too much going on on this one.

GOG takes up the most space, but I have alot stored on the server.

I know the feeling, I have a 5TB disk I use for gorepo. If you've done a full manifest update , clean , download and verify (and the verify verifies everything successfully) , you should probably be able to download the entire !orphaned folder. Though you might want to check through for stuff like removed manuals and such first (I've never seen GOG remove anything else but manuals without replacing them, and even that's usually because the manual is outdated but better safe than sorry).

Exactly. I'm thinking about adding something to the script to auto check for any files that are not .exe or .bin and alert me when this happens.

That might not be a bad idea to add to the main script, a delete or trash command that cleans up !orphaned with an option to keep non installer files (so stuff that isn't bin / exe / sh / dmg ).

I should probably add a command to delete everything in the !downloading folder for the rare occassion where that's an issue too or maybe have clean run throughr !downloading too and remove any files that aren't present in the manifest since other issues would resolve themselves.

Eh, honestly, who can use gog-repo but not "rm -R \!downloading"?

To be fair , the number of times you should have to remove !downloading is ~0 , it largely takes care of itself except in the case where GOG changes the file name in the middle of a download or a download is cancelled / interrupted part way through and the file name is changed before the download is resumed again.

It makes sense more to just delete it automatically at the end or something if it ever becomes a legitimate issue.

I'm curious, what causes the bottlenecks we're seeing when creating the initial manifest?

What bottleneck, sorry ?

I'm just saying if you start out fresh, it takes a while. You've done alot to reduce how often those slowdowns occur, but i'm curious what exactly it does when it's looking up game info and putting it into the manifest that's taking so long. In the current state, it's not an issue, but it does have me curious.

Lots of HTTP requests, it gets your account page, then each page of games, then after it knows all your games, it queries each game page, finds out what's listed on it, then fetches 0 bytes of each item (to get the real URL address of the item), and then fetches the checksum data for each item where it's available. It also does some processing to eg separate out games / extras , filter languages / os , remove duplicates, recognize if an item / game is the same as an existing one in the manifest and remember tt update their name on the next clean/download if the file alrady exists . GOG also seems to have put in a bit of a delay, it was much faster for a short time after I switched over to using requests, then it got much slower but still faster than it was before I used requests.

Is there a way to find out how big all my games are together without starting to download them? I would like to know how much disk space I need for a full backup.

I'm honestly surprised gog bothers with these half-baked urls. If they set the permissions right to begin with, they wouldn't have to worry about all that.

But, the delay doesn't surprise me. All those empty HTTP requests would put a huge strain on the servers (someone DDoSed my network by overloading my HTTP server with empty requests, actually: he called it "Sloloris" or something to that effect). IMO, that's their own fault, though. That doesn't have to be at all.

If they returned the correct error or had the approaprite metadata on their page headers which should indicate the desired timeout / and number of requests per hour respectively, I'd setup the script to respect it. But they don't, the error they return is the one for if an ISP is cutting off a site for having served too much data. I was actually going to rry and work out what GOG's per hour limit was by when the site started throwing 403 errors and then respect that, just before they added the delays.

I'm honestly surprised they haven't outright 403ed based on the user-agent string altogether. I get the impression that they don't like gogrepo, but don't really now how to get rid of it.