I can almost garuntee those are linux partitions, its on Western Digital PR4100. Been running gogrepo from my computer since my last post (as opposed to running from my NAS) seems to be working normally. It may not have liked me executing the python script from a remote path.

Edit: How I was running it orginally was in wndows I setup an environment path to \\192.168.1.10\Utilities\gogrepo\ so that when I run gogrepo.py it always starts in the remote path and then sends all downloads to \\192.168.1.10\Lab\Games\GOG. Only thing i changed is where the gogrepo.py was executed (C drive now)

That could well be the issue , I haven't done any testing on Linux formatted drives accessed from Windows or on remote paths using network addressing.

I am more inclined to say running it from a network source may have been introducing unintended variables, I have been downloading to a network share for over a year with no issues. What I recently switched to was running the script from a network path (so I can run the app from anywhere essentially) that seems to have introduced enough latency or some sort of permission issue IDK which but that is fine I will just run it from my computer from now on.

Thank you for your help

Is it possible to add a download limit to gogrepro.py, or to do this in a Powershell script that calls gogrepo.py?

The reason I ask is because occasionally when I do my monthly library update the downloads exceed my GB limit and I end up having to pay my ISP extra. This is exacerbated by GOGs obsession with repackaging large numbers of installers (often with no new content) such as just happened with the past week or two (I now have 55 updates pending and I dread to think how many GB of downloads that will turn out to be). What I would like to be able to do is to run gogrepo.py but specify a cap on GB. So, I would run a partial update this month and the rest next month after my usage has been reset to zero again by my ISP.

I know I can do the updates on a game by game basis but with 55 updates that is really tedious having to do each one manually.

It is possible but the measure of bandwidth used would only be approximate like the download speed is. It wouldn't be wildly inaccurate but nor would it be megabyte perfect. If that still sounds useful I'll put it on the to do list.

One problem with that approach probably is that many of the files you are postponing to download next month have have possibly changed again, so you may need to at least run update all over again. And when you do that, then you may end up redownloading some of those already downloaded last month, as they have also changed again.

I guess there is no way around that problem, but just a warning so that you don't wonder. In the past I have sometimes had problems like if I ran gogrepo update first, and then used it to download the games a couple days later, some of the entries in the manifest were not valid anymore (can't be found on GOG servers anymore). So usually it is a good idea to run update and download quite close to each other, even on the same day.

Anyway, until Kalanyr adds this data limitation to the script, one manual way to approximately do what you are seeking to do is to edit the manifest file manually. So if I were you, I'd possibly do something like this:

1. Run a full gogrepo.py update to get a manifest file.

2. Run gogrepo.py download -dryrun in order to see what would be the total amount of data to be downloaded. Then estimate roughly how much of that amount of data you'd like to download this month, let's say that you feel downloading half of that data would be fine.

3. Make a backup copy of the original gog-manifest.dat, and then edit gog-manifest.dat (if you are in Windows, I highly suggest you install e.g. Notepad++ and use it to edit the file). Remove certain amount of games from the end, let's say remove all the games which start with O, and anything after that. Like the first game to remove being "Oddworld: Abe's Exodus" etc.

Pay close attention to the formatting at the end and start of the manifest file that you have all the needed square brackets at the end (and start) etc. That's also why you should keep a copy of the original manifest file, in case you make an editing error.

General instruction for editing, since the manifest file may have so many lines (depending on the amount of games you have): Shift + Ctrl + End selects everything from the current point till the end of the file, while Shift + Ctrl + Home selects everything from the current point to the start. And when you have done either (still Ctrl+Shift pressed down), just release Ctrl (still keeping Shift pressed) that you can fine-tune your selection with arrow keys, like not removing the square and wave brackets from the end of the manifest file.

4. Save the edited manifest file, and run gogrepo.py download -dryrun again. If the amount of data to be downloaded sounds ok, run download without -dryrun. If it is still too much, edit the manifest file again, removing more games.

5. Then next month you edit the manifest file by removing games from the start of the list, like trying to download only games from N to Z. Depending of the amount of data, you may have to do that in several parts, not only two.


I am routinely doing pretty much this because I keep my GOG installers on two separate 2TB USB hard drives, and the amount of data already exceeds 2TB. So I download games # => N to one USB hard drive, and O => Z to another, by editing the manifest file and running gogrepo twice.


I am wondering though, aren't there really any Windows utilities and such which would calculate the amount of data transferred, and cut all internet connectivity when that is reached? Or can such be configured even in the router? It sucks though that your ISP automatically starts charging extra if you go beyond the data cap; isn't it more common that they just slow your net connection to minimum when the limit is hit, for the rest of the month?

EDIT: Maybe some hints:

https://www.howtogeek.com/192654/how-to-monitor-your-internet-bandwidth-usage-and-avoid-exceeding-data-caps/

https://www.netlimiter.com/

Netlimiter costs money, and I didn't quite get from its feature descriptions if it is possible to block an app or even all internet connectivity based on amount of data transferred. If there was, this would be a more generic solution to your data cap problem, not only for using gogrepo.

I had trouble running this but eventually got it to work so just going to throw out my experience so others searching might have luck figuring out their own problem if mine covers it (and for myself when I inevitably forget the next time I try to run it). Things in quotes "" are just commands for cmd.exe or .bat files and the "" aren't included. It doesn't seem like the forum has much in the way of formatting options so if you have an idea how to make this more readable I'm all ears.

Went to python.org and installed Python 2.7.14 for Windows.
Tried to run it but got errors about required modules.
Had to search how to install the required modules (I already had 3.5 installed as the main version so I had to go to cmd.exe and use "C:\Python27\python.exe -m pip install insertmodulehere" for html5lib, requests, requests[security] (it gave an error for OpenSSL which is apparently resolved by installing requests[security], alternatively you can install pyOpenSSL individually instead because r[s] installs two other unneeded packages) and html2text (optional but figured I might as well while I was at it).
Created .bat files for each step (start as .txt files and rename to .bat) to run it.
The .bat files wouldn't work because of spaces. Searched and tried double quotes but couldn't get that to work. Instead, in cmd.exe, I used "dir /X ~1 c:\" where I replaced c:\ with the directory containing the folder with the spaces. It gave an alternative file path without spaces (sample ex: C:\Program Files becomes C:\Progra~1) which I used in the .bat files
The .bat files were just "C:\Python27\python.exe F:\FolderNameWithSpace~1\gogrepo.py insertcommandhere" and then a line below with only "pause" (which makes the cmd.exe window stay open so you can read the error). The commands I used (based on some website and skimming the documentation to confirm) were "gogrepo.py login", "gogrepo.py update -os windows linux mac -lang en" (but I removed linux/mac), "gogrepo.py import . ." (if in current directory, checks existing files so you don't download same file multiple times), and "gogrepo.py download -os windows linux mac -lang en" (but I removed linux/mac). "gogrepo.py verify" is good afterwards to check for bad files.
Tried logging in with the .bat file and kept getting 'login failed, verify your username/password and try again.'
Still have no idea why that doesn't work but searching brought an alternate method using firefox, logging in to GOG.com and using a cookie exporter that specifically converts to Netscape/IE format.
I put cookies.txt in the gogrepo folder and skipped the login command to do the update command.
Now I got the error 'failed to load product data (are you still logged in?)'
It turns out the option to keep cookies 'until I close Firefox' (in firefox options) made the cookies unusable for gogrepo. Changing to keep 'until they expire' makes the cookies have long expiration dates so gogrepo works.
Ran the update command again (worked with the new cookies.txt), import command (it found nothing because I had nothing already downloaded) and then the download command.

Edit: Oops, forgot to mention update the second time. Thanks.
Edit2: Apparently, the download command defaults to windows+eng if you don't add the arguments (despite the update manifest and isn't mentioned in the current readme). May be changed in the future. Also, the example (post1390) Kalanyr gives adds those arguments to the verify command. Maybe needed there too? Also, apparently there's "gogrepo.py clean" that moves files it doesn't recognize (aka older ones) to the !orphaned directory for later review. Verify apparently has a -clean argument, maybe it's similar?

You need to use the update command to build a manifest so that downloading will work.

Thanks, that would be a really useful feature to ensure I never exceed my data cap again.


Splitting the manifest is a great interim solution - thanks for the suggestion.

Also, while waiting for Kalanyr, I may try and add this data limitation to my Powershell script that I use to do my updates.

I've just made a minorish update that corrects a typo in logging when a download did not complete successfully and that resolves the issue of the renaming loop when a game has identical installer files across multiple languages (if this applies to you , you will likely see an increase in your download size on the next update of such games as this is a temporary work around that simply treats files that don't share a language as always different even if the share the same MD5)

I've decided that the next thing I should do is update the read me and then make a new thread to avoid confusion.

I take it I'll get the outdated version if I follow the links in the first post, what's the link for the currently maintained version?

Many thanks!

A possibly relevant question: In addition to the dependencies specified in the readme, I also had to install requests and openSSL (the latter of which confused me for a while as python -m pip install openssl wouldn't work). I've got it working now, but my question is:

Are these things that need adding to the readme, or should I not have had to manually install them?

EDIT:

Hmm, I get this when doing the manifest Is that a fault at GOG's end or a bug in the script that I should report?

There's also

requests need to be added to the readme. I only just realized pyOpenSSL wasn't a built in, when you posted this and I checked, I must have already had it installed it for a previous project and forgot about it. I'll include it in the readme too, thank you for pointing that out. It would have gone unnoticed previously as it was used implicitly on my system by requests and I only made it explicit to handle some improperly caught errors in requests.

Yes, the first of those is an error by GOG , they haven't provided MD5 data for that file. The 2nd is GOG side but not really an error as such, it's an item promised as part of Kickstarter that hasn't been provided to GOG (usually such things have ended up on Backerkit or the companies own websites, some of them just haven't happened).