So a little progress report:
Can now get serials from DLC
Can now get background images from DLC
Using last modified data seems to be going well, I use the last modified from the MD5 if it's available, otherwise I use the header data from the 0 byte call used to get the file (I'd prefer to use HEAD but for some reason it returns completely different data))
For last modified the current default behaviour is that
Installers are treated harshly, if the current update date doesn't match the old one and there's not a positive MD5 match then the installer will be marked as changed ( you can overwrite this with a flag )
Extras are treated loosely, update data is tracked (with the old update data being the oldest non-updated one) but they won't get marked as changed soley because of updated (you can overwrite this with a flag)
Clean will remove the needs to be updated tag when it orphans files
Download will remove the needs to be updated tag when a file is successfully downloaded.
Am still considering using the files Last Modified date rather than using manifest heuristics.
ETA - Since I used BGIII to test this I now get to wait ~10 hours while the maaaaybe updated Mac installers download. but it'll be nice to not have to manually track that.
ETA2 - Think I'm going to adjust how this works, new additions will be presumptively marked as "changed" so that old files with the same name get removed, verification that passes MD5s will remove the change note as will a successful import of a file, MD5 matches from the manifest will inherit the changed state of the previous file, as will a successful download. I'll provide a flag to let verify mark anything that passes as unchanged. Clean probably shouldn't mark things as clear with this change, though it should still orphan things that are marked as changed. That'll mean that only successful downloads , successful imports, verification that passes MD5 (or size and name checks with the looser flag) will clear out a changed marker.
ETA3 - The subtests went well, time to try a full manifest update (though I won't turn on strict extra checking yet, I will probably add an option that lets you set that from current manifest state while offline so that you only need to do the initial full update ( to get the update times) rather than having to do another one just to compare the updated to the last_updated )
ETA4 - Seems promising so far, of the 100 games checked so far only had 2 installer items that got marked as changed (the vast majority are new entries or have MD5s ), so the cost for that was low as I anticipated. The largest pain is probably going to be the Mac game versions that have gamedata files without MD5s that are huge, which will probably cause pain once but unless you've been managing those manually the chances they are out of date is actually ~100% .
ETA5 - Seems good from a functionality stand space. I probably need to implement a debug manifest vs standard manifest though, there's quite a bit of extra data now that's super useful for debugging but that can make for a very large manifest (my ~2600 games makes for a ~450 MB manifest , which takes ~3 minutes to load or save ).
ETA6 - Yes, I definitely need to either slim the manifest down or alter how download works, currently i have to update the manifest after each download to prevent desyching between the real state and the manifest state if things get interupted and that's far far to slow at 3 minutes per write, especially since writing to the manifest and moving files requires total locking. Hmmm, maybe I need a !provisional folder where completed but unchecked files can be temporarily moved too, then I can check that at the end of a download, and for recovery I can check it at the beginning and treat those files as if they'd been moved there. That would allow me to do only one manifest write.
ETA7 - Okay, so got the installers all synched up and verified ( Baldur's Gate III, Tropico 6 and Bard's Tale IV on Mac are by far the largest issues since these all involve huge MD5 less files, either missing MD5s for packages, Zips without MD5s or GameData files that lack MD5s and will require downloading ~300-400 GB if you have all of them, Windows lhas about 40 GB of files and Linux about 15 GB of similar MD5less installers ). Functionality works but I definitely have to fix the performance issues before this is ready for prime time. Probably won't have a chance this week thoiugh, so maybe next week before I do a public release.
Looking at my extra's that's also another potential issue I have a bit over half a TB of extra across games (though a *lot* of that is the pre 2.0 Cyberpunk installer ) and so little of that has MD5s that I don't even check for MD5s on extra so it was right call to treat these loosely.
Post edited May 07, 2024 by Kalanyr