The ones I know GOG has in its store are "vampire the masquerade bloodlines", and "Requiem: Avenging Angel"
to be specific I am talking about games where the player character can use magic and guns and at least some of the enemies in the game are monsters

Torchlight 1 and 2

Grim Dawn

Shadowrun series.

Finding the line between magic and sci-fi can be tricky in a few cases (Borderlands and Fallout slide just over into sci-fi but otherwise fits).

A few of the Might and Magic games had guns, as I recall. WoW has guns (though I think just hunter uses them now?). Van Helsing as I recall has guns...victor vran has a lightning gun at least.

Just off the top of my head.

Probably a really long list if someone actually compiled it.

Clive Barker's Undying has guns, magic and monsters.

Enter the Gungeon. Sword-and-board dungeon crawling except now it's... guns-and-'nades with a sci-fi flavour? Lots of guns though... Even the enemies are guns. Except when they're gun-themed versions of classic fantasy monsters. :D

It's got a steep learning curve. And not an FPS. But it's really fun.

Battlechasers Nightwar

Arcanum: Of Steamworks and Magick Obscura

Arcanum for the win! \o/

There's also, Hard West, sort of.

I second Grim Dawn.

Terraria

Wizardry 7 and 8.

Also, if you don't restrict yourself to PC games, some of the SaGa games (specifically the Game Boy SaGas (Final Fantasy Legend 1-3), their remakes, and SaGa Frontier (not SaGa Frontier 2)) would qualify. All of those allow for all 3 of them to be used by your party (yes, that means you can get monsters in your party if you so choose).

Yeah, if you like old-school RPG/tactics games then get these right away. They have a pretty unique mix of cyberpunk setting and fantasy aspects, magic and monsters and elves and whatnot. They're very well written and presented, and the tactical battles are good, but they do suffer from a smaller budget and scope.

Not sure, but doesn't Pillars of Eternity have guns and magic in it ?

Early guns like muskets yeah. Fable 2 and 3 had those too I believe. Also Risen 2 and 3.