I finished all the quests related to Gwent (High stakes, Skeilige Style and all that stuff).

The innkeepers become a bit too easy now.. is there a point in continuing? I mean is there gonna be a good challenge somewhere else?

You can always play with fractions you haven't played so much before. Can be more challenging.

I do think though that some opponents get better the more complete your deck is. But you'll still win most of the time :)