Where and what the island is, isn't sure. But it certainly is a place only a powerful Source of the Elder Blood can reach by travelling through space, time and dimensions. So it might even be that it's part of Geralt's world, only somewhere else many years later. Or earlier.
What follows contains spoilers concerning the books, so if you still hope they'll be translated to English sometime soon and don't want to be spoilt, take care:
The Hunt is a group of elves from another dimension (might be another world or only another time and still Geralt's world), who hope to 'free' their brethren struggling for survival in Geralt's world. From what I gathered, their access to that world is very limited though and thus they (sort of) kidnap people from this world to be their minions - it's not the deads' souls though. The key to actual transitions between different dimensions is the Elder Blood. Therefore they want to get their hands on Ciri - Geralt's and Yen's foster child and of Elder Blood - and they hope to get to her through Geralt and/or Yen.
Also, be aware, that those transitions mean not only travelling to a different place, but can take you to a whole different world or into another time (for example, sometimes people taken by the Hunt show up many years later and find their family and friends long dead, but do not have aged a year themselves). So, we do not know how long Geralt has been with the Hunt and what he experienced in their company (could be 5 minutes, or many, many years) - all we know is that he showed up 5 years later.
I'd ask others to correct me / give their point of view. It's sometimes rather confusing and reading the books has been a while, so I might mess some stuff up or simply interpret it differently.
callofstalker: You are almost completely true; Geralt appears in Kaer Morhen's surroundings five years later from his death, not five years later from him being a part of the cavalcade which means he probably stayed just a few days(few days in the parallel world) as a part of the hunt because a year in the parallel world is several decades in ours(see the note about the patient claiming being a part of the hunt).
You're absolutely right and that's what I meant - but now that I read it again, it comes across plainly wrong. Thanks. :)