Can't access Mansion In The Mists?

I've cleared every map around this thing, found a Hengestone in a couple of them and yet it remains locked... Is it possible I missed a Hengestone somewhere and now I'm locked out of it?
I'm pretty careful to check for these things so I don't think I missed one. How many are there and how far out on the map do I need to look? Anyone know?

Real shitty if I'm locked out because I did miss something. Would make no sense to lock you out like that...