Unable to refund a keystone passive with nothing allocated in radius

I am playing Oracle with the Entwined Realities passive.

I socketed a Controlled Metamorphosis Diamond with a medium-small radius into a jewel socket. I then allocated a point into Zealot's Oath.

I allocated a point into Desensitisation and then refunded the passive. I can no longer refund the Zealot's Oath keystone. Nothing in the medium radius of Zealot's Oath is allocated.