Issue with PoisonBurst Arrow

If I put on rings my damage goes down by quite a bit.
I'm wearing PlagueFinger Gauze Wraps.

https://www.loom.com/share/34b8ba81517c4cb994fd284ce2f7aec0
Video shows me taking on and off rings and you can see the damage goes down when I add the rings. All rings have flat damage on them.
All spirit gems are turned off.


After testing and making the video now I can't even turn the spirit gems back on because it says:
Item cannot be used: - Not usable with current Weapon set settings.

I have enough dex/strength/int to use all of them, they were working prior to this.


Any help is greatly appreciated.
Thank you

I am using that passive, however I click the well after changing rings to make sure I'm on full health and the lower damage is still present.

I changed my bow/quiver into weapon slot 2 and was able to get spirit gems turned back on. Still having same issue with damage though.