[PC] Changing the active window via ALT + TAB causes the GPU to get < 1/4 fps ( 4x+ frametime)

Changing the active window via ALT + TAB causes my Nvidia RTX 2X desktop GPU to get < 1/4 fps compared to before ( the graph shows quadruple+ frametime above 40).
Restarting is the only way to get rid of this since changing graphics settings does not help.

I also started getting this issue since this new Druid patch.

I am using a RTX 2070 Super and have an AMD CPU.

I have to restart the game to fix it.

It sometimes doesn't allow me to Teleport as it says ingame that the game is paused. This doesn't always happen after I return after Alt Tabbing.