Frozen Concoction - Cant set weapon set affinty

When using frozen concoction, its impossible to set weapon set affinity.
Example: i want to use a shield and wand on offhand for shield charge and casting, and then once i press left m button to attack with concoctions, have it swap to my bow+quiver setup automatically.
It works on typical skills but not on this built in one.