There should be a difference in performance on all apps being used at all times. However, I much prefer the optimized mode as it will save up some battery and because your phone is already top of the line you won't really feel any slowness.
I would only use high performance for intensive always-on tasks such as DeX where you need to use quite a lot of your hardware.
Simply keep it to optimized and you will benefit a lot more throughout the day.
