It shouldn't matter as you manually select the BT source, but maybe unpair one set of buds and see, or before that try again and avoid any interaction with the remote, maybe that triggers a switch.
Odd I don't use mine often but, not had such an issue.
Regards Obsydian - (Car & Tech Evangelist)
MY DEVICES: ZFold4, Watch6 Classic, Book2 Pro i5 5G [Laptop], Buds Pro2, M70A 32 & 43”; [Monitor], Smart Tags, Dell Inspiron G5 [Laptop] / FAMILY DEVICES: ZFlip3, ZFlip4, ZFlip5, Watch6 Classic, S8 Ultra; S7+ [Tablet], Buds Pro, iPhone 12, iPad Air 2, Air tags, iPad Mini 5, iPad Pro 12.9 GEN2 [Tablets] / HOME DEVICES: QN95A [TV], Q900A / SWA900S [Soundbar, Rears], MC28M6075 and MC32K7055 [Microwave], Jet 75 Complete (Vacuum), Samsung Level [Speaker]