It sounds like it would be the ambient light sensor causing that instead of the proximity sensor. Tbh every phone I've had I turned it off as I've heard it uses more battery that it saves. It's also not something I ever use on any of my Samsung phones or other smartphones I've had because it can change so often as you said, its just a distraction.