The S Pen utilizes a capacitor to store charge, which means it can quickly draw a small amount of power from your phone to fully recharge when inserted, even if you only used it for a brief period.
Even brief S Pen usage can trigger the phone to recharge the pen's capacitor, leading to a noticeable drain on the phone's battery