Hi @saman1989 ,
Not at all. In fact it will help save the battery which in turn means that you will charge less and thus prolong the battery life. I do it often.
The drawback is that the "Raise wrist to wake" will not work. Photo and music sync between the watch and the phone will not work. Notifications may get delivered silently and the screen may not turn on.