What steps should I take to get back to Android nougat without losing my settings on my phone?
There is no steps, I quess, to save your personal data through downgrade process. I believe, your data "converted" from Nougat to Oreo so that it will be compatible for new OS.
And actually, it is better to wipe data along with downgrading, because you will bypass a lot of issues.
EDIT: Backup your data with Kies and using other options (Samsung Cloud, Google Drive), downgrage to Nougat and restore the backup then. By these steps you will save a lot of time.
if only flash modem ereo, will i receive updates ota?
No, you will NOT receive OTA after flashing different part of ROM than was included in official version of ROM. You will have to flash back the original modem in order to receive OTA.