It sounds like the underlying styles are automatically updating.
Choose Tools - Templates and Add-ins, and see if Automatically update
document styles is checked. If it is, then click to remove the check, then
click OK. Back in the document, fix the page numbers (if necessary), and
they should stay put the next time the document is opened.
A better solution, of course, would be to redefine the style that's used to
format the page numbers so that you don't have to rely on direct formatting.