The encryption key
You don't have to supply one. If REAPER_SECRET_KEY is unset, Reaper generates a key on first boot and saves it to <data_dir>/secret.key with mode 0600. It'll reuse that key on every subsequent boot.
Your key file sits beside <data_dir>/secret.salt, which is minted once per install. Reaper stretches your key with the salt before encrypting anything so the two only work as a pair. Reaper writes a salt even when you supply the key yourself.
Your key is generated once and won't rotate automatically. If the key changed on restart, it would silently make every stored credential unreadable. You'd only find out the next time a scan tries to reach Sonarr.
Setting REAPER_SECRET_KEY explicitly always wins. Reaper won't write a key file at all in this case, so your secret manager stays the single source. Reaper never overwrites an existing generated key file because your database may still be encrypted with it.
Rotating it deliberately
You can do this safely while it's running. Put the new key in REAPER_SECRET_KEY and the retired one in REAPER_SECRET_KEY_OLD, using commas to separate them for a chain:
REAPER_SECRET_KEY=<new>
REAPER_SECRET_KEY_OLD=<previous>,<older>
Reaper encrypts under the new key while still decrypting whatever the old ones wrote so nothing gets bricked halfway through. Drop the old value once every credential has been saved again.
What encryption at rest buys you
Your reaper.db file doesn't leak any working credentials on its own because the key is a separate file. Treat the whole data folder as sensitive anyway. It carries your titles, your decisions and your audit trail, and one careless copy of the data folder undoes that separation.
Where it stops
Since both files sit in the same folder, anyone who can read your filesystem can read both. Supply the key from a secret manager when you need them genuinely apart.
Your backup is more sensitive than the database alone because it carries the key and the salt so that it restores. See Back this up.