Generally, I would keep the "previous" version of the Version Reference in the Management Console and the corresponding file in the Content share. Older Versions of the file might be backed up to a different location.
While there is no App-V realted technical reason for keeping the last version, you might see that a newly published version generates issues (that were not found during Acceptance Testings). Having the "previous" version available allows for a quite fast
stepping-down.
However, after stepping-down applications may fail to launch. Then, the User Settings would have to be deleted ("Repaired").
Falko