Setup infra needed to manage old major versions

[jackshirazi]

Is your feature request related to a problem?

When we release a major version, we'll need to maintain the previous major version for a while, including applying patches, especially pointing it at updated upstream versions. Currently the repo can't handle this. We need to make the changes to let it do so and document how to do releases