Multiple Project Repositories

I meant this to be about how to organize branches to enable a faster release cycle, but it's turned out to just be about dealing with multiple project repos. I just want to point out that we may want additional repos (or policies with regard to repos) connected with the faster release cycle that are not covered here. And multiple repositories could certainly be applied without modifying the speed/style of the release cycle at all.

Issues

Contributors will need to maintain too many repositories

Summary: A user who wants to contribute to multiple divergent areas within the Mozilla codebase will need to have half a dozen repositories checked out.

Not a problem. I first tried to keep all of my changes in one mq repository, shuffling patches constantly, but I eventually discovered that I needed half a dozen personal repositories to keep separate things separate -- and this was all for work under the js/src tree. So from my experience, sprouting lots of repos is the Right Way anyway.

The trick is how to do it to keep the overhead low. First, do not clone the upstream repo N times. You'll burn tons of disk space and it'll take forever, and when you periodically decide to update all your repos (which you probably shouldn't unless you're going to actively use them, but it's nice to have that fresh minty up-to-date feeling), don't update against upstream.

Instead: when using one upstream repo, clone it once, and then clone it once for each "topic" repo you want. The I/O will all be local, and under Unix-derived OSes it'll make hard links. In fact, I usually use |cp -rlp| instead of |hg clone|; the main difference is that you'll need to edit .hg/hgrc afterward to point to your master repo.

When using multiple upstream repos, you can either do the same thing, once for each repo. Or perhaps better, pull all of the upstream repos together into one local master repo with a bunch of heads. Then, when cloning into your topic repos, use |hg clone -r <head-rev>| to make a subrepo that only has the head from the appropriate upstream repo, but can still pull updates from the master. (You can push from this subrepo normally.) The bookmarks extension might make this even better, but I haven't tried it.

Warning: I haven't actually tried the above, so I don't know what problems may arise in practice.

Merge Nightmare

Summary: The merges of the project repos into mozilla-central will be messy, time-consuming, and error-prone

Sayrer probably has good advice and practical experience for minimizing the pain. I'll add one more: mozilla-next

The Linux kernel uses a "linux-next" repo for precisely this purpose. See the original announcement, but in brief, this is an automatically-maintained repository that merges together all of the upstream repos daily. If any merge fails, it fails and is skipped over. The end result is a merger of all repos that successfully merged. Maintainers of the repos that failed merge are notified. I assume that when a merge fails, the repo maintainer pulls in the master repo's changes and manually fixes up the merge, so that the next merge is more likely to succeed (it won't even have to try to merge any of those changesets next time.)

In practice, linux-next has apparently been very helpful in reducing merge problems, since new problems are discovered quickly and reported. But it is important to be aware of what this tree is not: successfully landing in this tree (as in, having your tree make it through the automated merge procedure) is no guarantee of inclusion in the "real" tree. It does help give people an idea of what could be in the next version of Linux, well before those features are ready for inclusion (let alone accepted). You might think it could be useful for forward-looking QA, but it really isn't: if a bug is found in mozilla-next, there's no way to bisect it because the tree is rebuilt from scratch every day, so there's no useful revision history. Still, it is good for what it is: a way to minimize merge conflicts between some 80 or so upstream repositories (in the case of Linux).

Spreading QA Thin

Summary: We don't have enough QA bandwidth if it's split up among the multiple project repos

Fundamentally, the incoming patch load doesn't change, so in an ideal world it wouldn't matter: early on in a release cycle, you would only do testing on the repo that is relevant to what you're testing. After the features land in mozilla-central, you move testing there.

In practice, breakage can come from anywhere, so the above ideal wouldn't work even if it were possible. But approximating it might be good enough. QA is necessarily an approximation in the first place, so this isn't fundamentally crazy. We also get some offsetting gains by testing within an environment with a more constrained scope of changes -- we will get some assurance that a feature works within its "native" environment before smashing it in with everything else. If it doesn't break in the project repo but it does break in mozilla-central, then you have a better idea of where to look for the cause.

The above is a lie, because project repos would probably be merging mozilla-central back into themselves fairly often. But if that's the case, then there's no benefit to doing QA on mozilla-central; every project repo will contain all changes on the way to release minus very recent work on other project repos.

I don't understand our QA setup well enough to say anything more intelligent here. So far, I'm not seeing any fundamental reason why QA would be worse with project repos.

Spreading philor Thin

Drawing and quartering philor?

Summary: Too many places to look for intermittent/known test failures to be starred

The aggregate push rate doesn't change by splitting things up into project repos, so this seems like a tooling issue: we need a TBPL that can show a Grand Unified View of all non-Try push results. Then we'll put that in place of the original URL that philor has bookmarked and pretend like nothing has changed. :-)

Theoretically, the splitup ought to be a good thing, since we'd have some extra bits of evidence about where random oranges come from. If an orange is truly intermittent, but only shows up on one project tree until it merges into m-c, that's useful information.

(An an aside, I should point out it's really not philor's personal responsibility to star everyone's oranges. He's providing an incredibly valuable service for us all. Is it because he has a generous heart, or is he just batshit insane? Or both?)

Too hard to watch changes

Summary: Right now you can watch the mozilla-central pushlog. With project repos, you'd need to either watch them all, or get giant clumps of changes when they hit mozilla-central.

Create a unified pushlog view. Done. You even get added functionality by being able to filter by repo. How is this bad?

Contributor Confusion

Summary: A prospective contributor won't know where to make changes