Multiple Project Repositories

There have been various grumblings for a while that we ought to be using more project repositories, following the model of the TraceMonkey branch. The recent "Post-FF4 Branch organization" megathread and the faster release cycle discussion have brought the idea into the light. This page is for exploration of the idea in a more persistent format (it's gotten hard to keep track of everything covered in the thread.) Note that nothing has been decided yet. This page is not an official policy statement, just a place to enumerate concerns, propose solutions, and archive discussion.

Note: multiple project repos are not necessarily connected to a faster release cycle, but they could be part of the overall release cycle change plan.

Specifically, this page talks about splitting up development into a set of project repositories, similar to the current tracemonkey repo, where changes from the repositories flow into a global repo (perhaps the current mozilla-central). For most of these, I would expect m-c changes to be merged back into the project repo fairly frequently, but that's up for discussion.

Main Advantages/Disadvantages

• When bustage is committed, fewer people are affected
• More experimental changes can be made and the problems shaken out because of the smaller sandboxes
• Easing of the patch bottleneck when the tree gets busy (you just need to space pushes with respect to others in the same tree, not with everyone who might commit anything anywhere)
• Commit policy can be more closely matched to the needs of a restricted area (eg churn from small changes is less harmful and so more acceptable => more conducive to refactoring, fewer change freeze periods to discourage external contributors) (Note that I would not expect review policies to differ between repos)
• More merges are required, and more "clumps" of changes move around the system
• Less clarity on where individual changes should go, especially for less active (new/occasional) contributors
• More overhead and resource contention for release engineering, QA, nightly users
• Better correspondence between repos and IRC channels
• Harder to see the latest version of everything
• More uncertainty about when a specific change will land in a given repo or nightly

Questions

Should all work go into project repos, and mozilla-central be a dedicated merge destination? Or should m-c live on as the default repo, and we'd just increase the number of project repos that we already have?

Will some project repos only feed into mozilla-central, and never get changes merged back?

Do we need "urgent" sub-repositories like Linux kernel development uses? (Changes made to these repos are expected to go directly into the central repo rather than waiting on the next merge.)

Issues

A number of issues came up during the mailing list thread. The ones I can recall are captured here and (one) response given. Please add more, and add/improve the responses. You can try to initiate discussions here, or (probably better) post to the mailing list for discussion and document the result here.

Contributors will need to maintain too many repositories

Summary: A user who wants to contribute to multiple divergent areas within the Mozilla codebase will need to have half a dozen repositories checked out.

Losing a single unified work area

Not a problem. I first tried to keep all of my changes in one mq repository, shuffling patches constantly, but I eventually discovered that I needed half a dozen personal repositories to keep separate things separate -- and this was all for work under the js/src tree. So from my experience, sprouting lots of repos is the Right Way anyway.

The trick is how to do it to keep the overhead low. First, do not clone the upstream repo N times. You'll burn tons of disk space and it'll take forever, and when you periodically decide to update all your repos (which you probably shouldn't unless you're going to actively use them, but it's nice to have that fresh minty up-to-date feeling), don't update against upstream.

Instead: when using one upstream repo, clone it once, and then clone it once for each "topic" repo you want. The I/O will all be local, and under Unix-derived OSes it'll make hard links. In fact, I usually use |cp -rlp| instead of |hg clone|; the main difference is that you'll need to edit .hg/hgrc afterward to point to your master repo.

When using multiple upstream repos, you can either do the same thing, once for each repo. Or perhaps better, pull all of the upstream repos together into one local master repo with a bunch of heads. Then, when cloning into your topic repos, use |hg clone -r <head-rev>| to make a subrepo that only has the head from the appropriate upstream repo, but can still pull updates from the master. (You can push from this subrepo normally.) The bookmarks extension might make this even better, but I haven't tried it.

Warning: I haven't actually tried the multiple-head approach above, so I don't know what problems may arise in practice.

Disk space for source

Assuming mercurial and your OS supports it, hard links resolve this fairly well, as discussed above. (Mercurial on all Linux filesystems, and Windows' NTFS filesystem, hard link your clones for you. So that's pretty good coverage.)

Backups do not necessarily check for hard links, and may explode in size. Also, Mercurial doesn't do a great job of sharing the history data for files that change across clones. And the links tend to get broken during regular usage. (hg relink helps recover from this, but is a slow and manual step.)

Disk space and build times for obj dirs

You may have to share obj dirs across repos. This preserves single-repo disk usage and build times (especially if using ccache), but adds complexity. It doesn't really decrease the chance of having a ready-to-go build for any particular area you might want to work on, because with a single mq-controlled repository you're constantly clobbering your old builds anyway.

Merge Nightmare

Summary: The merges of the project repos into mozilla-central will be messy, time-consuming, and error-prone

Sayrer probably has good advice and practical experience for minimizing the pain. I'll add one more: "mozilla-next"

The Linux kernel uses a "linux-next" repo for precisely this purpose. See the original announcement, but in brief, this is an automatically-maintained repository that merges together all of the upstream repos daily. If any merge fails, that repo is skipped. The end result is a merger of all repos that successfully merged. Maintainers of the repos that failed merge are notified. I assume that when a merge fails, the repo maintainer pulls in the master repo's changes and manually fixes up the merge, so that the next merge is more likely to succeed (it won't even have to try to merge any of those changesets next time.)

In practice, linux-next has apparently been very helpful in reducing merge problems, since new problems are discovered quickly and reported. But it is important to be aware of what this tree is not: successfully landing in this tree (as in, having your tree make it through the automated merge procedure) is no guarantee of inclusion in the "real" tree. It does help give people an idea of what could be in the next version of Linux, well before those features are ready for inclusion (let alone accepted). You might think it could be useful for forward-looking QA, but it really isn't: if a bug is found in mozilla-next, there's no way to bisect it because the tree is rebuilt from scratch every day, so there's no useful revision history. Still, it is good for what it is: a way to minimize merge conflicts between an assortment of upstream repositories (80 or so in the case of Linux).

Spreading QA Thin

Summary: We don't have enough QA bandwidth if it's split up among the multiple project repos

Fundamentally, the incoming patch load doesn't change, so in an ideal world it wouldn't matter: early on in a release cycle, you would only do testing on the repo that is relevant to what you're testing. After the features land in mozilla-central, you move testing there.

In practice, breakage can come from anywhere, so the above ideal wouldn't work even if it were possible. But approximating it might be good enough. QA is necessarily an approximation in the first place, so this isn't fundamentally crazy. We also get some offsetting gains by testing within an environment with a more constrained scope of changes -- we will get some assurance that a feature works within its "native" environment before smashing it in with everything else. If it doesn't break in the project repo but it does break in mozilla-central, then you have a better idea of where to look for the cause.

The above is a lie, because project repos would probably be merging mozilla-central back into themselves fairly often. But if that's the case, then there's no benefit to doing QA on mozilla-central; every project repo will contain all changes on the way to release minus very recent work on other project repos.

I don't understand our QA setup well enough to say anything more intelligent here. So far, I'm not seeing any fundamental reason why QA would be worse with project repos.

Spreading philor Thin

Drawing and quartering philor?

Summary: Too many places to look for intermittent/known test failures to be starred

The aggregate push rate doesn't change by splitting things up into project repos, so this seems like a tooling issue: we need a TBPL that can show a Grand Unified View of all non-Try push results. Then we'll put that in place of the original URL that philor has bookmarked and pretend like nothing has changed. :-)

Theoretically, the splitup ought to be a good thing, since we'd have some extra bits of evidence about where random oranges come from. If an orange is truly intermittent, but only shows up on one project tree until it merges into m-c, that's useful information.

(An an aside, I should point out it's really not philor's personal responsibility to star everyone's oranges. He's providing an incredibly valuable service for us all. Is it because he has a generous heart, or because he's batshit insane? Or both?)

Too hard to watch changes

Summary: Right now you can watch the mozilla-central pushlog. With project repos, you'd need to either watch them all, or get giant clumps of changes when they hit mozilla-central.

Proposed solution: Create a unified pushlog view. You even get added functionality of being able to filter by repo.

Contributor Confusion

Summary: A prospective contributor won't know where to make changes

This is a real problem I don't have a quick answer to. I think there's probably a flip side, though -- if the contributor does find the repo or IRC channel corresponding to the area being modified, they'll be able to work in a smaller, less overwhelming environment. The rules and procedures for contributing are pretty complex, and right now you have to either not care, or spend a lot of time in advance figuring out exactly what you're supposed to do. If you know you're only going to affect the dozen or so people active in a particular area, it's a lot less intimidating to get involved. But I still haven't addressed the original issue.

Fragmenting The Nightly Tester Community

Mozilla currently benefits from a (relatively) small but dedicated nightly tester community. It is unclear what impact having multiple repositories churning out different nightly builds will do to that community and for early regression testing. Potentially, this could lead to delays during the beta release cycle due to more regressions being discovered nearer to the intended release date.