Firefox/Projects/Tabcandy/Panorama 1 Debrief Chat
After Firefox 4 shipped, the Panorama team and other interested parties got together and discussed how things had gone. This is a transcript of that chat. You can also see the summary.
iangilman: So, welcome all to the official Panorama debrief!
iangilman: I figure this can be a free flowing discussion
iangilman: howdy zpao
mitcho looks to iangilman to get us started with some structure, a guiding question, etc.
iangilman: anant, beltzner, dietrich, ehsan, juanb, msucan, not_gavin, shorlander, please join us if you have something to add!
iangilman: yup... I think I'll just dive in
msucan: hello everyone
iangilman: So, one of the obvious things about Panorama was that it was largely an outside team, not part of the mozilla establishment
iangilman: I think this was good and bad
iangilman: it was good because we had fresh eyes, and we were focused on just this one feature
iangilman: in fact, I think without a dedicated strike team, a large feature like this would never get done
iangilman: It was bad because we had a steep learning curve, and we made a number of naive decisions that had to be reversed later
mitcho: more institutional support would have been beneficial, though
iangilman: right, and maybe a single insider to help us navigate
mitcho: there were disconnects both in terms of code (a lot of us were new to mozilla firefox dev, so there was a lot we had to dig around to find ourselves) and also in project management/support, e.g. a main reviewer, platform contact, etc.
mitcho: at least
pcwalton: my biggest issue with the front end in general is review bandwidth
iangilman: on the other hand, we did have a number of good "friends of Panorama", like ehsan, mardak, and zpao, who really made it possible
mitcho: pcwalton: +1
mitcho: iangilman: true
iangilman: .... but we had to drag them away from their real jobs
zpao: i think part of the issue with the disjoint team was the lack of involvement at meetings (firefox team, platform). i think it's something that if it were in front of people more often, it would have gotten better attention
kevinh: interfacing with the Moz hive mind should have been done more frequently, and earlier
kevinh: not just for engineering, but across the board. In some sense, we played catch-up a lot.
pcwalton: I think all sub-projects should have a reviewer who can review things before they make it into at least a branch
pcwalton: patches sitting in bugzilla for months is the worst of all possible worlds IMHO
iangilman: zpao: good point! I think part of our problem is we didn't even know about those things
mitcho: we also weren't active in dev community involvement, like on planet.
pcwalton: get 'em checked in somewhere, even in a project branch
iangilman: it might be interesting to have a "so you want to start a feature strike team?" FAQ
mitcho: iangilman: good point
raymondlee: iangilman: yes, agree.
shorlander: I agree this is valuable insight for any future projects like this
zpao: definitely (for the team doing it and also for the people who should pay attention to the strike team)
mitcho: also, for me at least, it was unclear early on what exactly the hurdles would be in getting this in the product. whether this was really in-scope, whether we would have the support from the main product team, what the review and testing hurdles were, etc.
iangilman: pcwalton: getting me set up as module owner, so I could review patches, was a huge turning point
mitcho: early on, it was unclear how likely or unlikely it was that (a) we would ever land and (b) it wouldn't be pulled out
pcwalton: yes, that was really good
pcwalton: I wish the Web Console had had a module
iangilman: speaking of FAQ, I think our team wiki page worked out well: https://wiki.mozilla.org/Firefox/Projects/TabCandy/Work
kevinh: I see this conversation being echoed on the moz-dev planning group, but contributions from the community were really crucial for us
zpao: mitcho: that was an issue. partly because of lack of communication, but also because the schedule as a whole was in flux
kevinh: Making that process easier and more encouraged seems like both a good idea for strike teams and moz in general
mitcho: I feel that, if it was clear that this was a stated goal of fx4, a little earlier, within the firefox team, ux, and to dev/community communications, it would have been easier to get people on board and help
juanb: I don't know if there was a an assessment of the scope of the project in the beginning, which would have included people in the platform and front end team to sort out the doable vs future work within the timeframe of fx4.
iangilman: mitcho: indeed... it seemed like a couple of months of trying to land and not really knowing how far along the process we were... it was a spinning cursor, not a progress bar
mitcho: even having particular milestones which would be required for inclusion would have been helpful
juanb: I don't know that there was a set criteria of memory and performance metrics we could have used to check that we were doing ok, but that is also true of other projects.
mitcho: well, it seems that there are some criteria like that... don't hurt talos, don't explode memory, don't break things, etc... but that wasn't immediately clear at the beginning.
kevinh: seems like what I'm seeing is 1. we need better communication, preferably with a team member in MozHQ, 2. We need better road-mapping and planning, 3. We need better information on how to onboard people to the process. Most of that says PM to me, though I could be biased
ttaubert: hey everybody!, sorry I got stuck in traffic
iangilman: of course, if there had been too much thought put into whether Panorama should be part of Fx4, it never would have happened
mitcho: ttaubert: welcome!
kevinh: ttaubert: hey
raymondlee: ttaubert: hi
mitcho: iangilman: true.
iangilman: ttaubert: glad to have you here! I'm particularly interested in your perspective on how you found the project and getting up to speed this late in the game
mitcho: iangilman: and to a certain extent it did take actually landing and then iterating to really convince more people that this was worthwhile, or so it felt
iangilman: Right... Panorama really only happened because we were so passionate we blew past all of the road blocks
msucan: i am working on some patches for devtools, so not very active in here, but my comments would be:
kevinh: and had a decent proof-of-concept add-on
raymondlee: Yeah, we didn't really know whether it would be in Fx 4 or not right at the beginning.
iangilman: right! Having a prototype people could see in action was huge for convincing people
raymondlee: the proof-of-concept add-on really helped.
iangilman: (perhaps aza will have more to say about that when he's back from his other meeting)
msucan: i think panorama's code is really nice. i dived into the code really late in the process, to help with a blocker, still it was easy to understand and dive into. i like the code, the structure, but it felt somehow too different from the code style of other mozilla js code
iangilman: msucan: excellent point... I actually find the mainstream mozilla code base really difficult to wrap my head around... h
msucan: panorama's code felt more like a typical web page
iangilman: ... so what's better? Having good clean code, or matching the rest of Mozilla?
msucan: with lots of js
iangilman: right... no xul
mitcho: on the other hand, though, I think there are issues with the way panorama is integrated with tabbrowser which happened *because* we built it first as an add-on and then a possibly-removeable addition to the browser
msucan: iangilman: i think clean code does not exclude matching the rest of Mozilla
iangilman: mitcho: good point... like what?
mitcho: i.e. the notion of "tab groups" should really have been a real xul/js construct in the tabbrowser level, ideally
mitcho: we still don't have that... only "hidden"
msucan: just follow code style guidelines. same structure, everything. just indent the same, use arguments with "a" prefix, and so on
mitcho: and that seems like a real pain point down the line, and is a clear reflection of this being developed as an "addition"
msucan: break lines like the rest of the code does
fryn: mitcho: as someone who works on tabbrowser code, i concur with you.
msucan: wrt. tab browser. it feels like panorama duplicates some stuff from tabbrowser
iangilman: msucan: I guess I don't really know the details on how our code is different. We did look at the coding guidelines and tried to follow them. This was, however, a while after we got started
mitcho: if, instead, the whole firefox team made it clear that panorama (or some kind of group functionality) was going to be a first class citizen, we would have built in groups in tabbrowser + sessionstore + sync early on, and then built the interface on top of that, like it should have been, imho
msucan: (but i didn't dive into "what could improve")
zpao: mitcho: i still feel the same way about sessionrestore. many issues came up because it wasn't tightly integrated
fryn: we may want to rewrite tabbrowser to be better modularized and be designed from the ground up to support the integration of features like panorama.
msucan: iangilman: for me it was confusing. you don't use lets, and so on. i had to change "my mode" for panorama
ttaubert: mitcho, right, it's a too defensive integration at the moment
kevinh: mitcho: I agree it wasn't ideal, but there IS a bootstrapping problem. I am not sure the Fx team would let in a project of such magnitude without first being proofed out
msucan: fryn: exactly
pcwalton: all I can say is that I'm glad that Panorama is a series of .js files and not a .xml like tabbrowser.xml
msucan: haha, me too
pcwalton: panorama code feels more like "modern" JS code
pcwalton: is unsurprising, given that it is... well, much newer
pcwalton: I like iQ. mad props to iQ.
kevinh: iQ is funny, some people really didn't want us to do it
mitcho: jeresig thought I was a little crazy
mitcho: when I told him about it
iangilman: So obviously there are things Panorama still needs to take from mainstream mozilla
iangilman: ... and sounds like there may be some things Panorama can contribute
fryn: because of panorama's uintentionally obfuscated hooks into tabbrowser code, i find that i often run into bugs due to how panorama overrides and changes the behavior of some getters in tabbrowser. i'll likely arrange a meeting or two to discuss that once i have more concrete ideas on how to address this problem.
iangilman: like maybe there will be more html/js modules rather than xul? Maybe iQ could be used elsewhere?
mitcho: can we talk a little more about the integration of panorama before we get to good things about the process?
mitcho: (iangilman: though I agree with what you're starting to say...)
iangilman: I really like the idea of the core tech that makes Panorama possible being built into the guts of the browser (unlike how it is now) and the presentation of Panorama being just a thin layer of html/js that only reacts to and guides that core tech
iangilman: mitcho: sure... lead us
mitcho: so... (this sort of is counter to what I was just saying about better tabbrowser integration, but...) I think it was a mistake to not land it with a very easy code kill switch.
mitcho: especially as we didn't know whether it would really ship, even after landing
mitcho: there was a lot of community backlash at points, which may have been more easily alleviated if we had an about:config setting, at least.
kevinh: I actually think not caving to the killswitch was pretty key for us, as the reviews for us (on twitter, since launch) have been overwhelmingly positive
pcwalton: I think some of that was also due to a slightly overzealous land grab of keyboard/mouse shortcuts
pcwalton: most notably three-finger swipe
kevinh: the problem wasn't that Panorama was IN Fx, it was that it was broken, and too easy to trip on. I think we eventually struck a close-to-right, conservative balance on that
pcwalton: perhaps keyboard shortcuts should be phased in gradually as the product matures
mitcho: but for future projects, say
mitcho: what if we never improved the performance? for example
pcwalton: to avoid a backlash resulting from people accidentally triggering a feature that isn't ready yet
mitcho: it would have been a pain if months later we had to back the whole thing out
iangilman: looks like kill switches are going to be required for future features, for what it's worth (if you've been following moz-dev-planning)
ttaubert: sounds reasonable
raymondlee: should be easy to implement.
kevinh: The new structure might be better for new features. Iterate and polish on a separate branch until it is ready to hit m-c. Hopefully some perf work is part of that
mitcho: (not that we necessarily want to keep code "latent"... lord knows there's also other code which is in trunk but disabled... *cough*tabpreviews*cough*)
mitcho: iangilman: yeah
kevinh: pcwalton: very much so
zpao: (yea i think there's been talk of taking that out)
zpao: (though people us it so...)
mitcho: another thing on process:
iangilman: ttaubert: you're quiet today... what are your thoughts?
mitcho: starting with an add-on was very good for prototyping it for starters, but moving it from an add-on's "shell" into trunk, I think, also is a reason for why the final product is so "compartmentalized" in a sense
ttaubert: I'm reading and agreeing so far I slipped later into the whole process so that's not all new to me but some things are
mitcho: perhaps starting from a branch rather than from an add-on would have been better? though harder to get people to try it and get excited.
kevinh: maybe a tiered process, addon > branch > m-c
ttaubert: are branches contrary to kill switches?
mitcho: kevinh: that's essentially what we did, though
mitcho: ttaubert: perhaps.
raymondlee: mitcho: but would it be hard for people to try it if we were not an add-on
iangilman: mitcho: that gets into the question of whether we should have thrown out the prototype and written the integration from scratch
mitcho: ttaubert: but there's still a lot more you can do in a branch in terms of code organization that you can't with an addon
iangilman: .... thoughts on that?
mitcho: iangilman: I say yes.
mitcho: iangilman: probably starting from tabbrowser on up
kevinh: iangilman: it sounds like, for our needs, Fx's code should be easier to get TO from an add-on
ttaubert: I don't know the prototype but this makes sense
iangilman: I think I agree, although there's no way the existing team could have done that... we would have needed a dedicated "man on the inside"; probably multiple
mitcho: ttaubert: you should go find it. it's kind of fun.
mitcho: iangilman: true.
ttaubert: mitcho, ok
iangilman: ... in which case we would have been gated behind people's existing priorities, and it never would have happened
mitcho: ttaubert: don't think it'll work with fx4, though.
ttaubert: so at least in an ideal world we should have thrown away the prototype
mitcho: iangilman: unless, as discussed above, we actually had an employee or two dedicated to the project early on
mitcho: the prototype was indeed great for getting people excited and mocking this up, though
mitcho: kudos to aza!
aza: Man, this is a lot of scrollback.
iangilman: mitcho: I finally upgraded my primary browser to Fx4 now that it's released, and guess what? There was an ancient copy of the add-on sitting dormant in my profile... freaked me out!
ttaubert: (*sshhh* aza is reading
mitcho: hey seandunn !
seandunn: iangilman: good afternoon, sorry I'm late, long lunch.
seandunn: mitcho: hey!
mitcho: iangilman: hehe, maybe you can use that one too and get panorama-in-panorama, infinite loop style.
iangilman: seandunn: glad you could make it! We're still at it
raymondlee: hi seanunn
ttaubert: hey seandunn
raymondlee: seandunn ^
seandunn: raymondlee, ttaubert: hello!
iangilman: yeah, having a prototype is definitely a good idea, even if you do throw it away
iangilman: the prototype allowed us to do all sorts of crazy experimentation at the beginning
ttaubert: I think that's a good way to try yourself out
mitcho: also in terms of integration and code flow... I think it would have been beneficial to start writing mochitests earlier, like when we started to move to a branch. looping in the mozmill/qa folks earlier would also have been nice, though a lesser priority, imho.
kevinh: ... which was easy enough to do that just iangilman and aza could riff on varied ideas at a rapid pace
mitcho: (although our code coverage is not bad!)
aza: Absolutely, an internal champion is necessary. Better integration with the Firefox team is right as well (my fault, really, for not being more on top of that). Trying to formalize the inclusion process would have been nice, but I don't actually think the Firefox team knew what those metrics are. Do, and ask forgiveness later, is the right way to go.
aza: I would have liked to get in front of the perf people earlier.
iangilman: right... if things get too formalized, then this sort of big feature may never make it
iangilman: it's a delicate balance
aza: Although the goals around less than 1% impact on Ts was really helpful.
iangilman: Now that aza's back, I'd like to call out another important factor: Aza's evangelism, especially the blog posts with videos
iangilman: I think getting such a huge public buzz on the outside helped us get action on the inside
mitcho: it was great, but I wish we'd done a little more on that.
iangilman: Also, I believe ttaubert found us because of this?
ttaubert: iangilman, yep I just wanted to say that
aza: With one caveat: my biggest learning this time around is never ever use public buzz as a way of trying to directly influence key decision makers within Mozilla. That bounces really poorly. Did that with Ubiquity to general badness. Let this buzzosphere speak for itself this time and it worked much much better.
mitcho: when I was working on ubiquity previously, I was encouraged to blog often about progress and thoughts, and that was pushed to planet, and a lot of people within the organization could latently follow progress that way
iangilman: Also the tweets with links to "good first bug" lists... seemed like a good idea
iangilman: ... not sure how much impact that had
mitcho: iangilman: I think it was good, though b.m.o is kind of scary, which is unfortunate.
aza: Although we let the "how to get involved" webpage languish, at the beginning it really helped with rallying. Similarly, not using Bugzilla to triage and plan at the beginning helped a lot too.
iangilman: aza: you mean never directly say "we have to do this, look at the buzz!"?
aza: iangilman, exactly
iangilman: mitcho: b.m.o?
aza: b.m.o = bugzilla.mozilla.org
iangilman: yeah, knowing at what point in the project to engage with the higher overhead of mozilla process
kevinh: probably when we graduated from 2-3 people to 4-5 people, actually
iangilman: oh, here's another difference in the Panorama codebase... we used naturalDocs-style comments, and actually used them to generate a help page: http://hg.mozilla.org/labs/tabcandy/raw-file/tip/content/doc/index.html
aza: Did other people find that useful in getting into our code?
iangilman: ... did anyone find those pages helpful? ttaubert? seandunn?
ttaubert: it's the first time I see it
ttaubert: I'm quite used to self-documenting code
pcwalton: I never saw it
pcwalton: but that's cool
ttaubert: I appreciate it too, just didn't know it existed
seandunn: I never used it, although I probably would have ended up just looking at the code anyhow.
iangilman: fair enough
iangilman: I think it was useful for the original reviewers and the QA folks... aza?
mitcho: again, though, as nice as that was, it may have been beneficial in the long run to follow mozilla's commenting guidelines
aza: I wonder if the docs and adherence to having our code documented thusly held us to a higher standard.
iangilman: yes, sounds like it
aza: And yes, I think it was useful when brain dumping to our original reviewers.
aza: And moving forward, I'd assume it will help with modders
seandunn: iangilman: What I would have found much more helpful is a description on every class that, in detail, described all other classes it interacted with and how it was used. But, only when I was trying to grok everything.
iangilman: yeah, some sort of comment documentation in the code is good, but I think using the mozilla standard would be fine (though I think it would be great if the mozilla comment docs actually got turned into web page documentation)
aza: Another thing I found that the team did really well (especially coming from outside as we did) was form personal relationships with particular FX team members. That, I think, was absolutely necessary to the success of Panorama.
iangilman: hmm... I suppose we should have invited the read it later guy to this meeting... he was our add-on early adopter... would be interesting to have his perspective
kevinh: I've been engaged with him some of the last month or so
kevinh: he says our code was easy to dive into, and I believe, that the docs helped. He whipped up his Panorama integration in, what? A day?
iangilman: aza: how so?
iangilman: Like all the early help from mardak? and ehsan and zpao ?
iangilman: speaking of docs, we still don't have any official MDC docs for Panorama, nor do we have an API other than what naturally evolved
mitcho: creating the simple add-on sdk module for panorama integration was easy too (of course, that was me, though), though that's been on hold
iangilman: getting that landed will be great
mitcho: it would
iangilman: seandunn: would you like to talk about the review process? One of the things we've been discussing is the challenges of being sort of "outsiders" trying to make this happen
aza: iangilman: In terms of getting review. If we hadn't known dolske, ehsan, etc., then our crazy insane asks would have fallen on deaf ears.
iangilman: aza: for sure
raymondlee: Getting review was really a bottleneck of the whole landing process.
iangilman: speaking of which, #tabcandy has been crucial, and the fact that there are a number of browser core folks who are willing to just be here
seandunn: Any frustrations I had with the review process are in two areas: 1. The combination of lack of documentation on the "proper" way to do something combined with the iterative pain of satisfying a reviewer, and 2. the constant churn of code.
iangilman: Part of that bottleneck in the original landing was the fact that we had this gigantic wad of code... it would have been great to get the reviewers involved earlier.
iangilman: right... we had a number of occasions where we were review bound or approval bound, and a huge pile of patches would build up... when we finally got approval, everything was bitrotted
iangilman: I suspect these are issues for all of Firefox, not just Panorama, but definitely worth mentioning here
pcwalton: that's why I like project branches
iangilman: In someways we probably had it better than some other parts of firefox; at least we had advocates
pcwalton: having patches sit in bugzilla for a long time is a big drag on productivity
ttaubert: yep, I find it kind of problematic that waiting for review produced a big amount of context switches for everyone involved and reduced productivity everywhere
aza: Another complaint: One of the things I liked about early Tab Candy dev was that we were constantly fixing things and getting to a place of awesome dogfoodability. I.e., we were always polishing. As we got to the rush for features by feature freeze in FX that went by the wayside and the product suffered for it.
aza: ttaubert: *nods*
pcwalton: if they're landed *somewhere* then you have history, you have builds that people can try out, you have an ordering to prevent bitrot, you can take merges from trunk in an orderly fashion, etc.
pcwalton: for JS, anyone on the JS team can review patches before landing into tracemonkey
pcwalton: regardless of official module owner/peer status
pcwalton: it works great for them
iangilman: pcwalton: there's two ways you could take a project branch... review everything before landing there, so it's not a huge review process to land it on m-c, or review it afterwards; I can see advantages to both. for a big feature like Panorama (and perhaps even for the upcoming "all windows" work), it's possible you might even want two such branches, one for each. Or maybe that's just crazy
iangilman: pcwalton: sounds cool
iangilman: aza: we kind of ended up in "ship emergency" mode in september and stayed that way for months...
kevinh: our triaging was, like all of Fx, pretty brutal
iangilman: part of our challenge there was incompatibility with new things that had just landed in Firefox, like app tabs and sync
mitcho: iangilman pcwalton: I agree, the tracemonkey approach is interesting
iangilman: ashughes, Mil, philikon: anything to add, or just observing?
iangilman: seems like we're winding down here
kevinh: One thing we could have done earlier was start to plan for transition. We're at a stage now where it's hard to say who will be working on Panorama.
ashughes: iangilman: I wasn't paying attention, to be honest
iangilman: good point!
mitcho: kevinh: +1
iangilman: ashughes: no worries... just curious if you have any thoughts about what went well or poorly in the process, from where you sit
philikon: iangilman: sorry, was just observing
kevinh: and in general, should walled-off projects like Panorama get integrated into the greater moz-dev community?
juanb: From a QA perspective and probably others, it would have been the awesome sauce if we had clear use cases. While watching the tabcandy component in bugzilla I often felt like we were defining behavior along the way, which is fine because things change, but I'd get a little lost.
philikon: but i agree with project branches!
kevinh: ... or should we keep a small dedicated team?
juanb: "how should tab candy work in private browsing?" and then I'd just try to use common sense.
iangilman: juanb: I think we didn't know either! Probably having a common place we could continue to evolve the "spec" would have helped
ashughes: to carry off of juanb thoughts, I'd like to add that while that is true, I found developer response on bugs to QA requests was both timely and efficient
mitcho: iangilman: +1 on spec
juanb: Dev response was great.
iangilman: philikon: no worries. I'll take that as a sign that we didn't aggravate you too much during the last year
juanb: Without you having that much of mozilla experience, you were very interactive here and in bugzilla.
iangilman: ashughes, juanb good to hear!
iangilman: So having well-defined points of team accessibility is good
iangilman: How about our weekly "scrum"? Did the team find that helpful? It was pretty casual...
raymondlee: yes the weekly scrum is a good thing to know what others are doing.
mitcho: twas good
raymondlee: especially, I am in different time zone
iangilman: raymondlee: any other thoughts on how we did with the timezone thing?
ttaubert: I like(d) it, too
iangilman: ttaubert just solved it by staying up all night
ttaubert: sad but true!
iangilman: Not really a viable, long-term, plan, though
ttaubert: but actually working in a different timezone is not that bad
ttaubert: if you've got a todo list you can work on that over the day
mitcho: the timezone, and the remote-ness, forced us all to make sure to communicate well via scrum and bugs
raymondlee: yes, a todo list is a good thing.
ttaubert: the last 2-3 hours of the day can be spent with team communication and getting some reviews
mitcho: of course it would have been great to be all in the same room :), but there's an upside
ttaubert: so you know what to start with the next day
iangilman: yeah, I really liked how the project was moving forward 24 hours... I'd always wake up to a pile of new stuff
iangilman: So, mitcho has suggested we also use this time to look forward to next steps as well... sound good to folks? Any other backward looking items anyone wants to cover first?
raymondlee: One thing I like it's the bugzilla emails so I could read all those emails in the morning and knowing what happened while I was sleeping.
mitcho: raymondlee: same here.
ttaubert: yep that's great
mitcho: often from bed
iangilman: Yeah, I have to say bugzilla is horrifically ugly and clunky, but I've grown to love it
iangilman: ... and the email is a great way to know what's going on
kevinh: which I think speaks to how useful it is to have smaller, modularized, features
mitcho: atul's bugzilla dashboard didn't hurt
kevinh: we didn't catch the firehose from the rest of bugzilla, we only got Panorama stuff
iangilman: alright... let's call the debrief to a close... thank you for all the great thoughts! Kevin and I will distill this chat down and add it to the wiki, probably also a planet mozilla blog post