Update:Remora Meeting Notepad/Archive Dec-Jul 2007

From MozillaWiki
Jump to: navigation, search

« Back to Update:Remora

June 20th

  • bad file name characters
    • see bug 382854: Is this a general (migration?) problem we should solve in the code or by executing SQL on a per-case basis? --wenzel
  • 3.2 update

June 13th

  • Language packs
    • How can we make the user experience better?

June 6th

  • IT has suggested that we change the way we serve file downloads so they are cacheable (besides sandbox add-ons, of course)
    • We'd use a cron job to generate stats
    • problems with this?

May 30th

  • PHP5
  • Facebook Platform
  • Turkish Support: So far no issues that we can tell. woo.

May 23rd

  • Fixing Turkish support (clouserw)
    • try #3, if IT won't do - try #2
  • crap from shaver:
05:42 < shaver> hrm, still seeing \r and \n all over the place in email
-- this was fixed
05:45 < shaver> do we link to the policy doc in the denial mail?
-- no, shaver to file bug
06:17 < shaver> and when did nl become a supported localization?
-- enabled 13 days ago; live on monday
07:16 < shaver> while I'm thinking aloud here: are we planning to revert the My
                Account change, so that we unbreak all our localizations?
07:17 < shaver> until we can get the strings updated?
-- wil emailed web.dev.projects.l10n.amo.localization about updated string
07:17 < shaver> alternatively, we can use the detect-untranslated-key pattern
                and use the email address instead
  • triage crap from shaver:
07:35  * shaver tries to figure out a decent system for triage
07:49 < shaver> so I added a "3.x (triaged)" TM
07:50 < shaver> to indicate that we've looked at a bug, it's not targetted
               against a currently-planned dev cycle (3.1 or 3.2) and probably
               has severity and such set correctly
-- 3.1 is try to fix over next week or two
-- 3.2 is end of June or mid-July push
  • crap from fligtar
    • Deleting translations from LCP bug
      • yes
      • shaver working on tomorrow irl
    • add-ons blog
      • yes
    • extension security progress
      • yes
    • edit add-on problems
      • yes
    • list of add-ons not compat. with fx2
      • yes
  • wenzel's .02 EUR
    • rather no provisional legal checks (re. bug 376207)
  • UI review

Feb 7th

  • Where are we at with release blockers?
    • Load testing
      • Grinder is unsatisfying
      • Used ab instead, results in morgamic's section on this page
      • Still need to test discussions pages
  • How hard is it going to be to deploy this?
    • morgamic suggests it is up and we are testing it on thursday
    • potential complications include:
      • special stuff in the .htaccess
      • getting domains right (like pfs.m.o, etc.)
      • any client URLs - most of them are rewrites on our end
    • We need to contact QA once this is up to get them to help test (or at least so they know about it)
  • What do we want to do with v2 files with multiple apps?
    • morgamic will detail changes for migration.py
  • Setup redirects for in-client URLs:
    • We need a list of these - shaver sounded like he had ideas


  • working on getting translations for categories
  • PR is going well


  • load testing
    • Grinder issues, unresolved
    • Used AB as plan b. Here are results from 3 test runs: http://morgamic.khan-vm.mozilla.org/results/
    • discovered that using APC on the MPT cluster improves performance dramatically, especially for Cake
    • in general things looked okay except for search which is an area of concern (not surprising) everything else was arbitrarily close to the 10 req/s target (100 req/s and 10 LVS machines plus we have no caching during these tests):
browse1.out:Requests per second:    16.60 [#/sec] (mean)
browse4.out:Requests per second:    15.64 [#/sec] (mean)
browseall.out:Requests per second:    9.16 [#/sec] (mean)
display.out:Requests per second:    14.01 [#/sec] (mean)
home.out:Requests per second:    120.77 [#/sec] (mean)
pfs.out:Requests per second:    2427.77 [#/sec] (mean)
reviews.out:Requests per second:    15.15 [#/sec] (mean)
rss.out:Requests per second:    12.86 [#/sec] (mean)
search.out:Requests per second:    2.33 [#/sec] (mean)
update.out:Requests per second:    677.83 [#/sec] (mean)
  • update service done
  • pfs done
  • need to map old client urls onto these like it was done in v2
  • probably should look at updating client URLs so we don't have to keep doing rewrites until the end of time
  • ACLs - got groups working, modified models for users and groups to have the habtm relationship. Working on integrating the simpleacl component.
  • blocklist - worked on block list service, but not ready to check in yet
  • What do we want to do about files with multiple apps? delete the files? Let's verify before we delete data
  • When is the final migration going to happen?
  • at vet this afternoon, will call from car


  • download controller/download counting
    • functional, merging with file controller
    • maintenance script done today, command line stand-alone as before


  • merged new categories into new db
  • migrated addons from old categories to new

Jan 31st


  • ACL - picked up db acl again and I'm moving on. It's overkill, so I'm going with plan b -- 33 lines of magic that Shaver mentioned. By default the world will have access to everything (less overhead, kind of shaky but we can manage). We can lock new actions down with a beforeFilter (if locking down entire controller) or adding checks to a specific controller action.
  • Do we want to use admin routing for anything? Do you guys want to see admin/foo/edit or just foo/edit?
    • don't really need to use admin routing for now
  • Hardware boxes up (last night) instead of VMs so we can test again this week... in the meantime we should be looking for ways to speed things up as always.
  • HTML 4.01 and validation -- after acl and services (or as a break) I'm volunteering my soul to go through all this sometime this week.
  • Been doing a lot of interviews recently and was looking at software update stuff last week.
  • migration of old URLs and other-app stuff needs to be figured out
  • download counting, what do we want to do for this?
    • use the same model as v2 for now
    • sancus will do this, morgamic to help


  • Worked with lars over the weekend to get a full dump of amo
    • files are approx 2gb, db is just under 400mb, both on khan
    • I'd like to use both of these for our testing and push to preview when we do that next
      • This is waiting on handling files through a controller
  • Roundtable
    • I plan on pushing to preview at some point soon (timeframe?)


  • changed translation background so additional array juggling isn't necessary anymore
  • added preview and "previous versions" pages
  • a lot of minor bugfixing, mostly from feedback page
  • going to do theme install, dictionaries
    • dictionaries just a simple port of the current (JS-y) v2 stuff for now
    • we'll need to whack the DB to make the @dictionaries.addons.mozilla.org stuff get the right add-on type
  • download size, database is in kb, view is currently in bytes
    • add localization key for KB and use it


  • need category list and other final-localization stuff
  • PR stuff starting to eat a bunch of time
  • working on some visual improvements to keep fligtar from slaughtering me


  • done previews, going to work through devcp list


Jan 24th

  • Reminder: incoming parameters (like $id or so) absolutely have to get clean()ed before they are thrown into an SQL query. As a rule of thumb, always sanitize incoming parameters before you use them anywhere. (wenzel)


  • wenzel:
    • browse pages re-done. Should be more straightforward to use now.
    • Fought possible XSS vulnerabilities:
      • html helper now uses htmlentities() on link() calls.
      • forms should now be made by a regular form tag (not $html->formTag() anymore, it's deprecated in cake), and the action="" attribute should come from a $html->url() call and, if it contains user data, be sent through htmlentities() afterwards: <form action="<?=htmlentities($html->url())?>" ...>
    • Added means for handling named parameters like /addons/browse/type:1/cat:all
    • Changed the search engines and plugin page URLs so they have the same format as the other browse pages (...browse/type:1)
    • Gave plugins an addontype ID (7) and a constant (ADDON_PLUGIN) for consistency's sake
    • Many minor bug fixes, most of which picked right off the feedback page.
    • Added thickbox JS lightbox effect (with sancus) and used it for the preview display page.


  • parameters, presence of custom sql everywhere. reason why there is custom sql -- dynamic l10n and the need to use the translations table for simple things like sort by name.
  • wenzel is pretty sure we can get the cake models to automatically take the dynamic l10n into consideration. the problem is that we don't do l10n until after we're done finding.
  • agreed to take a look at improving cake models once more important things are taken care of.


  • sancus is sick, working on adding test data in for sandbox backend. should be done tomorrow/friday.

load testing

  • took hours to actually be able to contact all the machines we needed to. problems with VPNs, can't get there from here, etc. one thread, one process (4 simulatneous hits) worked fine with four nodes. 200 simultaneous requests made stage angry and everything went down.
  • need to be able to load test AMO against a real app node. worried about releasing to the cluster based on the current tests.


  • looking at rss, xss
  • looking at models and l10n to see if we can cakify and avoid as much manual sql


  • silverorange
  • partnering stuff
  • spend some time on code audit


  • updated languages; sk locale coming in this week


  • services
  • maintenance script
  • permissions - db_acl too granular and the plugin from noswad that was looked at is overkill for our permissions requirements


  • preview, push before beta that contains visual updates and sandbox functionality
  • migrate the world


  • half our controllers still use scaffolding; will remove that.
  • will yank controllers that don't have unique code in them.

Jan 17th

  • What about the beta release? Which are the issues we *do* have to address before we can release that one? (wenzel, fligtar)
    • priority stuff for beta
      • browse page needs fixing, per comments in feedback etc
      • public sandbox stuff
    • multi-app support less important for now
    • (shaver) No Language Packs for add-ons right now
  • dates
    • beta before the end of this week
    • feb 5th is the "publication date"
    • clouserw going to push trunk to preview immediately
  • recommended list
    • trailfire, couple of search engines
  • sandbox
    • display page(warning, etc)
    • need to go into your user profile and turn sandbox on to be able to see it
    • filtering search/browse
    • addons with inactive set should not be shown anywhere, that's the deletion method
    • third state for add-ons? sandbox completely ungoverned, public side very polished, do we need a middle ground?
      • not for release, maybe later
  • browse page
    • ugly, needs to be browsable, large boxes with full information detail
    • maybe more compact javascript-using layout later
  • plugins page
    • static, not dynamic for now
    • remove redundant links to different platforms that lead to the same place anyway? (shaver, morgamic) nah, leave it alone for now
  • What about the browse pages? These are the pages -by far- the most people in the feedback complained about. What changes do we want to/have to do there? What are the medals for? (wenzel)
    • medals are for recommended add-ons, remove them for now.
  • People asked: Why is the navigation on the right? Will we move it to the left? If so, will we keep it on the right for rtl locales? (wenzel)
    • Generally agreed that we should move it to the left.
  • What do we do to the search box at the bottom, now that we put a search box on top of every page? (wenzel)
    • (random babble) generally agreed to take this out.
  • There was a suggestion to put the version/date line back under the addon name on the display page to remove redundancy (similar to v1). I like it. Should we do that? What about the release notes then, though (currently underneath the version/date line)? (wenzel)
    • (shaver) can move the version/date line but not sure what to do with releasenotes so leave 'em alone for now.
  • How/where do we want to display an addon's rating, based on the reviews it got? (wenzel)
    • (shaver) review ratings near review details/text
    • (wenzel) sort browse by review ratings
  • wenzel:
    • User info page
    • Wiki cleanups and addressing feedback
    • Breadcrumbs and search on top of every page
    • Browsing addon types improved but not "fixed" yet. -- Which ones are supposed to be shown here, and where should the links go?

Jan 10th

  • Could we use jquery to augment "flash" behavior and not require user context to change in order to flash an error or "success" message? Think flickr.
  • Bugzilla vs. Wiki vs. Grouphub -- are we okay here?
  • Wiki is out of date -- need to make a good effort at updating some our pages.
  • shaver:
    • Conduit stuff this week, situation seems mostly fine now
    • thanks to fligtar, morgamic, clouserw for their help with that
    • working with PR team on outreach stuff for launch
    • policy stuff in good shape with legal
    • connected to security guys now
    • working on HTML for bannerized front page
  • wenzel:
    • lots of bug fixes
    • working on bread crumbs now
  • morgamic:
    • did naval add-ons research in Mexico
    • some bouncer stuff, lots of meetings
    • going to work on plugin pages
  • fligtar:
    • model work to allow editing of arbitrary languages from any other languages
    • work on add-item and edit-item and edit-version
    • translation-box element for working with other languages
    • preview uploader work soon
    • reviewer side of the sandbox as well
  • sancus:
    • EULA and privacy policy work, should be in
    • browse page improvments
    • public sandbox after EULA and privacy policy
  • lars:
    • some more migration work, quel surprise
    • lars' time will be less available ASAP
    • still need to do load testing, oy
    • need to decide on migration changes for names
  • clouserw:
    • reset script and test data and localization-merging, etc.
    • updated alpha
    • ngettext support for more complicated pluralization
  • multiple app support?
    • fligtar: developer side handles it
    • public side: would need install changes for tbird, f.e.
    • public side: no current notion of current app, would need to parameterize and test
    • cameron: doing it twice is inefficient
    • sancus: will start a task list for multiple app support todos
    • we'll complete the app requirements by next meeting, solicit input from other-app communities
    • morgamic: doing multi-app support after we're done Remora will provide a better experience for the other-app users
    • shaver: how many add-ons support multiple applications?
    • morgamic: non-web presentation for non-browser clients (tbird, sunbird, etc.) might be an alternative option
  • morgamic: use jquery for the flash behaviour to give better experience (currently redirects after the flash, loses context)
  • morgamic: we have three to-do sources (wiki, bugzilla, grouphub)
  • shaver: they're sort of hierarchical for me: wiki = maybe; bugzilla = yes, but we're not sure when;
  • shaver: TM 3.1 list for post-Remora feature work somewhere to capture
  • morgamic: docs need update (server requirements and deployment stuff needs to be clarified for production deployment; other general cleanup)
    • shaver: would be nice to defer non-vital doc update until after launch
  • fligtar: web services and XML, yay

Jan 3rd

  • (clouserw) How are we going to handle multiple products for an addon? (Specifically, the interface for downloading them)
  • Read the Alpha Feedback
  • (sancus)
    • Why do we want to support add-ons that use seperate files to support multiple products?
    • How do we want to do this? Do we need an applications_files table? Do we want to keep applications_versions? Remove it?
  • (fligtar - absent)
    • I reskinned additem for rustico and revamped it for the new d-l10n structure. This involved modifying beforeFind and beforeSave to allow for using a locale other than the currently selected. So, if you want to find/save something to the German translation regardless of what locale is currently in use, you just set $this->Addon->useLang = 'de';
    • I became very concerned about Cake's willingness to save any POST data passed to it. (For example, if someone were just updating their add-on and sent a post var data[Addon][status]=1 or data[Addon][downloadcount]=10000000000000, Cake would happily save it.) I think Cake has something to help fix that, but I wanted something better, so I made a method of the Amo component called filterFields, where you can pass a whitelist or a blacklist of fields and it will return a copy of the passed array with the appropriate filters. I think that anywhere we use save() we need to use this, as it only takes one vulnerability to get in. An example usage: $addonData = $this->Amo->filterFields($this-data['Addon'], array(), array('id', 'status'));
    • I also made a translation box element that I'm going to use everywhere people need to translate things into multiple languages.
    • This week I'm working on edit and editversion and adding/updating all my tests for additem. After that, I will go back and finish the review queue for the sandbox. After that, I'll fix the developers index. After THAT, I'll start on the preview uploader if no one else has.
  • Cameron
    • It seems that people whose names are one word in v2 have that inserted as their last name in remora. I think that it should be inserted as their first name.
    • Additionally, I think that no matter what we decide to do with regard to the above, the entire v2 name should be used as the nickname for remora.
    • Subscribe to http://blog.mozilla.com/webdev/comments/feed/ if you're interested :) (and you should be!)
  • (wenzel)
    • I got rid of some inconsistencies in the German translations.
    • The review ratings are now limited to 0-10; the model validates that as well before saving.
    • On the addons display page, the three reviews can now be clicked to show them immediately without clicking on the whole review page.
    • The language selector in the footer now points to the current language, not en-US all the time.
    • next, I will make a search engine page like v2. Will also add test data.

Dec 20th

  • File browser in public pages?
    • No real reason to keep it under lock-and-key
  • Alpha status
  • CSS confusion
  • Tasks left to do
    • Make the site presentable and consistent
  • Pages that need "design attention" -- once we define what "attention" and "design" mean.
    • ./addons/browse.thtml
    • ./addons/rss/addons.thtml
    • ./addons/recommended.thtml
    • ./addons/browseTypes.thtml
    • ./addons/browseCategories.thtml
    • ./addons/home.thtml
    • ./addons/display.thtml
    • ./search/index.thtml
    • ./search/rss/index.thtml
    • ./reviews/add.thtml
    • ./reviews/display.thtml
    • ./users/login.thtml
    • ./users/email/pwreset_plain.thtml
    • ./users/email/confirm_plain.thtml
    • ./users/register.thtml
    • ./users/pwreset.thtml
    • ./users/edit.thtml
  • Dynamic l10n and SQL
    • Do we need to be able to delete a singular translation? (yes, you can't have just a null field)
    • Will remove the FK relationships that may delete too much.
  • Add non-user data to default SQL that doesn't change over time (including translations)
  • Set auto increment numbers in default sql to arbitrarily high numbers
  • Migrate add-ons into a production SQL to be used when we set up the public alpha
  • Delete unused static l10n strings, truncate developers/reviewers stuff with bad IDs

Dec 13th


  • We don't have a software license field in the db yet


  • We're leaving several fields (like application.name and addons.privacypolicy) out of the translations table - I think they should be in there:
    • Someone might want to translate the $field sometime and we could support that (and with our current system we get it for free!)
    • Consistency - all strings in the same place
    • It would let us echo the lang="" tags around the strings if they were in a different language
    • Fields I'd consider changing:
      • addons.developercomments
      • addons.homepage
      • addons.eula
      • addons.privacypolicy
      • applications.name
      • applications.shortname
      • cannedresponses.name
      • cannedresponses.response
      • platforms.name
      • platforms.shortname
      • previews.caption
      • reviews.title
      • reviews.body
      • versions.releasenotes


  • working on static and dynamic l10n all over the place. Any other places left to be localized?
  • What about that text in the footnote? It's obviously c&ped from here and I don't think "Applications" fits well here.
  • was working on:
    • fixing search to search through d-l10n fields
    • statically localizing the forums (with morgamic)
    • building a so-far-complete German localization


  • about that translations table:
    • missing foreign key relationships in remora.sql
      • addons.name --> translations.id
      • addons.description --> translations.id
      • addons.summary --> translations.id
      • addontypes.name --> translations.id
      • tags.name --> translations.id
      • tags.description --> translations.id
    • it would also be handy to also have "on delete casacade" for these foreign key constraints
    • should translations be shared? Assume not.
  • about the locale strings
    • should there be a table showing all the valid locale strings?


  • schedule, etc.
  • basecamp love
  • alpha milestone To-do's

Dec 6


  • dynamic l10n not quite done yet, back-end rearrangements being made by clouserw, supposedly done except for tests/optimization
    • work done, we're going ahead with the change
  • static l10n on public pages !done
    • forum l10n (fred)
    • click-through-ness (morgamic will fish through pages, though everyone should)
    • invalid id errors should have a localized template, and we should probably come up with a better default way to display errors like these to use throughout (haven't seen one -- correct me if I'm wrong)
  • starting work on install script
    • porting over current impl is okay
    • additional logic for sandbox
    • present eula before install -- shaver said AJAX
    • get rid of download count


  • vacation schedules still fuzzy! when are you guys going to be taking off?


  • working on policy
  • need web design list
    • morgamic will look through pages for design needs