Compatibility/Meetings/Sync w Mike

From MozillaWiki
Jump to: navigation, search

August 26 2020

OKR 2.2 Manual testing of 5 open source editor libraries (SV)

We'd like to start this OKR and do some testing on it. Could you provide more info on what should we focus our testing? Is there a list of issues that we should verify?

mike: the DOM team is focusing on contenteditable bugs this half

mike: for example, -- if you search for firefox

mike: do these issues still reproduce?

mike: leave a comment only if it is no longer reproducible, skip bug that we don't understand, create a list with the issue that are still reproducible.

mike: Test plan - what to test in an editor

mike: some of these pages have a sample editor available, e.g.,

oana: has a similar test/run the code

[FYI] Needscontact verification (SV)

We finished verifying Needscontact issues:

- Fixed: 179

- Invalid: 14

- Reproducible: 286

cipri: ~40% issues gone

oana: a lot of the issues were related to the certificates and TLS

Moderation process enhancement (SV)

After setting the "accepted" milestone to an issue during the moderation process, the issue is automatically closed and removed from the private repo.

We noticed that by setting the milestone to a batch of issues (25 issues per page) not all the issues are removed from the repo (usually 3 or 4 are still displayed) and a page refresh is needed in order to remove them. Is this the correct behavior?

mike: ?????? this sounds like a Github bug.

mike: related to this,

mike: I added "accepted: invalid" and "accepted: incomplete" milestones

mike: I'll let y'all know when it's actually ready

A11Y issue verification (SV)

We should schedule a testing session to verify all currently reported issues after fixes were merged.

Let as know when to perform this task.

mike: i should be done with a lot of these maybe in the next 2 weeks.

oana: perhaps we'll wait until the end of the quarter to test them all in a batch.

July 29 2020

Login Credentials special sites (SV)

Whenever we triage issues that require login on banking/governamental/educational sites, we close them as incomplete with a message where we ask for more details (test account credentials, error, logs).

We don't ask for the reporter's credentials, but a test account credentials if the reporter is able to create an extra account.

For example:

Should we change the message into something different (like - "could you provide a test account credential to our email address?")

mike: email on our personal account, continue discussion there

Report site issue (SV)

We've seen a decrease in the number of reported issues. Could it be related to "Report Site Issue" button being moved Add-ons section?

Most of the users are used to old version, when the button was in Settings (quick access and user friendly). Are there any limitations in regards to moving it back to Settings?


mike: will talk to Tom about tackling this issue.

mike: related, trying to come up with a standard metric for valid vs invalid: might be interesting for y'all to look at and comment (feel free).

cipri: as a personal note, it seems like people think that the title is where they should put the description, and for the description they just put random letters to pass the validation. unsure how we can improve this.

[FYI] A11Y testing (SV)

We started testing and reporting accessibility issues. I've added a label `a11y` to have quick access to all of them.

mike: this is awesome, thank you.

[FYI] Interventions/Overrides re-check (SV)

Firefox Nightly 82 Soft Freeze will be on 2020-08-20, thus we will perform the re-check on 6th of August.


[FYI] Romania Webcompat Test Party (SV)

There will be a test party organized by our SV community. I don't have more info about it for now, will come with an update.

oana: hopefully see a spike in reported issues, sports sites is the theme. maybe in a couple of months a tracking protection party.

Update: 10-11 August 2020

Update on ETP shimming project (Mike)

mike: just landed today.

oana: we can start testing tomorrow, or early next week.

July 1 2020

Monthly Interventions check (SV)

Firefox Nightly 81 soft freeze will be on 2020-07-23, thus Oana will perform the verification? When is the best time to do it?

mike: perform it 2 week after release - 10-13 July

Also in regards to progress, is it ok to add it here ?

mike: yes

cipri - we can link to a document with status for all the sites, but only comment the ones that need removal

mike: let's aim to do this regularly, 2 weeks after the new release date. that should give enough time for testing, and landing patches.

ETP shimming (SV)

Since Firefox Nightly 80 has landed 80, we've checked to see it we can start testing, but the `extensions.webcompat.enable_shims` is not yet available. When do you think it will land?

mike: - when the issue is resolved-fixed - we can start testing it - wait until it is in Nightly.


mike: report issues on `Product: Web Compatibility` - `Component: Interventions`

Triage help (SV)

oana: since Cipri will be 2 weeks in PTO, I might need some help on triaging, if the number of issues exceeds, lets say 50-70 on Wednesday, in order to finish the week with 0 untriagged/unmoderated issues.

cipri: July 6 through 17

mike: ok! i can help.

oana: i will ping when i need help, maybe on wednesday.

Keyboard a11y testing of form (SV)

oana: I did a bit of testing around this topic and found a few issues, and also that is not possible to report a new issue just by using the keyboard (it stops at category selection).

[Intro - Activity - Use the web without a mouse]

- Tab key moves to the next link, form element, or button.

- Shift+Tab moves to the previous link, form element, or button.

- Enter or space bar activates the current link or button.

- Try the arrow keys, Escape key, and other keys, too.


OKR preview (mike)

Sitewait / Needscontact issues re-check (SV)

We didn't run a re-check on the Sitewait/Needscontact issues in the last quarter and we think it would be a good ideea to clean up the list. When do you think would be the best time to do this?

- at the end of the quarter if the other OKR's are completed

- punt this over to the next quarter

mike: let's focus on sitewait/needsconctact first

mike: and create a reserve card for needsdiagnosis

June 10 2020

Google Sitewait issues (SV)

We perform TSCI testing every week and there is no activity on Google Sitewait issues.

We were thinking of adding a comment on the issues (some of them are quite old and no comment on them for months), or maybe on the next quarter perform a SiteWait verification and add a comment at that time.

mike: leaving a comment would be great. would going back be annoying?

oana, cipri: no, no issue.

mike: then I should send another email to Google.

Montly Interventions 4th run (SV)

We were wondering when should be the best time to perform Interventions & Overrides re-check since Soft Freeze will occur on 2020-06-26. Should we consider testing them between 19-23 of June?

mike: v12 tracking bug:

mike: can we aim for the 19th?

cipri: or the 17th or the 18th.

oana: I'm only taking 2 days next week PTO, then more days another week.

Webcompat Staging testing (SV)

When is it scheduled to be released in production?

We were considering testing staging for 1-2 hours on Thursday, if its not too urgent.

mike: Release new Monday, Thursday is perfect. Thanks!

mike: (speaking of form v2, some stats)

oana: it might be interesting to look at total number of users who are reporting now.

[FYI] All Hands participation (SV)

We can take part in discussions related to future planning, what went well/what to improve.

mike: feel free to propose agenda items here!

Q3 OKRs / KPIs (mike)

1. End every week with 0 untriaged and unmoderated bugs (as measured by the webcompat metrics dashboard).

Why: we want to ensure that we’re able to react in a timely manner to incoming site breakage reports, and not grow a large backlog. Also, see if we’re able to handle the incoming work with only 2 headcount.

2. Manual testing of top 5 open source JS editor libraries (to be identified by team). Why: this is part of a strategic goal to improve interoperability in the DOM editor space.

For example, ,

Also, triage existing bugs related to Firefox in each repo to be compiled and highlighted to team

3. (aspiration) Author and ship 3 interventions in the webcompat system addon. Why: the team has an interest in making these contributions, and there’s a real benefit to having more folks understand how these systems work.

oana: another idea, do another check on needsdiagnosis, sitewait, needscontact.

oana: related to #2, like we did for COVID, testing educational sites would be good. but, accounts are difficult to get.

May 27 2020

Google Tier 1 Experiment testing (SV)

Adam: "Google is running a Tier 1 experiment right now on Firefox for Android"

SV: Are we going to do some Google Tier 1 testing?

Mike: I'll follow up with Adam and find out.

Tracking Protection help wanted (SV)

Are there any news on how we can help with TP issues?

Mike: alternate proposal here:

Mike: unresolved question: Tracking Protection issues.

SV: continue reporting TP issues.

UA Override research (SV)

We've started gathering a list of webcompat issue where UA Override is needed. We were thinking of doing some reasearch on them, when time permits and come up with a proposal.

Mike: once we know it's probably safe, filing a bug in the Web Compatibility::Interventions component

"" "Testing in Firefox for Android" (maybe have to make sure this is set up first:

May 06 2020

(FYI) Hunt from Home (SV)

We received a low number of issues.

Ovi was asigned on this task only for a limited time, but in case more issues will be reported he will help.


oana: i think the limited number of sites made people uncomfortable to report issues.


oana: probably they used the usual form

cipri: we're not seeing a lot of new reports

mike: it goes until May 11

oana: maybe we can reschedule something like this later on

oana: we're still doing COVID testing

(FYI) Team changes impact (SV)

Sergiu is no longer in our team and for now, we don't consider this to have a high impact on OKRs, unless a large number of issues pops up.

mike: we'll have to see how this affects us, probably fine with incoming bugs, less manual testing perhaps

Machine Learning (aka "Emilut") (SV)

There seems to be a problem related to the bot - it is not working properly and it is closing a low number of issues (or none sometimes).

mike: we should open a bug, and ping karl...

Action item: SV report bug on webcompat

Tracking Protection (mike)

Thinking about ways our team can help out with these issues.

Building XPI for Addons (oana)

oana: I will file a bug if building locally doesn't work.

April 22 2020

Monthly Intervention check (SV)

Issue Salesforce Community intervention tracking bug - which is a "meta" bug, contains a list of webcompat issues


1. Test all webcompat issues from the list and create a Bugzilla issue to remove the intervention if the issue is no longer reproducible (for each domain)? The meta bug remains open until all issues no longer reproduce without the intervention.

2. Test 1-2 webcompat issues and if one of them is still reproducible with interventions off, consider the meta bug still reproducible.

Mike: I think testing a few sites is sufficient.

Oana: And if we find those are fixed, we can test all of them to verify.

Interventions are added based on site popularity.

SV team can also propose sites to which interventions could be added -

UA Overides documentation:

Focus issues not reproducible on Firefox Preview Nightly (SV)

We noticed that you closed the issue that are reproducible in Focus but not in Fenix as WON'T FIX.


1. We proceed on closing similar future issues as WON'T FIX.

2. (also) Forward the issues to the Focus repo.

Sergiu: how about we open a new Focus bug, and close the webcompat one:

Mike: COOL

Tracking Protection Strict issues - desktop (SV)

Previously, François Marier told us he is not interested in TP-Strict issues since it is expected to break stuff. Also, the default ETP setting in desktop is Standard.

We've been reporting Bugzilla issues which are reproducible on desktop with ETP-Strict enabled. Should we continue doing that or should we just skip them?

Action item for Mike: Get clarity on this.

SV Team: Continue reporting them for now.

Moderation improvements?

Action item for SV Team - document typical work flow, improvements - leave a comment.

April 10 2020

Tracking Protection same domain (SV)

Meta bug: Embedded content (Twitter/Instagram/Facebook) is not displayed with ETP Strict enabled

- Should we report different issues related to missing embedded Twitter/Instagram/Facebook domains for every site we encounter or add them to the "See also" of the Meta bug

Action item: Let's ask Tom (we pinged him on the internal channel).

Tom reply: Add separate issues fro every domain

Interventions & Overrides recheck (SV)

Beta has been migrated to Fenix. We will run the montly Interventions and Overrides on Release, unless we can get a Fenix build without the interventions (?).

Action item: Test on Release.

SV team - run Interventions on Monday/Tuesday (with Release)

mike: We should ask Dennis what we can do here... how we can disable before about:compat is ready. Custom build?

sergiu: Another solution would be to take the last known Beta build from FTP server and test with that (before the Fenix migration).

Pull to Refresh issues on Fenix (SV)

We are receiving lately a bunch of issues tied to page refreshing while performing different actions on a page.

- Should we consider closing them (if that is the case) as duplicates of the meta-bug: ?

- Maybe find a more relevant duplicate from the list: ?

Action item: Mark them as duplicate of (according to Petru Lingurar comment)

March 25 2020

(FYI) Q2 OKR planning (SV)

We added our ideas/suggestions in the Q2 OKRs planning in a comment

Covid-19 exploratory testing (SV)

The sites and the progress is visible here

- add new sites to the list (maybe we can crowdsource with the team, or with the larger Mozilla community?)

- maybe we could ask for help testing + diagnosis, outreach, etc.

- there's a #covid19 channel on slack, we could scrape the comments / link for other sites (or ask them for suggestions)

March 12 2020

Timing of Interventions verification & Bugs (Mike)

Checklist for Intervention Release:

Meta bug for each release:

Release Calendar:

If we identify sites that no longer require interventions or overrides, can we please file a bug in Bugzilla in Web Compatibility::Interventions & Tooling to remove it? Following along in GitHub comments can get confusing.

It's probably a good idea to create a link to the original landing patch via See Also. And leave a comment.

We should probably time new testing to happen after we land a new version -- that way we have a few weeks to work on removing uneeded interventions.

Issues related to Login (SV)

We have noticed a lot of Login related issues. Most of them we cannot reproduce.

mike: This sounds like

mike: There's a pref as well `network.cookie.sameSite.laxByDefault`

Action item: link the Bugzilla bug to the webcompat Login issues

explanation here: ->

(FYI) ML is helping a lot (SV)

(11 issues closed in the last 24 hours)

The great sickness (World)

We're trying to protect ourselves as best as we can

(FYI) Test Party Webcompat (SV)

- successful testing event - 13 participants - 2h - 45issue reported

- participants enjoyed having the report site issue directly on the browser

February 19 2020

FYI Anonymous reporting feedback (SV)

Overall it is working fine, moderating takes about 1h for 50-60 issues.

It would be useful to have the public link in the issue, in order to moderate and triage it at the same time - Karl is working on it.

Concern: issues will pile up like crazy since a lot of them are being reported (seems to be more than before /day), and we also need to focus on the OKRs, and also take PTO soon in this quarter - ML can help with this.

mike: will sent an email to Nemo about ML, CC SV team in the email

Anonymous reporting closed issues before being accepted (SV)

Some issues get closed before being accepted due to GH's lag/or by mistake and no longer end up in Public repository (thus they get lost).

Is there any mechanism to bypass this behavior so no issue will get lost?

mike: reopen a private closed issue and try to accept it again (test on staging)

Action item: SV team to check the flow for private closed issues, log issue in case it does not work as expected

Fenix issue on Tablet layout (SV)

Tablets receive phone UA layout on Fenix istead of tablet layout.

We recommended to test on Firefox Release or Firefox Beta until Fenix issue is fixed. What do you think ?

mike: continue testing on Firefox Release or Firefox Beta

New Webcompat form (SV)

Any news if the new form will remain or will it be closed/disabled?

mike: leave it on and collect data

oana: maybe request feedback from the users

mike: ksenia is writing tests for the new form, probably she will re-write the new form (JS - implementation)

mike: make small improvements

mike: meeting with Open Innovation and discuss about it next Monday

February 5 2020

Anonymous reporting on Staging (SV)

Are there manual tests needed here?

mike: is this on staging yet?

sergiu: not sure, but we saw it's merged.

mike: I'll get in touch with Karl, and we'll get some documentation. some light testing would be good.

mike: I'll deploy to staging in a few hours.

Action item: Testing anon-reporting tomorrow (06.02.2020) on staging

Action items following the All Hands (SV)

Are there any action items for us, as a result of the Berlin talks?

Any Tracking Protection testing news?

mike: keep doing what we are doing

mike: no news on TP for now

mike: focus on interventions/TSCI?

Presentations (good to watch):

Berlin 2020 - Firefox Web Technologies All Hands - get a sense of future directions

Berlin 2020 - Thursday Plenary - different cultures & communication

Interventions testing (SV)

Is it ok to test the Interventions on Fennec Beta?

mike: it's probably good enough!

sergiu: we still have issues that are reproducible in beta/release but not Fenix Nightly.

mike: let's go ahead and close those.

January 22 2020

Non-compat issues rule (SV) - involved payment, valid data was needed to create account, not reproducible after clearing cookies.

Update on (anon) private reporting for (Mike)

mike: we hope to have this ready by the end of the week!

Update on Sergiu's surfing trip.

Went snow surfing.

January 09 2020

Enterprise TSCI (SV)

More context on

sergiu: what's up with this?

mike: this is a list of sites that are important for Enterprise. We may build a TSCI report for these sites this quarter, or propose a new Outreachy mentorship for it.

sergiu: probably we can come back to these after OKRs are finished.

oana: probably the next sites we test are 10 to 20.

Top Site Testing (10 - 20) (Mike)

mike: let's focus on severity-critical and -important!

Content Moderation (Mike)

mike: if anything illegal or harmful comes up during triage, please ping miketaylr immediately. no need to comment on issues, mike will take care of it until we have a better process in place.

mike: please read the below resources as well, they may be helpful.

December 11 2019

ML issues with >95% probability (Sergiu)

We find it quite useful and we should continue closing these issues without checking them. We did not see any users that commented on closed "ml-probability-high" stuff.

Although, we ran into a contradictory labeling situation on a valid login related bug where the issue was filed by a user ( which was not labeled with `ml-probability-high` and another issue filed anonymously ( which was labeled (`ml-needsdiagnosis-false` and `ml-probability-high`).

Quality of reports (miketaylr)

mike: Rosana wondered if the subcategories were helpful. sergiu: When we don't have enough info from the description, it can be helpful. But so many are invalid.

oana: It's useful to have that icon, nice for the user.

Form v2 Metrics (miketaylr)


GA completion metrics:

TP (resource blocking) & ETP (cooking blocking) site breakage (Oana)

Are there any news from Security team, related to mobile device issues? Should we continue logging them on "Firefox - Protection UI" ?

mike: We can continue to file mobile ones there, according to Privacy team.

November 20 2019

ML issues probability > 95% (Cipri)

Starting next week we could close all the issues that have a probability above 95% to be invalid.

mike: what’s the rationale for waiting a week? ciprian: just wanting to gather some feedback, or know if it’s safe. mike: let’s go ahead and close them.

New Webcompat form (Oana)

We see more and more issue reported using the new form. We did not see any improvement in the quality of the issues reported, the content (description) is still vague and with few details.

oana: it’s probably easier to report an issue. looks nicer, but the content – it’s still as vague as ever. it’s probably better to focus on how to improve quality. mike: matt (the UX designer) shared some thoughts here:

Incompat add-on (Sergiu)

sergiu: we got in touch with the developer, and now besides its original functionality, when we’re on Github looking at a web-bug it tells us how many issues are open and closed for a given domain. we can take a look at closed issues and see if the current bug is a duplicate.

mike: wowowowow, we should send him something.

Triage backlog

OI team trying to understand v1 vs v2 numbers (Mike)

November 6 2019 testing

oana: Will there be another iteration on the new form? If needed, we can take a shot at it.

Action item: test the new form while doing TSCI top site testing, report bugs and suggestions.

ML add-on

sergiu: Do we have some data about how the ML performed? Is there a way for it to go live?

Proposal: turn the system on and have it

sergiu: has the system ever predicted something at 100%?

Another idea: automatically close (and label) issues at 95%, flag ones at 90% for manual verification, etc.

Facebook testing

cipri: You mentioned some time ago, about a session of Facebook testing. Are we still on?

Action item: mike to send profile of test account to st peter. Also, ask FB to flag the user to not get banned since it’s a test account.

October 16 2019

Secure Connection failed issues

Can we close them if the certificate is not A grade? mike: yes, except TSCI level sites.

ML add-on

How come apparently identical issues get different results from the add-on? mike: here's an older document with the experiment design, mike: some word salad about statistics mike: how accurate is it? sergiu: it's about 75% accurate oana: for duplicates, it's not useful. sergiu: is the bot already trained? or still being trained? oana: let's continue testing for 2 weeks. sergiu: we could improve this addon by having some kind of interaction method. when it's wrong, give it some feedback as to why it's wrong (or right). mike to follow up with nemo about this. oana: this is only possible when we get a notification. - new design

oana: Can I have the link to the new form? I want to fiddle around with some automated tests, as part of my own curiosity. mike: send oana the cookie name/value for

IRC follow up

mike to send email to dennis