MozillaWiki

Compatibility/Meetings/Sync w Honza: Difference between revisions

Page Discussion

Compatibility/Meetings/Sync w Honza (view source)

Revision as of 13:00, 16 April 2024

26,947 bytes added ,  16 April 2024
sync 16th of April
(meeting minutes October 17)
(sync 16th of April)
 
(11 intermediate revisions by 2 users not shown)
Line 1: Line 1:
== April 16 ==
=== New dashboard system (Honza) ===
Honza: I've shared the link, I am curios if you can login. It takes some time to load. This is the first version.
The second link shows a graph. It is a submission count, shows the reports submitted per day. +- 2000 per day. As you can see, the numbers are low at first, due to the experiment period. Then it jumps, since more users have had this new reporting system.
As you can see, a lot of data is present. From the overview dashboard, we can see the number of reports received by site, by day, with comment, without comment, based on OS, etc.
We can also see that if an issue is currently known or not.
Paul: How does QA fits in this new dashboard? We will have a shorter list that QA can test and try to reproduce. Not sure how we should tackle this, with what to start. Triaging 1000s of issues per day is not really feasable.
Honza: Right. So we need features that will help QA in their triaging process, so that they can focus on the valid, reproducible issues. The first approach is to see what domain has the most reports. From the data shown, the first indication is that youtube has the most issue, so we should focus there. Sorting those issues somehow is another priority.
Paul: Maybe a keyword in comments, a root cause, would help.
Honza: Right, we should look at the most promising comments. You mentioned the kanban board with top most issues.
Paul: We can somehow filter that most of the people reporting the same incomplete issue, raises the urgency of checking that issue.
Honza: That is the future, lower quality, high number of reprots. So that is our focus now, to somehome implement some feature to make small progress.
Paul: Maybe the ability to search by comments and be ranked based on the information provided.
Honza: I agree but this more for statistics to figure it out where there could be more potential issues.
Paul: We could schedule a meeting and brainstorm ideas on how to make it better, to extract the most info from the reports we are receving.
Raul: Will the Ml-bot still be active?
Honza: Sure.
Paul: We will have as mentioned, an internal meeting, and we will present the brainstorming ideas to the team regarding the features that would be useful.
Honza: Sure, both triaging and work tracking should be supported. How much time do you thing you need?
Paul: Most likley this week.
Raul: Let's say we have everything in place, what if we get 200 valid reports a day because most likely we won't be able to keep up.
Honza: We don't know exactly yet, we will have to check how it goes and decide then. We will identify the bottleneck issues, maybe we will see that we are received less valid reports, and uppermanagement will need to take action towards this.
== March 19 ==
=== QA help needed for some Meta bugs(Honza) ===
* QA help needed for:
  * Bug [1778322](https://bugzilla.mozilla.org/show_bug.cgi?id=1778322) - [meta] Bad, removed or changed file extension after download
  * Bug [939897](https://bugzilla.mozilla.org/show_bug.cgi?id=939897) - [meta] Support shifting / splitting flex items between continuations of a fragmented flex container (flexbox content truncated when printing)
* Go over all linked bugs and test
* Leave one comment in each meta summarizing the status of the dependencies
Honza: Go over these 2 meta bugs and test the dependencies and blockers.
Raul: Should we look on the closed or the open bugs?
Honza: Basically the open bugs but if you see something worth testing on the closed ones you should test them as well.
=== Quick update on the new Dashboard(Honza) ===
It is not deployed yet, but it should be ready for the next meeting. Maybe Dennis will do a quick demo in the meantime.
Paul: Doing queries inside the dashboard to export metrics and numbers would be nice.
Honza: As soon as you have the dashboard in your hands, we will look for and encourage feedback. This will be taken into consideration so that your work is more easily done.
=== Internal updates (SV) ===
Calin: Just making sure you are aware about my long PTO.
Paul: We will also have help while Calin is in PTO, to take the share of the load off Radu and Raul.
== March 5 ==
=== Bugzilla triage (SV) ===
We have completed the Bugzilla backlog triage. Progress and data [here](https://docs.google.com/spreadsheets/d/1Vun7EDha-R6p7t_rgOTPu70K1FLCMTrukmHvHZr3vOQ/edit#gid=663205964).
=== Firefox for Android 126 will be moved to mozilla-central (SV) ===
Is this something that might affect our project?
Honza: I do not foresee any troubles. Maybe Sync with Tom regarding the Automation Interventions run, just to be on the safe side. I will talk with Dennis.
=== New dashboard System (Honza) ===
Honza: I will share my screen, just to get an idea. We shipped the new reporting tool, available in Firefox - report broken sites.
Watching the graph, we can see the number of received reports during the experimental phase, roughly 200 per day. Then we can see the numbers when the tool was shipped in 123. We can see 10x times more, which corresponds to what we were expecting. A lot of data.
Tyler, from the Data Team, put together a dashboard to view what we are getting from users (broken site summary). The blue column shows the reports with comments, the red one without comments. In parallel, we are trying to group it in a way, to see the most useful comments. Ksenia is using ML to filter out without comments, or with high probability to be helpful. We can also see that most Firefox users are running Windows.
We can see the reports by OS, and by reports in the last 24hr, also per site. We can see the Trend and what is important to look at. Something similar might be available in the QA Dashboard.
The bottom piece allows filtering by URL and OS. Some things will be improved, this is just to get a general idea. It would be great to see the actual comment in the table.
Based on this information, I would like to hear feedback, e.g. what would you like to see on the dashboard, useful data, to see only reports with comments, only useful comments, high probability for it to be useful, and automatic translations of comments. etc.
We will try to automate sorting the issues automatically so it shows up on top, so we can connect the dots more easily.
The next step might be that we need screenshots to better understand what is the issue.
Dennis is working on the QA Dashboard, the only blocker is it is not yet deployed. Soon will have something for you to look over. Feedback will be most welcomed, we need to figure out what features you need to understand and do your work.
Raul: Can the links be shared?
Honza: Sure, will leave a DM with them.
Honza: It would be a lot different work. For example, in Git, we will get more valuable reports. The new report will get a lot more reports per day, 2k+, but there will be less info there. We need to figure out how to be useful. Get the most from the data. Our goal is to locate the major problems that Firefox is having when surfing the web. As soon as the QA Dashboard is deployed, we will let you know.
== February 20 ==
=== Welcome Radu to the team 🎉(SV) ====
We would like to welcome our new team member, Radu. Yesterday he got the LDAP setup email, we're still configuring things and he started looking through ramp up documentation.
Mozilla email: railioaie@mozilla.com
Github account: https://github.com/ailioaie
Slack: railioaie
Let's add him to our Github repo, with all the permissions, and hand out calendar invites for meetings. Also on Slack channels.
# Increase in the number of incomplete reports received (SV)
At the start of the week, an increase in incomplete reports (no description mostly) has been observed. Did we released the new reporting flow?
Honza: In that new reporting flow, that the users see to create a new report, there is a link, "send more info", that link goes to webcompat.com, so I imagine that people are using that link as well to report issues.
Raul: That makes sense.
Honza: Yes, the new pop-up is just simple. Some people might get confused and try to send quickly a report on webcompat.com without filling in any data.
=== Bugzilla triage updates (SV) ===
Currently, we are at 70% coverage rate. Progress can be seen [here](https://docs.google.com/spreadsheets/d/1Vun7EDha-R6p7t_rgOTPu70K1FLCMTrukmHvHZr3vOQ/edit#gid=663205964). So far, things are going smoothly.
Honza: Awesome. We are also doing a triage session today.
The meeting today will be shorter, with not too many agenda items. We will use the rest of the time from the meeting for triage.
== February 6 ==
=== Reporting tool updates (Honza) ===
Getting around 100 reports a day for a 10% user base. We will probably get x10 on the full population.
Release of the reporting tool around 23rd of Feb. Once it is shipped the next step is to go for mobile as well. We gathered a lot of data but it's not deployed yet. We are making progress.
Dashboard will be done in Q1 and we'll probably start using it in Q2.
Expecting that in Q2 it will be mostly to identify any issues with it and adjust the flow to suit our needs.
As soon it gets deployed we will share the link and we will start working together.
After the deployment we'll need to work together on:
* identifying the candidates for QA
* confirming them by finding STR
* prioritizing the issues and diagnosing the problems.
We will have to be smart about how we put our time into it.
As soon we have the triage dashboard, we will need some kind of metric to track the progress if we are going in the right direction. We have to come up with a reasonable number we can influence.
The time spent on a website can indicate if the site is broken or not. How visited is a website (popular) on Firefox vs Chrome?
Tomorrow we have an important kick-off meeting to try to come up with that metric. A lot of important data people will attend.
Tracking data available in Q2, and start using everything in H2.
Sometime in Q2, we will run a study on users via Zoom on how they perceive Webcompat in Firefox and in general.
== January 23 ==
=== Bugzilla Backlog triage (Honza) ===
Honza: We have an additional task now, we need to triage the list of opened bugs in our Webcompat repo.
Link:  https://bugzilla.mozilla.org/buglist.cgi?resolution=---&f1=keywords&v1=webcompat%3A%2A&bug_status=UNCONFIRMED&bug_status=NEW&list_id=16870269&product=Web%20Compatibility&component=Desktop&component=Mobile&query_format=advanced&o1=notregexp
We append the webcompa keyword. We are looking at the latest ones, recently updated. We will need help triaging this, but starting from the end.
If they are reproducible, we need to see what label we are adding. It might be an OKR.
Paul: We will do a document to assign between QA
Honza: There is no deadline, it depends on your estimates.
Paul: Is this more important than Top 100? This might affect the estimate for the Top 100.
Honza: It depends what impact it might have. I am not sure of that, but definetly we should keep doing our Top 100 OKR. We can play with the definition, to see how quickly it can be done, and base our estmations on that. This will be further discussed in the other meeting. We should just burn down that backlog as we can, as there is no deadline.
Raul: How are these issues going to be split between us?
Honza: Yes, we will start from the latest, and QA will start from the bottom.
Paul: Will the labels be decided in the next meeting?
Honza: Yes, hopefully we can 100% be done with the labeling system. We are already using some keywords. Any report with the "webcompat:*" keyword is excluded from the list.
Raul: We had a plan previously for a similar triaging process. Is this something we should talk in the next meeting with the team?
Honza: Yes, that would be a good idea.
== January 9 ==
=== 2024 Plan for QA (SV) ===
Since we are unsure how our work style will change/be after the transition to the Dashboard, we have no new proposals for 2024. We will stick to the 2023 plan until then.
For 2024, here is the updated [plan](https://github.com/mozilla/webcompat-team-okrs/projects/26) containing our OKRs
Should we still split the OKRs for H1 and H2, meaning that we will create a new project for H2 in our Github repo, importing the tasks from H1? Or we can keep this one for the whole year?
Honza: I would wait, for Q1 nothing changing too much.
We should come up with a metric to measure the state of the webcompat and in the next quarter move it in the direction we want to.
Let's wait on Q1 and in the 2nd quarter we will have a solid base for OKRs.
=== QA Dashboard ===
Do we have any updates on this?
- The Dashboard is not ready yet, I assume we will need about 2 more months for it to be done.
- There is a lot of data gathered through the Reporting tool from the Experiment that is live (see link from meeting)
- So far, we receive in average of 200 reports per day, and it's growing. Mostly noise for now, but we managed to gather some data and categorize some reports.
- The experiment is set at 10% of the population with the new Reporting tool. We are now around 7m, and we might get to 10m. That will mean an increase in the daily reports received.
- The experiment will run for a couple more weeks, it's due on February 14th.
When it's done, we need to see if we will keep the Categories drop down, and ship to all users, likely in Firefox 122, sometime in February. By then, we expect to see at least the Dashboard being usable.
Paul: Firefox 122 is set to be released on the 23rd of January, it will be hard to release in 122 if the code still needs to land.
Honza. Good point, that might be for version 123.
Paul: That gives us more time to decide what to deploy.
Honza: I'll ask Karen.
Paul: You can also use a Nimbus rollout to release it if it's only about flipping a pref.
Honza: Sure, most of the stuff is enabled by prefs. Good point
=== Labeling (Honza) ===
Honza: For the next 2 months, we are still foreseeing Github triage. However, what we produce of data with the Dashboard, should be trusted info.
Eventually, we will come to the platform team with data, showing the most encountered and important issues, and how fixing them would impact our browser. We need to explain this data. This is one of the reasons why we are switching and collecting data.
The big picture, the real State of Firefox Webcompat, is recommending trustful. There should be minimum human judgment for this. Each category should have simple steps to reproduce, and it should be as clear as possible. This might be easier in the new system, where people can choose the category.
Paul: Do we want to automate this somehow, or we will still rely on QA to confirm the categories?
Honza: I'm not sure we can automate this, although it would be nice.
Paul: I think it would be hard to automate, as you'll need something specific like a keyword in the report to categorize it automatically. And each problem is specific.
Paul: Also, some users might not select the correct category.
Honza: There are 2 things, picking the category - which we know the limitations for this, and the 2nd thing should be creating labels. For now, we have 4 clear categories: Unsupported, ETP, Private Browsing, and  Performance.
Paul: I think we should at least offer an "Other" category especially if it's mandatory to select one.
If nothing applies, there is no label. Whatever we have in February, feedback for this will be taken into consideration.
Honza: As a first update/enhancement for this flow, we are aiming to ship this for Firefox mobile as well.
Also, we are thinking about screenshots provided by users - but that might be harder due to storing possibilities and other limitations.
So a lot fewer labels, but trustful.
Raul: Should we edit the labeling descriptions on Git Hub?
Honza: I think for now we shouldn't edit anything since in around 2 months we are migrating to the new dashboard.
Raul: Should we create a trend label for the private browsing issues? We normally have a label for private but not for trends.
Honza: No action for now. Q2 will shed some light, as we will have more data then.
=== Google is going to kill cookies soon ===
What is Mozilla's position on this? Any plans to do the same?
Honza: I do not know what will happen on Firefox's side, but that will be interesting from the webcompat point of view.
== December 12 2023 ==
=== Top 100 Testing and Blockers (SV) ===
The number of discovered issues is lower compared to when we started (most are known issues - discovered in previous runs). Also, for Google and other sites where a valid mobile number is needed, we are blocked due to our test account mobile numbers being used for other test accounts. Some blockers have been observed for Financial sites where a valid banking account is needed, or where a mobile app is needed to perform various operations (Netflix, Google-related apps, etc)
=== Upcoming Holidays and Top 100 (SV) ===
Just a heads-up that this month we might not finish the whole Top 100 Test Suite for Q4 - Batch 1 due to upcoming holidays and PTO
Honza: Sure, please add the PTOs to the Calendar.
=== Work Week in Berlin (Honza) ===
Honza: I'll share the DevTools folder. This is a summary of planning for 2024. Our top-level goal is to improve webcompat compatibility.
*'''The first part is Webdriver BiDi'''
WebDriver Bidi will involve more browsers, like Chrome and Safari, as it involves a protocol for crossbrowsing operability - for Automation - and Pupetteer. That would be the first use case, for Automation.
Raul: Will that involve a programming language?
Honza: Pupeteer is a library for JavaScript. It is in the experimental phase. We will be collecting feedback.
Raul: Could QA be involved in running and creating automated tests?
Honza: Sure, let's think about that for the next year.
Honza: The main goal is to build the BiDi protocol and build an ecosystem on top of it which could be used for running automated tests and debug browsers.
*'''The second Part of the Work Week is the Reactive Webcompat'''
Most people use the reporting tool in Nightly now. Partially this is on purpose, due to the number of reports used. We want to see the big picture: the state of WebCompat. The idea is to get more reports. This will be achieved with the new reporting tool - shipping in Firefox 121 - the plan is to cover mobile as well, not just desktop. We also need to triage the incoming reports, and since the number is high (thousands) we are looking to automate some parts. That is addressed in Reactive webcompat composed of 3 parts: Reporting tool, Triage dashboard (likely just for QA), and Knowledge Base.
Based on the data collected from Firefox not being supported, we will know what actions to take, to understand the webcompatibility around Firefox.
Collecting engagement data will show some clarity as well. How much time users are spending on certain sites, top sites popular in Firefox, etc. This might show us if Firefox is gaining popularity or losing popularity. For example, some sites might not be popular in Firefox but are popular in Chrome, and we need to see why.
Private browsing mode will be looked into it.
We want to understand the state of Firefox from all of this. We can then recommend to the Platform teams what to work on.
*'''The third part is Dev Tools'''
The same goal - we want to contribute here as well. Provide tools that enable developers to debug and optimize pages to work well in Firefox with a focus on performance and reliability.
We want to help them to diagnose webcompat problems if they happen. Quickly and efficiently. We do not want to complicate the existing system. Whatever we have, should be more quick and efficient, with small features that will make a bigger impact.
Many issues happen on the Production version of the site. Features will help them diagnose bugs in the Production version using Firefox DevTools.
Baseline- all features are well supported across major browsers, they belong here.
We are looking to run an Audit regarding this on pages.
Raul: For remote debugging, we have some feedback.
Honza: Great, let's talk about it in the next meeting.
Honza: Also, the profiler will be more user-friendly.
All of this culminates in making Firefox more compatible, and thus better.
== November 17 2023 ==
=== Trends ===
Paul: I've set up the metrics for the TREND OKR, and we can go through each label present in the list and used by QA, to see which are irrelevant and which are relevant, and to provide more clarity over some of them and the process that they are being used.
Honza: What happens when you can use more than 1 label for an issue?
Raul: There are cases where we do that, and we have reports that we have moved to `needsdiagnosis`, as QA was unable to pinpoint exactly the label behind the report. For example, the video is not playing because the play button is not responding. That is a case where 2 labels can be used.
Honza: Is there a way we can improve this?
Raul: We could use just 1 label where more than 1 label is needed. This means that QA should stick to 1 labe, as they see fit for the issue.
Honza: Can we create more labels?
Paul: If the need arises, sure.
Honza: I can see the `graphic glitch` label. What exactly does that mean?
Raul: Elements not being rendered properly, broken items from a graphic point of view, text overflowing, elements overlapping other elements, cut text, etc
Honza: So, would it be better to rename this label? Something that is more related to the issue.
Paul: We could try using layout instead of the graphic glitch title.
Honza: Sure, that sounds better.
Honza: What tools are you using for the `performance` trend label? What helps you identify the issue as being related to `performance`?
Raul: We are using the Task Manager, and the performance tab from the `about:performance` config option, to see and compare if Firefox is using more resources compared to another browser
Paul: Regarding the `other` or `unknown` label? Should we keep it?
Honza: We could keep it and use it in some cases when we are not sure how to classify the issue, for example, if we are trying to pick the best fit between 3 or more labels.
Honza: We can amend the label list after this meeting. Surely, we will use this system in the new dashboard system, and most likely this will evolve.
Honza: We will discuss with the team regarding the labels used for TRENDS, to make further clarifications. Thanks for that.
== October 31 2023 ==
=== QA Triage Trends ===
Since our last meeting, we've started using the new format on how we submit the QA Triage Trends.
We are looking for feedback: https://github.com/mozilla/webcompat-team-okrs/issues/275#issuecomment-1772768903
Regarding the trend metrics, we are still working on the document.
We were wondering if the total number of issues (per label) received each month regardless if it's reproducible or not would be enough for the metrics.
Paul: Should we go this deep, or is it enough to mention a link and several total issues per milestone, instead of copying the link for each issue?
Honza: I am also thinking about the new system. We can search for individual reports by label. An important thing to add, if we are not sure about a label, it is best to not label the issue. We should use categories where we are certain. To answer your question, the numbers are more important, is that trend growing or not? Higher management wants to have some kind of metrics to see the impact the platform is making, we are trying to understand all the user reports, estimate trends, which issues have the most impact, and how to prioritize them so that we can give all the relevant info to the platform team. Whatever numbers we have should help the higher management to see if the platform team is going in the right direction.
Paul: How do we measure the impact?
Honza: That is the big question. They need to know that the actions we are taking are making things better or worse. We can not base this metric on the number of reports, getting more reports vs getting fewer reports, the goal of the system is to have as much data as possible. It is in our interest to have more reports. We might want to identify things like Firefox not supported, and we can somehow follow that trend. We would not collect the number of reports, but the domains regarding this. We can measure how quickly the platform fixes issues. We can do issue scoring, like the State of Webcompat report, top 20 issues, that we think the platform should fix. Or the popularity of sites, using Firefox Telemetry and comparing it with Chrome Telemetry. Like are there sites used by a lot of people just in Chrome? We should come up with the same data to see if we are in the right direction. The trends, the numbers, is part of that goal. Are we able to spot some trends, and trust the numbers?
The overall numbers indicate the trend, what is our main focus.
Paul: So will show the numbers based on links, and show them by month.
Honza: yes.
=== Gathering info for the New Reporting System (Honza) ===
Honza: For the new system, I am interested in the triage process of the QA. That means that you would assign labels to issues. Right now you are using GitHub label, which would likely remain the same.
Paul: Every bug tracker has a way to separate things, and labels are the way to go, free to create and easier to use.
Honza: When you split the work, as per your email that Dennis asked for info, what if there was one more person?
Raul: We would have a batch each day and split the issues in 3.
Paul: Or we could assign the issues directly to the QA member.
Raul: It would help us if we could mass-assign a batch directly to a member without assigning each issue manually.
Paul: Right now we are using keywords for specific OKRs. I'm not sure if its feasible to add a label for each issue (for e.g. counting number of issues triaged in a week, for now, we use "[qa_44/2023]" or "[inv_44/2023]" for investigated, 44 stands for week 44). The new dashboard should have something in order to count the issues received each week.
Raul: These keywords help us a lot when counting our issues in different metrics/reports.
Honza: The current system is based on the reports received on Git Hub. But in the new system, it will be harder to filter that out because more reports would be received. Maybe we would have to mass close an issue that has the same domain.
Paul: The bot also closes issues that he finds inappropriate or irrelevant.
Honza: Right, we want to measure only the work that has been done by the triage team.
The issues will be grouped by the domain. There might be some tooling that groups issues that has similar description. Once we have more data, we will start learning how to make the Triage process much better.
== October 17 2023 ==
== October 17 2023 ==


Ctanase
14

edits

Retrieved from "https://wiki.mozilla.org/Special:MobileDiff/1248596...1250539"