Breakpad/WorkWeek: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
 
(13 intermediate revisions by 2 users not shown)
Line 14: Line 14:
* Develop better/best practices for PostgreSQL
* Develop better/best practices for PostgreSQL


= Monday =
= TODO =
* Orientation, introductions, goals
* Performance
* Access to everything (IT)
**pg.conf changes
* Report from IT on environments
**fs/os changes
* Discuss hardware, optimization
**query tuning
* Database optimization, structure
***monitor
***reports
***interactive query tuning
****memory checks
**index/schema check
**performance monitoring
***install tools
***check results
**re-tuning & hw recommendations
**application performance check
***monitor/processor -- python
***reporting -- php
**full text for signatures
* Failover
**Uptime requirements
**Prepare failover plan
**Failover config & scripts
* Partitioning
**Partition scheme check
**Check for bad queries
**Rolloff plans
**Deal with archive partition
* IT Admin
**Monitoring -- plans
***Implementation
**Backup & Redundancy plan
***Implementation
* Capacity Planning
**(see above) Archiving Discussion
***Implement archiving?
***RRD?
**Data growth analysis
**New feature planning
* [https://intranet.mozilla.org/Socorro:DumpingDumpTables DumpingDumpTables]


= Tuesday =
= Schedule =
* Software and query optimization
* Tuesday
* How to use explain properly
**Partitioning
* Discuss replication options, splitting workload away from operational DB
***RRD/Archving?
 
**Python Check
= Wednesday =
**Full Text
* Database normalization (whiteboard) -- need more? less?
**Initial Replication
* Future schema changes?
* Wednesday
 
**Continue replication
= Thursday =
**Query analysis etc.
= Friday =
**Capacity planning data
**Bug list and future feature planning
**Interactive query checking
* Thursday
**Capacity Planning session
***Schema changes?
**PHP analysis?
**Deploy replication?
* Friday
**Query analysis pt.2
**Troubleshooting


= Travel Schedules =
= Travel Schedules =
* aking
* aking
** Arrive SJ @ 10:30 - will miss first half of Monday's meetings
** Arrive SJ @ 10:30 - will miss first half of Monday's meetings
** Depart SJ @ 17:10
** Depart SJ @ 17:10 - cab @ 3pmish
** Wild Palms
** Wild Palms
* lars
* lars
Line 46: Line 89:
** Depart SJ @ 19:20 Friday, March 13
** Depart SJ @ 19:20 Friday, March 13
** Wild Palms
** Wild Palms
= Indexes to Remove =
* productdims.productdims_product_version_key (not used, table is small)
* reports_########.reports_########_product_version_key
* topcrashers.idx_topcrashers_total
* urldims.urldims_pkay
* drop urldims.id altogether
* urldims.urldims_url_domain_key
* create pkey on url
= Notable Queries =
* select * from pg_stat_user_tables where seq_scan > 100;
* select * from pg_stat_user_tables where seq_scan > 100 and pg_relation_size(relname) > 800000;
* select * from pg_statio_user_tables where relname = 'urldims';

Latest revision as of 22:54, 13 March 2009

Mission

Make the web better.

  • Harvest crash report data
  • Generate aggregate statistics on data to understand trends
  • Publish individual reports for drill-downs and forensics
  • Determine the cause and frequency of all major crashes
  • Help Mozilla focus development resources in areas of highest impact

Goals this Week

  • Understand why reporter queries time out
  • Evaluate partitioning strategy
  • Evaluate hardware configuration
  • Evaluate backup and replication methods
  • Develop better/best practices for PostgreSQL

TODO

  • Performance
    • pg.conf changes
    • fs/os changes
    • query tuning
      • monitor
      • reports
      • interactive query tuning
        • memory checks
    • index/schema check
    • performance monitoring
      • install tools
      • check results
    • re-tuning & hw recommendations
    • application performance check
      • monitor/processor -- python
      • reporting -- php
    • full text for signatures
  • Failover
    • Uptime requirements
    • Prepare failover plan
    • Failover config & scripts
  • Partitioning
    • Partition scheme check
    • Check for bad queries
    • Rolloff plans
    • Deal with archive partition
  • IT Admin
    • Monitoring -- plans
      • Implementation
    • Backup & Redundancy plan
      • Implementation
  • Capacity Planning
    • (see above) Archiving Discussion
      • Implement archiving?
      • RRD?
    • Data growth analysis
    • New feature planning
  • DumpingDumpTables

Schedule

  • Tuesday
    • Partitioning
      • RRD/Archving?
    • Python Check
    • Full Text
    • Initial Replication
  • Wednesday
    • Continue replication
    • Query analysis etc.
    • Capacity planning data
    • Bug list and future feature planning
    • Interactive query checking
  • Thursday
    • Capacity Planning session
      • Schema changes?
    • PHP analysis?
    • Deploy replication?
  • Friday
    • Query analysis pt.2
    • Troubleshooting

Travel Schedules

  • aking
    • Arrive SJ @ 10:30 - will miss first half of Monday's meetings
    • Depart SJ @ 17:10 - cab @ 3pmish
    • Wild Palms
  • lars
    • Arrive SJ @ 17:00 Sunday, March 8
    • Depart SF @ 22:12 Saturday, March 14
    • Sweaty Palms
  • morgamic
    • Arrive SJ @ 22:30 Sunday, March 8
    • Depart SJ @ 19:20 Friday, March 13
    • Wild Palms

Indexes to Remove

  • productdims.productdims_product_version_key (not used, table is small)
  • reports_########.reports_########_product_version_key
  • topcrashers.idx_topcrashers_total
  • urldims.urldims_pkay
  • drop urldims.id altogether
  • urldims.urldims_url_domain_key
  • create pkey on url

Notable Queries

  • select * from pg_stat_user_tables where seq_scan > 100;
  • select * from pg_stat_user_tables where seq_scan > 100 and pg_relation_size(relname) > 800000;
  • select * from pg_statio_user_tables where relname = 'urldims';
Retrieved from "https://wiki.mozilla.org/index.php?title=Breakpad/WorkWeek&oldid=133857"