ReleaseEngineering/How To/Fix Build4h not updating

• Nagios alert

<nagios-releng> Fri 02:57:18 PDT [4051] builddata.pub.build.mozilla.org:http file age - /buildjson/builds-4hr.js.gz is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - Last modified 0:11:57 ago - 2896 bytes in 0.003 second response time (http://m.mozilla.org/http+file+age+-+/buildjson/builds-4hr.js.gz)

• Causes
  • one possible reason for this is invalid data being imported on one or more buildbot-masters

• Fix

• check 'statusdb' for invalid entries:

mysql> select * from builds, build_properties, properties where builds.id = build_properties.build_id and build_properties.property_id = properties.id and name = 'request_times' and value like '"%';

build_properties.property_id = properties.id and name = 'request_times' and value like '"%';
+-----------+-------------+------------+----------+-----------+---------------------+---------------------+--------+------------------------------------------------+-----------+------+-------------+-----------+-----------+---------------+------------+--------------------------------------------------------------------------------------------------------------------------------------+
| id        | buildnumber | builder_id | slave_id | master_id | starttime           | endtime             | result | reason                                         | source_id | lost | property_id | build_id  | id        | name          | source     | value                    |
+-----------+-------------+------------+----------+-----------+---------------------+---------------------+--------+------------------------------------------------+-----------+------+-------------+-----------+-----------+---------------+------------+--------------------------------------------------------------------------------------------------------------------------------------+
| 110001696 |         705 |     615690 |    13071 |       213 | 2016-10-20 09:08:06 | 2016-10-20 10:32:59 |      2 | scheduler                                      |  16077227 |    0 |   403588549 | 110001696 | 403588549 | request_times | postrun.py | "{'128606532': 1476951285L}"                    |
| 110002178 |         707 |     615690 |    12017 |       213 | 2016-10-20 09:57:12 | 2016-10-20 11:21:48 |      2 | scheduler                                      |  16077868 |    0 |   403590537 | 110002178 | 403590537 | request_times | postrun.py | "{'128608528': 1476952425L}"                    |

• in this particular case the issue was that we had a number of rows with invalid request times, all coming from a single buildbot master.
• to fix it, someone had to decode the values as json, eval()'ing as python expressions, re-encoding as json, and writing back to the database.

• See also

 Bug 1311964