Webdev/Meetings/2014/July 1: Difference between revisions
Jump to navigation
Jump to search
No edit summary |
(Added my stuff.) |
||
| Line 18: | Line 18: | ||
Any updates with our libraries or with libraries we use? Anyone looking for help with a library they maintain? | Any updates with our libraries or with libraries we use? Anyone looking for help with a library they maintain? | ||
* (ErikRose) There's now an official #peep channel. It has occasional flurries of activity. | |||
* (ErikRose) Moved [https://github.com/mozilla/spiderflunky spiderflunky] repo to the mozilla org, as it's starting to get attention. | |||
* (irc nick) Topic | * (irc nick) Topic | ||
| Line 29: | Line 31: | ||
* (Osmose) Input/edits wanted: Etherpad for writing down what activities/things webdev does, what [[Webdev/Goals|goals]] they contribute to, who owns them, and what could be improved: https://etherpad.mozilla.org/mkelly-webdev-activities | * (Osmose) Input/edits wanted: Etherpad for writing down what activities/things webdev does, what [[Webdev/Goals|goals]] they contribute to, who owns them, and what could be improved: https://etherpad.mozilla.org/mkelly-webdev-activities | ||
* (ErikRose) concurrent.futures is a great way to do parallelism. | |||
from futures import ProcessPoolExecutor, as_completed | |||
from more_itertools import consume, chunked | |||
def run_jobs() | |||
... | |||
with ProcessPoolExecutor() as pool: | |||
futures = [pool.submit(index_chunk, tree, tree_indexers, paths) for | |||
paths in chunked(unignored, 50)] | |||
consume(show_progress(futures)) | |||
def index_chunk(tree, tree_indexers, paths): | |||
"""Do a bunch of computation, and throw the results in elasticsearch.""" | |||
def show_progress(futures): | |||
"""Show progress and yield results as futures complete.""" | |||
num_jobs = len(futures) | |||
for num_done, future in enumerate(as_completed(futures), 1): | |||
print num_done, 'of', num_jobs, 'jobs done.' | |||
yield future | |||
Revision as of 16:47, 1 July 2014
| Webdev Meeting | |
|---|---|
| Date: | July 1st, 2014 |
| Time: | 10:00 AM Pacific |
| How to Join | |
| Vidyo: |
|
| Teleconferencing: | Room 9 798 |
| IRC: | #webdev on irc.mozilla.org |
| Notes | |
| Etherpad: | https://etherpad.mozilla.org/webdev-2014-07-01 |
| Note-taker: | Osmose |
Webdev Extravaganzas are open to the public and serve as a gathering point for anyone in the Mozilla community who is interested in web development and what Mozilla has been doing in it.
The meeting will be streamed and recorded on Air Mozilla: https://air.mozilla.org/webdev-extravaganza-july-2014/
Shipping Celebration
What did we ship this month? Alternatively, what didn't we ship to spare the world the horror?
- (bsternth) MobilePartners Release: Salesforce moar integration, Fancy Legal Agreements, No More Labs Hardware
Open-Source Citizenship
Any updates with our libraries or with libraries we use? Anyone looking for help with a library they maintain?
- (ErikRose) There's now an official #peep channel. It has occasional flurries of activity.
- (ErikRose) Moved spiderflunky repo to the mozilla org, as it's starting to get attention.
- (irc nick) Topic
New Hires / Interns / Volunteers
Anyone new that we want to induct into the secret order?
- (irc nick of new person) Name / Role / Team
The Bikeshed / Roundtable
Anything else to talk about?
- (Osmose) Input/edits wanted: Etherpad for writing down what activities/things webdev does, what goals they contribute to, who owns them, and what could be improved: https://etherpad.mozilla.org/mkelly-webdev-activities
- (ErikRose) concurrent.futures is a great way to do parallelism.
from futures import ProcessPoolExecutor, as_completed
from more_itertools import consume, chunked
def run_jobs()
...
with ProcessPoolExecutor() as pool:
futures = [pool.submit(index_chunk, tree, tree_indexers, paths) for
paths in chunked(unignored, 50)]
consume(show_progress(futures))
def index_chunk(tree, tree_indexers, paths):
"""Do a bunch of computation, and throw the results in elasticsearch."""
def show_progress(futures):
"""Show progress and yield results as futures complete."""
num_jobs = len(futures)
for num_done, future in enumerate(as_completed(futures), 1):
print num_done, 'of', num_jobs, 'jobs done.'
yield future