Confirmed users
574
edits
Webdev/Meetings/2014/July 1: Difference between revisions
Added my stuff.
No edit summary |
(Added my stuff.) |
||
| Line 18: | Line 18: | ||
Any updates with our libraries or with libraries we use? Anyone looking for help with a library they maintain? | Any updates with our libraries or with libraries we use? Anyone looking for help with a library they maintain? | ||
* (ErikRose) There's now an official #peep channel. It has occasional flurries of activity. | |||
* (ErikRose) Moved [https://github.com/mozilla/spiderflunky spiderflunky] repo to the mozilla org, as it's starting to get attention. | |||
* (irc nick) Topic | * (irc nick) Topic | ||
| Line 29: | Line 31: | ||
* (Osmose) Input/edits wanted: Etherpad for writing down what activities/things webdev does, what [[Webdev/Goals|goals]] they contribute to, who owns them, and what could be improved: https://etherpad.mozilla.org/mkelly-webdev-activities | * (Osmose) Input/edits wanted: Etherpad for writing down what activities/things webdev does, what [[Webdev/Goals|goals]] they contribute to, who owns them, and what could be improved: https://etherpad.mozilla.org/mkelly-webdev-activities | ||
* (ErikRose) concurrent.futures is a great way to do parallelism. | |||
from futures import ProcessPoolExecutor, as_completed | |||
from more_itertools import consume, chunked | |||
def run_jobs() | |||
... | |||
with ProcessPoolExecutor() as pool: | |||
futures = [pool.submit(index_chunk, tree, tree_indexers, paths) for | |||
paths in chunked(unignored, 50)] | |||
consume(show_progress(futures)) | |||
def index_chunk(tree, tree_indexers, paths): | |||
"""Do a bunch of computation, and throw the results in elasticsearch.""" | |||
def show_progress(futures): | |||
"""Show progress and yield results as futures complete.""" | |||
num_jobs = len(futures) | |||
for num_done, future in enumerate(as_completed(futures), 1): | |||
print num_done, 'of', num_jobs, 'jobs done.' | |||
yield future | |||