29
edits
Changes
Long-running computation update
3. Canceling execution of a notebook cell doesn't cancel any spark jobs that might be running in the background. If your spark commands seem to be hanging, try running `sc.cancelAllJobs()`.
=== How can I keep running after closing the notebook? ===
For long-running computation, it might be nice to close the notebook (and the ssh session) and look at the results later. Unfortunately, '''all cell output will be lost when a notebook is closed''' (for the running cell). To alleviate this, there are a few options:
1. Have everything output to a variable. These values should still be available when you reconnect.
2. Put %%capture at the beginning of the cell to store all output. [https://ipython.org/ipython-doc/3/interactive/magics.html#cellmagic-capture See the documentation].
