Changelog¶
2.0.1 - 2019-06-26¶
We neglected to include python_requires=
in our setup.py file, resulting in
confusion for Python 2 users who erroneously get packages for 2.0.0.
This is fixed in 2.0.1 and we have removed the 2.0.0 files from PyPI.
- Add python_requires entry to setup.py (GH#2807) Matthew Rocklin
- Correctly manage tasks beyond deque limit in TaskStream plot (GH#2797) Matthew Rocklin
- Fix diagnostics page for memory_limit=None (GH#2770) Brett Naul
2.0.0 - 2019-06-25¶
- Drop support for Python 2
- Relax warnings before release (GH#2796) Matthew Rocklin
- Deprecate –bokeh/–no-bokeh CLI (GH#2800) Tom Augspurger
- Typo in bokeh service_kwargs for dask-worker (GH#2783) Tom Augspurger
- Update command line cli options docs (GH#2794) James Bourbeau
- Remove “experimental” from TLS docs (GH#2793) James Bourbeau
- Add warnings around ncores= keywords (GH#2791) Matthew Rocklin
- Add –version option to scheduler and worker CLI (GH#2782) Tom Augspurger
- Raise when workers initialization times out (GH#2784) Tom Augspurger
- Replace ncores with nthreads throughout codebase (GH#2758) Matthew Rocklin
- Add unknown pytest markers (GH#2764) Tom Augspurger
- Delay lookup of allowed failures. (GH#2761) Tom Augspurger
- Change address -> worker in ColumnDataSource for nbytes plot (GH#2755) Matthew Rocklin
- Remove module state in Prometheus Handlers (GH#2760) Matthew Rocklin
- Add stress test for UCX (GH#2759) Matthew Rocklin
- Add nanny logs (GH#2744) Tom Augspurger
- Move some of the adaptive logic into the scheduler (GH#2735) Matthew Rocklin
- Add SpecCluster.new_worker_spec method (GH#2751) Matthew Rocklin
- Worker dashboard fixes (GH#2747) Matthew Rocklin
- Add async context managers to scheduler/worker classes (GH#2745) Matthew Rocklin
- Fix the resource key representation before sending graphs (GH#2733) Michael Spiegel
- Allow user to configure whether workers are daemon. (GH#2739) Caleb
- Pin pytest >=4 with pip in appveyor and python 3.5 (GH#2737) Matthew Rocklin
- Add Experimental UCX Comm (GH#2591) Ben Zaitlen Tom Augspurger Matthew Rocklin
- Close nannies gracefully (GH#2731) Matthew Rocklin
- add kwargs to progressbars (GH#2638) Manuel Garrido
- Add back LocalCluster.__repr__. (GH#2732) Loïc Estève
- Move bokeh module to dashboard (GH#2724) Matthew Rocklin
- Close clusters at exit (GH#2730) Matthew Rocklin
- Add SchedulerPlugin TaskState example (GH#2622) Matt Nicolls
- Add SpecificationCluster (GH#2675) Matthew Rocklin
- Replace register_worker_callbacks with worker plugins (GH#2453) Matthew Rocklin
- Proxy worker dashboards from scheduler dashboard (GH#2715) Ben Zaitlen
- Add docstring to Scheduler.check_idle_saturated (GH#2721) Matthew Rocklin
- Refer to LocalCluster in Client docstring (GH#2719) Matthew Rocklin
- Remove special casing of Scikit-Learn BaseEstimator serialization (GH#2713) Matthew Rocklin
- Fix two typos in Pub class docstring (GH#2714) Magnus Nord
- Support uploading files with multiple modules (GH#2587) Sam Grayson
- Change the main workers bokeh page to /status (GH#2689) Ben Zaitlen
- Cleanly stop periodic callbacks in Client (GH#2705) Matthew Rocklin
- Disable pan tool for the Progress, Byte Stored and Tasks Processing plot (GH#2703) Mathieu Dugré
- Except errors in Nanny’s memory monitor if process no longer exists (GH#2701) Matthew Rocklin
- Handle heartbeat when worker has just left (GH#2702) Matthew Rocklin
- Modify styling of histograms for many-worker dashboard plots (GH#2695) Mathieu Dugré
- Add method to wait for n workers before continuing (GH#2688) Daniel Farrell
- Support computation on delayed(None) (GH#2697) Matthew Rocklin
- Cleanup localcluster (GH#2693) Matthew Rocklin
- Use ‘temporary-directory’ from dask.config for Worker’s directory (GH#2654) Matthew Rocklin
- Remove support for Iterators and Queues (GH#2671) Matthew Rocklin
1.28.1 - 2019-05-13¶
This is a small bugfix release due to a config change upstream.
- Use config accessor method for “scheduler-address” (GH#2676) James Bourbeau
1.28.0 - 2019-05-08¶
- Add Type Attribute to TaskState (GH#2657) Matthew Rocklin
- Add waiting task count to progress title bar (GH#2663) James Bourbeau
- DOC: Clean up reference to cluster object (GH#2664) K.-Michael Aye
- Allow scheduler to politely close workers as part of shutdown (GH#2651) Matthew Rocklin
- Check direct_to_workers before using get_worker in Client (GH#2656) Matthew Rocklin
- Fixed comment regarding keeping existing level if less verbose (GH#2655) Brett Randall
- Add idle timeout to scheduler (GH#2652) Matthew Rocklin
- Avoid deprecation warnings (GH#2653) Matthew Rocklin
- Use an LRU cache for deserialized functions (GH#2623) Matthew Rocklin
- Rename Worker._close to Worker.close (GH#2650) Matthew Rocklin
- Add Comm closed bookkeeping (GH#2648) Matthew Rocklin
- Explain LocalCluster behavior in Client docstring (GH#2647) Matthew Rocklin
- Add last worker into KilledWorker exception to help debug (GH#2610) @plbertrand
- Set working worker class for dask-ssh (GH#2646) Martin Durant
- Add as_completed methods to docs (GH#2642) Jim Crist
- Add timeout to Client._reconnect (GH#2639) Jim Crist
- Limit test_spill_by_default memory, reenable it (GH#2633) Peter Andreas Entschev
- Use proper address in worker -> nanny comms (GH#2640) Jim Crist
- Fix deserialization of bytes chunks larger than 64MB (GH#2637) Peter Andreas Entschev
1.27.1 - 2019-04-29¶
- Adaptive: recommend close workers when any are idle (GH#2330) Michael Delgado
- Increase GC thresholds (GH#2624) Matthew Rocklin
- Add interface= keyword to LocalCluster (GH#2629) Matthew Rocklin
- Add worker_class argument to LocalCluster (GH#2625) Matthew Rocklin
- Remove Python 2.7 from testing matrix (GH#2631) Matthew Rocklin
- Add number of trials to diskutils test (GH#2630) Matthew Rocklin
- Fix parameter name in LocalCluster docstring (GH#2626) Loïc Estève
- Integrate stacktrace for low-level profiling (GH#2575) Peter Andreas Entschev
- Apply Black to standardize code styling (GH#2614) Matthew Rocklin
- added missing whitespace to start_worker cmd (GH#2613) condoratberlin
- Updated logging module doc links from docs.python.org/2 to docs.python.org/3. (GH#2635) Brett Randall
1.27.0 - 2019-04-12¶
- Add basic health endpoints to scheduler and worker bokeh. (GH#2607) amerkel2
- Improved description accuracy of –memory-limit option. (GH#2601) Brett Randall
- Check self.dependencies when looking at dependent tasks in memory (GH#2606) deepthirajagopalan7
- Add RabbitMQ SchedulerPlugin example (GH#2604) Matt Nicolls
- add resources to scheduler update_graph plugin (GH#2603) Matt Nicolls
- Use ensure_bytes in serialize_error (GH#2588) Matthew Rocklin
- Specify data storage explicitly from Worker constructor (GH#2600) Matthew Rocklin
- Change bokeh port keywords to dashboard_address (GH#2589) Matthew Rocklin
- .detach_() pytorch tensor to serialize data as numpy array. (GH#2586) Muammar El Khatib
- Add warning if creating scratch directories takes a long time (GH#2561) Matthew Rocklin
- Fix typo in pub-sub doc. (GH#2599) Loïc Estève
- Allow return_when=’FIRST_COMPLETED’ in wait (GH#2598) Nikos Tsaousis
- Forward kwargs through Nanny to Worker (GH#2596) Brian Chu
- Use ensure_dict instead of dict (GH#2594) James Bourbeau
- Specify protocol in LocalCluster (GH#2489) Matthew Rocklin
1.26.1 - 2019-03-29¶
- Fix LocalCluster to not overallocate memory when overcommitting threads per worker (GH#2541) George Sakkis
- Make closing resilient to lacking an address (GH#2542) Matthew Rocklin
- fix typo in comment (GH#2546) Brett Jurman
- Fix double init of prometheus metrics (GH#2544) Marco Neumann
- Skip test_duplicate_clients without bokeh. (GH#2553) Elliott Sales de Andrade
- Add blocked_handlers to servers (GH#2556) Chris White
- Always yield Server.handle_comm coroutine (GH#2559) Tom Augspurger
- Use yaml.safe_load (GH#2566) Matthew Rocklin
- Fetch executables from build root. (GH#2551) Elliott Sales de Andrade
- Fix Torando 6 test failures (GH#2570) Matthew Rocklin
- Fix test_sync_closed_loop (GH#2572) Matthew Rocklin
1.26.0 - 2019-02-25¶
- Update style to fix recent flake8 update (GH#2500) (GH#2509) Matthew Rocklin
- Fix typo in gen_cluster log message (GH#2503) Loïc Estève
- Allow KeyError when closing event loop (GH#2498) Matthew Rocklin
- Avoid thread testing for TCP ThreadPoolExecutor (GH#2510) Matthew Rocklin
- Find Futures inside SubgraphCallable (GH#2505) Jim Crist
- Avoid AttributeError when closing and sending a message (GH#2514) Matthew Rocklin
- Add deprecation warning to dask_mpi.py (GH#2522) Julia Kent
- Relax statistical profiling test (GH#2527) Matthew Rocklin
- Support alternative –remote-dask-worker SSHCluster() and dask-ssh CLI (GH#2526) Adam Beberg
- Iterate over full list of plugins in transition (GH#2518) Matthew Rocklin
- Create Prometheus Endpoint (GH#2499) Adam Beberg
- Use pytest.importorskip for prometheus test (GH#2533) Matthew Rocklin
- MAINT skip prometheus test when no installed (GH#2534) Olivier Grisel
- Fix intermittent testing failures (GH#2535) Matthew Rocklin
- Avoid using nprocs keyword in dask-ssh if set to one (GH#2531) Matthew Rocklin
- Bump minimum Tornado version to 5.0
1.25.3 - 2019-01-31¶
- Fix excess threading on missing connections (GH#2403) Daniel Farrell
- Fix typo in doc (GH#2457) Loïc Estève
- Start fewer but larger workers with LocalCluster (GH#2452) Matthew Rocklin
- Check for non-zero
length
first inread
loop (GH#2465) John Kirkham - DOC: Use of local cluster in script (GH#2462) Peter Killick
- DOC/API: Signature for base class write / read (GH#2472) Tom Augspurger
- Support Pytest 4 in Tests (GH#2478) Adam Beberg
- Ensure async behavior in event loop with LocalCluster (GH#2484) Matthew Rocklin
- Fix spurious CancelledError (GH#2485) Loïc Estève
- Properly reset dask.config scheduler and shuffle when closing the client (GH#2475) George Sakkis
- Make it more explict that resources are per worker. (GH#2470) Loïc Estève
- Remove references to center (GH#2488) Matthew Rocklin
- Expand client clearing timeout to 10s in testing (GH#2493) Matthew Rocklin
- Propagate key keyword in progressbar (GH#2492) Matthew Rocklin
- Use provided cluster’s IOLoop if present in Client (GH#2494) Matthew Rocklin
1.25.2 - 2019-01-04¶
- Clean up LocalCluster logging better in async mode (GH#2448) Matthew Rocklin
- Add short error message if bokeh cannot be imported (GH#2444) Dirk Petersen
- Add optional environment variables to Nanny (GH#2431) Matthew Rocklin
- Make the direct keyword docstring entries uniform (GH#2441) Matthew Rocklin
- Make LocalCluster.close async friendly (GH#2437) Matthew Rocklin
- gather_dep: don’t request dependencies we already found out we don’t want (GH#2428) tjb900
- Add parameters to Client.run docstring (GH#2429) Matthew Rocklin
- Support coroutines and async-def functions in run/run_scheduler (GH#2427) Matthew Rocklin
- Name threads in ThreadPoolExecutors (GH#2408) Matthew Rocklin
1.25.1 - 2018-12-15¶
- Serialize numpy.ma.masked objects properly (GH#2384) Jim Crist
- Turn off bokeh property validation in dashboard (GH#2387) Jim Crist
- Fully initialize WorkerState objects (GH#2388) Jim Crist
- Fix typo in scheduler docstring (GH#2393) Russ Bubley
- DOC: fix typo in distributed.worker.Worker docstring (GH#2395) Loïc Estève
- Remove clients and workers from event log after removal (GH#2394) tjb900
- Support msgpack 0.6.0 by providing length keywords (GH#2399) tjb900
- Use async-await on large messages test (GH#2404) Matthew Rocklin
- Fix race condition in normalize_collection (GH#2386) Jim Crist
- Fix redict collection after HighLevelGraph fix upstream (GH#2413) Matthew Rocklin
- Add a blocking argument to Lock.acquire() (GH#2412) Stephan Hoyer
- Fix long traceback test (GH#2417) Matthew Rocklin
- Update x509 certificates to current OpenSSL standards. (GH#2418) Diane Trout
1.25.0 - 2018-11-28¶
- Fixed the 404 error on the Scheduler Dashboard homepage (GH#2361) Michael Wheeler
- Consolidate two Worker classes into one (GH#2363) Matthew Rocklin
- Avoid warnings in pyarrow and msgpack (GH#2364) Matthew Rocklin
- Avoid race condition in Actor’s Future (GH#2374) Matthew Rocklin
- Support missing packages keyword in Client.get_versions (GH#2379) Matthew Rocklin
- Fixup serializing masked arrays (GH#2373) Jim Crist
1.24.2 - 2018-11-15¶
- Add support for Bokeh 1.0 (GH#2348) (GH#2356) Matthew Rocklin
- Fix regression that dropped support for Tornado 4 (GH#2353) Roy Wedge
- Avoid deprecation warnings (GH#2355) (GH#2357) Matthew Rocklin
- Fix typo in worker documentation (GH#2349) Tom Rochette
1.24.1 - 2018-11-09¶
- Use tornado’s builtin AnyThreadLoopEventPolicy (GH#2326) Matthew Rocklin
- Adjust TLS tests for openssl 1.1 (GH#2331) Marius van Niekerk
- Avoid setting event loop policy if within Jupyter notebook server (GH#2343) Matthew Rocklin
- Add preload script to conf (GH#2325) Guillaume Eynard-Bontemps
- Add serializer for Numpy masked arrays (GH#2335) Peter Killick
- Use psutil.Process.oneshot (GH#2339) NotSqrt
- Use worker SSL context when getting client from worker. (GH#2301) Anonymous
1.24.0 - 2018-10-26¶
- Remove Joblib Dask Backend from codebase (GH#2298) Matthew Rocklin
- Include worker tls protocol in Scheduler.restart (GH#2295) Matthew Rocklin
- Adapt to new Bokeh selection for 1.0 (GH#2292) Matthew Rocklin
- Add explicit retry method to Future and Client (GH#2299) Matthew Rocklin
- Point to main worker page in bokeh links (GH#2300) Matthew Rocklin
- Limit concurrency when gathering many times (GH#2303) Matthew Rocklin
- Add tls_cluster pytest fixture (GH#2302) Matthew Rocklin
- Convert ConnectionPool.open and active to properties (GH#2304) Matthew Rocklin
- change export_tb to format_tb (GH#2306) Eric Ma
- Redirect joblib page to dask-ml (GH#2307) Matthew Rocklin
- Include unserializable object in error message (GH#2310) Matthew Rocklin
- Import Mapping, Iterator, Set from collections.abc in Python 3 (GH#2315) Gaurav Sheni
- Extend Client.scatter docstring (GH#2320) Eric Ma
- Update for new flake8 (GH#2321) Matthew Rocklin
1.23.3 - 2018-10-05¶
- Err in dask serialization if not a NotImplementedError (GH#2251) Matthew Rocklin
- Protect against key missing from priority in GraphLayout (GH#2259) Matthew Rocklin
- Do not pull data twice in Client.gather (GH#2263) Adam Klein
- Add pytest fixture for cluster tests (GH#2262) Matthew Rocklin
- Cleanup bokeh callbacks (GH#2261) (GH#2278) Matthew Rocklin
- Fix bokeh error for memory_limit=None (GH#2255) Brett Naul
- Place large keywords into task graph in Client.map (GH#2281) Matthew Rocklin
- Remove redundant blosc threading code from protocol.numpy (GH#2284) Mike Gevaert
- Add ncores to workertable (GH#2289) Matthew Rocklin
- Support upload_file on files with no extension (GH#2290) Matthew Rocklin
1.23.2 - 2018-09-17¶
- Discard dependent rather than remove (GH#2250) Matthew Rocklin
- Use dask_sphinx_theme Matthew Rocklin
- Drop the Bokeh index page (GH#2241) John Kirkham
- Revert change to keep link relative (GH#2242) Matthew Rocklin
- docs: Fix broken AWS link in setup.rst file (GH#2240) Vladyslav Moisieienkov
- Return cancelled futures in as_completed (GH#2233) Chris White
1.23.1 - 2018-09-06¶
- Raise informative error when mixing futures between clients (GH#2227) Matthew Rocklin
- add byte_keys to unpack_remotedata call (GH#2232) Matthew Rocklin
- Add documentation for gist/rawgit for get_task_stream (GH#2236) Matthew Rocklin
- Quiet Client.close by waiting for scheduler stop signal (GH#2237) Matthew Rocklin
- Display system graphs nicely on different screen sizes (GH#2239) Derek Ludwig
- Mutate passed in workers dict in TaskStreamPlugin.rectangles (GH#2238) Matthew Rocklin
1.23.0 - 2018-08-30¶
- Add direct_to_workers to Client Matthew Rocklin
- Add Scheduler.proxy to workers Matthew Rocklin
- Implement Actors Matthew Rocklin
- Fix tooltip (GH#2168) Loïc Estève
- Fix scale / avoid returning coroutines (GH#2171) Joe Hamman
- Clarify dask-worker –nprocs (GH#2173) Yu Feng
- Concatenate all bytes of small messages in TCP comms (GH#2172) Matthew Rocklin
- Add dashboard_link property (GH#2176) Jacob Tomlinson
- Always offload to_frames (GH#2170) Matthew Rocklin
- Warn if desired port is already in use (GH#2191) (GH#2199) Matthew Rocklin
- Add profile page for event loop thread (GH#2144) Matthew Rocklin
- Use dispatch for dask serialization, also add sklearn, pytorch (GH#2175) Matthew Rocklin
- Handle corner cases with busy signal (GH#2182) Matthew Rocklin
- Check self.dependencies when looking at tasks in memory (GH#2196) Matthew Rocklin
- Add ability to log additional custom metrics from each worker (GH#2169) Loïc Estève
- Fix formatting when port is a tuple (GH#2204) Loïc Estève
- Describe what ZeroMQ is (GH#2211) Mike DePalatis
- Tiny typo fix (GH#2214) Anderson Banihirwe
- Add Python 3.7 to travis.yml (GH#2203) Matthew Rocklin
- Add plot= keyword to get_task_stream (GH#2198) Matthew Rocklin
- Add support for optional versions in Client.get_versions (GH#2216) Matthew Rocklin
- Add routes for solo bokeh figures in dashboard (GH#2185) Matthew Rocklin
- Be resilient to missing dep after busy signal (GH#2217) Matthew Rocklin
- Use CSS Grid to layout status page on the dashboard (GH#2213) Derek Ludwig and Luke Canavan
- Fix deserialization of queues on main ioloop thread (GH#2221) Matthew Rocklin
- Add a worker initialization function (GH#2201) Guillaume Eynard-Bontemps
- Collapse navbar in dashboard (GH#2223) Luke Canavan
1.22.1 - 2018-08-03¶
- Add worker_class= keyword to Nanny to support different worker types (GH#2147) Martin Durant
- Cleanup intermittent worker failures (GH#2152) (GH#2146) Matthew Rocklin
- Fix msgpack PendingDeprecationWarning for encoding=’utf-8’ (GH#2153) Olivier Grisel
- Make bokeh coloring deterministic using hash function (GH#2143) Matthew Rocklin
- Allow client to query the task stream plot (GH#2122) Matthew Rocklin
- Use PID and counter in thread names (GH#2084) (GH#2128) Dror Birkman
- Test that worker restrictions are cleared after cancellation (GH#2107) Matthew Rocklin
- Expand resources in graph_to_futures (GH#2131) Matthew Rocklin
- Add custom serialization support for pyarrow (GH#2115) Dave Hirschfeld
- Update dask-scheduler cli help text for preload (GH#2120) Matt Nicolls
- Added another nested parallelism test (GH#1710) Tom Augspurger
- insert newline by default after TextProgressBar (GH#1976) Phil Tooley
- Retire workers from scale (GH#2104) Matthew Rocklin
- Allow worker to refuse data requests with busy signal (GH#2092) Matthew Rocklin
- Don’t forget released keys (GH#2098) Matthew Rocklin
- Update example for stopping a worker (GH#2088) John Kirkham
- removed hardcoded value of memory terminate fraction from a log message (GH#2096) Bartosz Marcinkowski
- Adjust worker doc after change in config file location and treatment (GH#2094) Aurélien Ponte
- Prefer gathering data from same host (GH#2090) Matthew Rocklin
- Handle exceptions on deserialized comm with text error (GH#2093) Matthew Rocklin
- Fix typo in docstring (GH#2087) Loïc Estève
- Provide communication context to serialization functions (GH#2054) Matthew Rocklin
- Allow name to be explicitly passed in publish_dataset (GH#1995) Marius van Niekerk
- Avoid accessing Worker.scheduler_delay around yield point (GH#2074) Matthew Rocklin
- Support TB and PB in format bytes (GH#2072) Matthew Rocklin
- Add test for as_completed for loops in Python 2 (GH#2071) Matthew Rocklin
- Allow adaptive to exist without a cluster (GH#2064) Matthew Rocklin
- Have worker data transfer wait until recipient acknowledges (GH#2052) Matthew Rocklin
- Support async def functions in Client.sync (GH#2070) Matthew Rocklin
- Add asynchronous parameter to docstring of LocalCluster Matthew Rocklin
- Normalize address before comparison (GH#2066) Tom Augspurger
- Use ConnectionPool for Worker.scheduler Matthew Rocklin
- Avoid reference cycle in str_graph Matthew Rocklin
- Pull data outside of while loop in gather (GH#2059) Matthew Rocklin
1.22.0 - 2018-06-14¶
- Overhaul configuration (GH#1948) Matthew Rocklin
- Replace get= keyword with scheduler= (GH#1959) Matthew Rocklin
- Use tuples in msgpack (GH#2000) Matthew Rocklin and Marius van Niekerk
- Unify handling of high-volume connections (GH#1970) Matthew Rocklin
- Automatically scatter large arguments in joblib connector (GH#2020) (GH#2030) Olivier Grisel
- Turn click Python 3 locales failure into a warning (GH#2001) Matthew Rocklin
- Rely on dask implementation of sizeof (GH#2042) Matthew Rocklin
- Replace deprecated workers.iloc with workers.values() (GH#2013) Grant Jenks
- Introduce serialization families (GH#1912) Matthew Rocklin
- Add PubSub (GH#1999) Matthew Rocklin
- Add Dask stylesheet to documentation Matthew Rocklin
- Avoid recomputation on partially-complete results (GH#1840) Matthew Rocklin
- Use sys.prefix in popen for testing (GH#1954) Matthew Rocklin
- Include yaml files in manifest Matthew Rocklin
- Use self.sync so Client.processing works in asynchronous context (GH#1962) Henry Doupe
- Fix bug with bad repr on closed client (GH#1965) Matthew Rocklin
- Parse –death-timeout keyword in dask-worker (GH#1967) Matthew Rocklin
- Support serializers in BatchedSend (GH#1964) Matthew Rocklin
- Use normal serialization mechanisms to serialize published datasets (GH#1972) Matthew Rocklin
- Add security support to LocalCluster. (GH#1855) Marius van Niekerk
- add ConnectionPool.remove method (GH#1977) Tony Lorenzo
- Cleanly close workers when scheduler closes (GH#1981) Matthew Rocklin
- Add .pyz support in upload_file (GH#1781) @bmaisson
- add comm to packages (GH#1980) Matthew Rocklin
- Replace dask.set_options with dask.config.set Matthew Rocklin
- Exclude versions of sortedcontainers which do not have .iloc. (GH#1993) Russ Bubley
- Exclude gc statistics under PyPy (GH#1997) Marius van Niekerk
- Manage recent config and dataframe changes in dask (GH#2009) Matthew Rocklin
- Cleanup lingering clients in tests (GH#2012) Matthew Rocklin
- Use timeouts during Client._ensure_connected (GH#2011) Martin Durant
- Avoid reference cycle in joblib backend (GH#2014) Matthew Rocklin, also Olivier Grisel
- DOC: fixed test example (GH#2017) Tom Augspurger
- Add worker_key parameter to Adaptive (GH#1992) Matthew Rocklin
- Prioritize tasks with their true keys, before stringifying (GH#2006) Matthew Rocklin
- Serialize worker exceptions through normal channels (GH#2016) Matthew Rocklin
- Include exception in progress bar (GH#2028) Matthew Rocklin
- Avoid logging orphaned futures in All (GH#2008) Matthew Rocklin
- Don’t use spill-to-disk dictionary if we’re not spilling to disk Matthew Rocklin
- Only avoid recomputation if key exists (GH#2036) Matthew Rocklin
- Use client connection and serialization arguments in progress (GH#2035) Matthew Rocklin
- Rejoin worker client on closing context manager (GH#2041) Matthew Rocklin
- Avoid forgetting erred tasks when losing dependencies (GH#2047) Matthew Rocklin
- Avoid collisions in graph_layout (GH#2050) Matthew Rocklin
- Avoid recursively calling bokeh callback in profile plot (GH#2048) Matthew Rocklin
1.21.8 - 2018-05-03¶
- Remove errant print statement (GH#1957) Matthew Rocklin
- Only add reevaluate_occupancy callback once (GH#1953) Tony Lorenzo
1.21.7 - 2018-05-02¶
- Newline needed for doctest rendering (GH#1917) Loïc Estève
- Support Client._repr_html_ when in async mode (GH#1909) Matthew Rocklin
- Add parameters to dask-ssh command (GH#1910) Irene Rodriguez
- Santize get_dataset trace (GH#1888) John Kirkham
- Fix bug where queues would not clean up cleanly (GH#1922) Matthew Rocklin
- Delete cached file safely in upload file (GH#1921) Matthew Rocklin
- Accept KeyError when closing tornado IOLoop in tests (GH#1937) Matthew Rocklin
- Quiet the client and scheduler when gather(…, errors=’skip’) (GH#1936) Matthew Rocklin
- Clarify couldn’t gather keys warning (GH#1942) Kenneth Koski
- Support submit keywords in joblib (GH#1947) Matthew Rocklin
- Avoid use of external resources in bokeh server (GH#1934) Matthew Rocklin
- Drop __contains__ from Datasets (GH#1889) John Kirkham
- Fix bug with queue timeouts (GH#1950) Matthew Rocklin
- Replace msgpack-python by msgpack (GH#1927) Loïc Estève
1.21.6 - 2018-04-06¶
- Fix numeric environment variable configuration (GH#1885) Joseph Atkins-Kurkish
- support bytearrays in older lz4 library (GH#1886) Matthew Rocklin
- Remove started timeout in nanny (GH#1852) Matthew Rocklin
- Don’t log errors in sync (GH#1894) Matthew Rocklin
- downgrade stale lock warning to info logging level (GH#1890) Matthew Rocklin
- Fix
UnboundLocalError
forkey
(GH#1900) John Kirkham - Resolve deployment issues in Python 2 (GH#1905) Matthew Rocklin
- Support retries and priority in Client.get method (GH#1902) Matthew Rocklin
- Add additional attributes to task page if applicable (GH#1901) Matthew Rocklin
- Add count method to as_completed (GH#1897) Matthew Rocklin
- Extend default timeout to 10s (GH#1904) Matthew Rocklin
1.21.5 - 2018-03-31¶
- Increase default allowable tick time to 3s (GH#1854) Matthew Rocklin
- Handle errant workers when another worker has data (GH#1853) Matthew Rocklin
- Close multiprocessing queue in Nanny to reduce open file descriptors (GH#1862) Matthew Rocklin
- Extend nanny started timeout to 30s, make configurable (GH#1865) Matthew Rocklin
- Comment out the default config file (GH#1871) Matthew Rocklin
- Update to fix bokeh 0.12.15 update errors (GH#1872) Matthew Rocklin
- Downgrade Event Loop unresponsive warning to INFO level (GH#1870) Matthew Rocklin
- Add fifo timeout to control priority generation (GH#1828) Matthew Rocklin
- Add retire_workers API to Client (GH#1876) Matthew Rocklin
- Catch NoSuchProcess error in Nanny.memory_monitor (GH#1877) Matthew Rocklin
- Add uid to nanny queue communitcations (GH#1880) Matthew Rocklin
1.21.4 - 2018-03-21¶
- Avoid passing bytearrays to snappy decompression (GH#1831) Matthew Rocklin
- Specify IOLoop in Adaptive (GH#1841) Matthew Rocklin
- Use connect-timeout config value throughout client (GH#1839) Matthew Rocklin
- Support direct= keyword argument in Client.get (GH#1845) Matthew Rocklin
1.21.3 - 2018-03-08¶
- Add cluster superclass and improve adaptivity (GH#1813) Matthew Rocklin
- Fixup tests and support Python 2 for Tornado 5.0 (GH#1818) Matthew Rocklin
- Fix bug in recreate_error when dependencies are dropped (GH#1815) Matthew Rocklin
- Add worker time to live in Scheduler (GH#1811) Matthew Rocklin
- Scale adaptive based on total_occupancy (GH#1807) Matthew Rocklin
- Support calling compute within worker_client (GH#1814) Matthew Rocklin
- Add percentage to profile plot (GH#1817) Brett Naul
- Overwrite option for remote python in dask-ssh (GH#1812) Sven Kreiss
1.21.2 - 2018-03-05¶
- Fix bug where we didn’t check idle/saturated when stealing (GH#1801) Matthew Rocklin
- Fix bug where client was noisy when scheduler closed unexpectedly (GH#1806) Matthew Rocklin
- Use string-based timedeltas (like
'500 ms'
) everywhere (GH#1804) Matthew Rocklin - Keep logs in scheduler and worker even if silenced (GH#1803) Matthew Rocklin
- Support minimum, maximum, wait_count keywords in Adaptive (GH#1797) Jacob Tomlinson and Matthew Rocklin
- Support async protocols for LocalCluster, replace start= with asynchronous= (GH#1798) Matthew Rocklin
- Avoid restarting workers when nanny waits on scheduler (GH#1793) Matthew Rocklin
- Use
IOStream.read_into()
when available (GH#1477) Antoine Pitrou - Reduce LocalCluster logging threshold from CRITICAL to WARN (GH#1785) Andy Jones
- Add futures_of to API docs (GH#1783) John Kirkham
- Make diagnostics link in client configurable (GH#1810) Matthew Rocklin
1.21.1 - 2018-02-22¶
- Fixed an uncaught exception in
distributed.joblib
with aLocalCluster
using only threads (GH#1775) Tom Augspurger - Format bytes in info worker page (GH#1752) Matthew Rocklin
- Add pass-through arguments for scheduler/worker –preload modules. (GH#1634) Alexander Ford
- Use new LZ4 API (GH#1757) Thrasibule
- Replace dask.optimize with dask.optimization (GH#1754) Matthew Rocklin
- Add graph layout engine and bokeh plot (GH#1756) Matthew Rocklin
- Only expand name with –nprocs if name exists (GH#1776) Matthew Rocklin
- specify IOLoop for stealing PeriodicCallback (GH#1777) Matthew Rocklin
- Fixed distributed.joblib with no processes Tom Augspurger
- Use set.discard to avoid KeyErrors in stealing (GH#1766) Matthew Rocklin
- Avoid KeyError when task has been released during steal (GH#1765) Matthew Rocklin
- Add versions routes to avoid the use of run in Client.get_versions (GH#1773) Matthew Rocklin
- Add write_scheduler_file to Client (GH#1778) Joe Hamman
- Default host to tls:// if tls information provided (GH#1780) Matthew Rocklin
1.21.0 - 2018-02-09¶
- Refactor scheduler to use TaskState objects rather than dictionaries (GH#1594) Antoine Pitrou
- Plot CPU fraction of total in workers page (GH#1624) Matthew Rocklin
- Use thread CPU time in Throttled GC (GH#1625) Antoine Pitrou
- Fix bug with
memory_limit=None
(GH#1639) Matthew Rocklin - Add futures_of to top level api (GH#1646) Matthew Rocklin
- Warn on serializing large data in Client (GH#1636) Matthew Rocklin
- Fix intermittent windows failure when removing lock file (GH#1652) Antoine Pitrou
- Add diagnosis and logging of poor GC Behavior (GH#1635) Antoine Pitrou
- Add client-scheduler heartbeats (GH#1657) Matthew Rocklin
- Return dictionary of worker info in
retire_workers
(GH#1659) Matthew Rocklin - Ensure dumps_function works with unhashable functions (GH#1662) Matthew Rocklin
- Collect client name ids rom client-name config variable (GH#1664) Matthew Rocklin
- Allow simultaneous use of –name and –nprocs in dask-worker (GH#1665) Matthew Rocklin
- Add support for grouped adaptive scaling and adaptive behavior overrides (GH#1632) Alexander Ford
- Share scheduler RPC between worker and client (GH#1673) Matthew Rocklin
- Allow
retries=
in ClientExecutor (GH#1672) @rqx - Improve documentation for get_client and dask.compute examples (GH#1638) Scott Sievert
- Support DASK_SCHEDULER_ADDRESS environment variable in worker (GH#1680) Matthew Rocklin
- Support tuple-keys in retries (GH#1681) Matthew Rocklin
- Use relative links in bokeh dashboard (GH#1682) Matthew Rocklin
- Make message log length configurable, default to zero (GH#1691) Matthew Rocklin
- Deprecate
Client.shutdown
(GH#1699) Matthew Rocklin - Add warning in configuration docs to install pyyaml (GH#1701) Cornelius Riemenschneider
- Handle nested parallelism in distributed.joblib (GH#1705) Tom Augspurger
- Don’t wait for Worker.executor to shutdown cleanly when restarting process (GH#1708) Matthew Rocklin
- Add support for user defined priorities (GH#1651) Matthew Rocklin
- Catch and log OSErrors around worker lock files (GH#1714) Matthew Rocklin
- Remove worker prioritization. Coincides with changes to dask.order (GH#1730) Matthew Rocklin
- Use process-measured memory rather than nbytes in Bokeh dashboard (GH#1737) Matthew Rocklin
- Enable serialization of Locks (GH#1738) Matthew Rocklin
- Support Tornado 5 beta (GH#1735) Matthew Rocklin
- Cleanup remote_magic client cache after tests (GH#1743) Min RK
- Allow service ports to be specified as (host, port) (GH#1744) Bruce Merry
1.20.2 - 2017-12-07¶
- Clear deque handlers after each test (GH#1586) Antoine Pitrou
- Handle deserialization in FutureState.set_error (GH#1592) Matthew Rocklin
- Add process leak checker to tests (GH#1596) Antoine Pitrou
- Customize process title for subprocess (GH#1590) Antoine Pitrou
- Make linting a separate CI job (GH#1599) Antoine Pitrou
- Fix error from get_client() with no global client (GH#1595) Daniel Li
- Remove Worker.host_health, correct WorkerTable metrics (GH#1600) Matthew Rocklin
- Don’t mark tasks as suspicious when retire_workers called. Addresses (GH#1607) Russ Bubley
- Do not include processing workers in workers_to_close (GH#1609) Russ Bubley
- Disallow simultaneous scale up and down in Adaptive (GH#1608) Russ Bubley
- Parse bytestrings in –memory-limit (GH#1615) Matthew Rocklin
- Use environment variable for scheduler address if present (GH#1610) Matthew Rocklin
- Fix deprecation warning from logger.warn (GH#1616) Brett Naul
1.20.1 - 2017-11-26¶
- Wrap
import ssl
statements with try-except block for ssl-crippled environments, (GH#1570) Xander Johnson - Support zero memory-limit in Nanny (GH#1571) Matthew Rocklin
- Avoid PeriodicCallback double starts (GH#1573) Matthew Rocklin
- Add disposable workspace facility (GH#1543) Antoine Pitrou
- Use format_time in task_stream plots (GH#1575) Matthew Rocklin
- Avoid delayed finalize calls in compute (GH#1577) Matthew Rocklin
- Doc fix about secede (GH#1583) Scott Sievert
- Add tracemalloc option when tracking test leaks (GH#1585) Antoine Pitrou
- Add JSON routes to Bokeh server (GH#1584) Matthew Rocklin
- Handle exceptions cleanly in Variables and Queues (GH#1580) Matthew Rocklin
1.20.0 - 2017-11-17¶
- Drop use of pandas.msgpack (GH#1473) Matthew Rocklin
- Add methods to get/set scheduler metadata Matthew Rocklin
- Add distributed lock Matthew Rocklin
- Add reschedule exception for worker tasks Matthew Rocklin
- Fix
nbytes()
forbytearrays
Matthew Rocklin - Capture scheduler and worker logs Matthew Rocklin
- Garbage collect after data eviction on high worker memory usage (GH#1488) Olivier Grisel
- Add scheduler HTML routes to bokeh server (GH#1478) (GH#1514) Matthew Rocklin
- Add pytest plugin to test for resource leaks (GH#1499) Antoine Pitrou
- Improve documentation for scheduler states (GH#1498) Antoine Pitrou
- Correct warn_if_longer timeout in ThrottledGC (GH#1496) Fabian Keller
- Catch race condition in as_completed on cancelled futures (GH#1507) Matthew Rocklin
- Transactional work stealing (GH#1489) (GH#1528) Matthew Rocklin
- Avoid forkserver in PyPy (GH#1509) Matthew Rocklin
- Add dict access to get/set datasets (GH#1508) Mike DePalatis
- Support Tornado 5 (GH#1509) (GH#1512) (GH#1518) (GH#1534) Antoine Pitrou
- Move thread_state in Dask (GH#1523) Jim Crist
- Use new Dask collections interface (GH#1513) Matthew Rocklin
- Add nanny flag to dask-mpi Matthew Rocklin
- Remove JSON-based HTTP servers Matthew Rocklin
- Avoid doing I/O in repr/str (GH#1536) Matthew Rocklin
- Fix URL for MPI4Py project (GH#1546) Ian Hopkinson
- Allow automatic retries of a failed task (GH#1524) Antoine Pitrou
- Clean and accelerate tests (GH#1548) (GH#1549) (GH#1552) (GH#1553) (GH#1560) (GH#1564) Antoine Pitrou
- Move HDFS functionality to the hdfs3 library (GH#1561) Jim Crist
- Fix bug when using events page with no events (GH#1562) @rbubley
- Improve diagnostic naming of tasks within tuples (GH#1566) Kelvyn Yang
1.19.3 - 2017-10-16¶
1.19.2 - 2017-10-06¶
- as_completed doesn’t block on cancelled futures (GH#1436)
- Notify waiting threads/coroutines on cancellation (GH#1438)
- Set Future(inform=True) as default (GH#1437)
- Rename Scheduler.transition_story to story (GH#1445)
- Future uses default client by default (GH#1449)
- Add keys= keyword to Client.call_stack (GH#1446)
- Add get_current_task to worker (GH#1444)
- Ensure that Client remains asynchornous before ioloop starts (GH#1452)
- Remove “click for worker page” in bokeh plot (GH#1453)
- Add Client.current() (GH#1450)
- Clean handling of restart timeouts (GH#1442)
1.19.1 - September 25th, 2017¶
1.19.0 - September 24th, 2017¶
- Avoid storing messages in message log (GH#1361)
- fileConfig does not disable existing loggers (GH#1380)
- Offload upload_file disk I/O to separate thread (GH#1383)
- Add missing SSLContext (GH#1385)
- Collect worker thread information from sys._curent_frames (GH#1387)
- Add nanny timeout (GH#1395)
- Restart worker if memory use goes above 95% (GH#1397)
- Track workers memory use with psutil (GH#1398)
- Track scheduler delay times in workers (GH#1400)
- Add time slider to profile plot (GH#1403)
- Change memory-limit keyword to refer to maximum number of bytes (GH#1405)
- Add
cancel(force=)
keyword (GH#1408)
1.18.2 - September 2nd, 2017¶
1.18.1 - August 25th, 2017¶
- Clean up forgotten keys in fire-and-forget workloads (GH#1250)
- Handle missing extensions (GH#1263)
- Allow recreate_exception on persisted collections (GH#1253)
- Add asynchronous= keyword to blocking client methods (GH#1272)
- Restrict to horizontal panning in bokeh plots (GH#1274)
- Rename client.shutdown to client.close (GH#1275)
- Avoid blocking on event loop (GH#1270)
- Avoid cloudpickle errors for Client.get_versions (GH#1279)
- Yield on Tornado IOStream.write futures (GH#1289)
- Assume async behavior if inside a sync statement (GH#1284)
- Avoid error messages on closing (GH#1297), (GH#1296) (GH#1318) (GH#1319)
- Add timeout= keyword to get_client (GH#1290)
- Respect timeouts when restarting (GH#1304)
- Clean file descriptor and memory leaks in tests (GH#1317)
- Deprecate Executor (GH#1302)
- Add timeout to ThreadPoolExecutor.shutdown (GH#1330)
- Clean up AsyncProcess handling (GH#1324)
- Allow unicode keys in Python 2 scheduler (GH#1328)
- Avoid leaking stolen data (GH#1326)
- Improve error handling on failed nanny starts (GH#1337), (GH#1331)
- Make Adaptive more flexible
- Support
--contact-address
and--listen-address
in worker (GH#1278) - Remove old dworker, dscheduler executables (GH#1355)
- Exit workers if nanny process fails (GH#1345)
- Auto pep8 and flake (GH#1353)
1.18.0 - July 8th, 2017¶
- Multi-threading safety (GH#1191), (GH#1228), (GH#1229)
- Improve handling of byte counting (GH#1198) (GH#1224)
- Add get_client, secede functions, refactor worker-client relationship (GH#1201)
- Allow logging configuraiton using logging.dictConfig() (GH#1206) (GH#1211)
- Offload serialization and deserialization to separate thread (GH#1218)
- Support fire-and-forget tasks (GH#1221)
- Support bytestrings as keys (for Julia) (GH#1234)
- Resolve testing corner-cases (GH#1236), (GH#1237), (GH#1240), (GH#1241), (GH#1242), (GH#1244)
- Automatic use of scatter/gather(direct=True) in more cases (GH#1239)
1.17.1 - June 14th, 2017¶
1.17.0 - June 9th, 2017¶
- Reevaluate worker occupancy periodically during scheduler downtime (GH#1038) (GH#1101)
- Add
AioClient
asyncio-compatible client API (GH#1029) (GH#1092) (GH#1099) - Update Keras serializer (GH#1067)
- Support TLS/SSL connections for security (GH#866) (GH#1034)
- Always create new worker directory when passed
--local-directory
(GH#1079) - Support pre-scattering data when using joblib frontent (GH#1022)
- Make workers more robust to failure of
sizeof
function (GH#1108) and writing to disk (GH#1096) - Add
is_empty
andupdate
methods toas_completed
(GH#1113) - Remove
_get
coroutine and replace withget(..., sync=False)
(GH#1109) - Improve API compatibility with async/await syntax (GH#1115) (GH#1124)
- Add distributed Queues (GH#1117) and shared Variables (GH#1128) to enable inter-client coordination
- Support direct client-to-worker scattering and gathering (GH#1130) as well as performance enhancements when scattering data
- Style improvements for bokeh web dashboards (GH#1126) (GH#1141) as well as a removal of the external bokeh process
- HTML reprs for Future and Client objects (GH#1136)
- Support nested collections in client.compute (GH#1144)
- Use normal client API in asynchronous mode (GH#1152)
- Remove old distributed.collections submodule (GH#1153)
1.16.3 - May 5th, 2017¶
1.16.2 - May 3rd, 2017¶
- Support
async with Client
syntax (GH#1053) - Use internal bokeh server for default diagnostics server (GH#1047)
- Improve styling of bokeh plots when empty (GH#1046) (GH#1037)
- Support efficient serialization for sparse arrays (GH#1040)
- Prioritize newly arrived work in worker (GH#1035)
- Prescatter data with joblib backend (GH#1022)
- Make client.restart more robust to worker failure (GH#1018)
- Support preloading a module or script in dask-worker or dask-scheduler processes (GH#1016)
- Specify network interface in command line interface (GH#1007)
- Client.scatter supports a single element (GH#1003)
- Use blosc compression on all memoryviews passing through comms (GH#998)
- Add concurrent.futures-compatible Executor (GH#997)
- Add as_completed.batches method and return results (GH#994) (GH#971)
- Allow worker_clients to optionally stay within the thread pool (GH#993)
- Add bytes-stored and tasks-processing diagnostic histograms (GH#990)
- Run supports non-msgpack-serializable results (GH#965)
1.16.1 - March 22nd, 2017¶
- Use inproc transport in LocalCluster (GH#919)
- Add structured and queryable cluster event logs (GH#922)
- Use connection pool for inter-worker communication (GH#935)
- Robustly shut down spawned worker processes at shutdown (GH#928)
- Worker death timeout (GH#940)
- More visual reporting of exceptions in progressbar (GH#941)
- Render disk and serialization events to task stream visual (GH#943)
- Support async for / await protocol (GH#952)
- Ensure random generators are re-seeded in worker processes (GH#953)
- Upload sourcecode as zip module (GH#886)
- Replay remote exceptions in local process (GH#894)
1.16.0 - February 24th, 2017¶
- First come first served priorities on client submissions (GH#840)
- Can specify Bokeh internal ports (GH#850)
- Allow stolen tasks to return from either worker (GH#853), (GH#875)
- Add worker resource constraints during execution (GH#857)
- Send small data through Channels (GH#858)
- Better estimates for SciPy sparse matrix memory costs (GH#863)
- Avoid stealing long running tasks (GH#873)
- Maintain fortran ordering of NumPy arrays (GH#876)
- Add
--scheduler-file
keyword to dask-scheduler (GH#877) - Add serializer for Keras models (GH#878)
- Support uploading modules from zip files (GH#886)
- Improve titles of Bokeh dashboards (GH#895)
1.15.2 - January 27th, 2017¶
- Fix a bug where arrays with large dtypes or shapes were being improperly compressed (GH#830 GH#832 GH#833)
- Extend
as_completed
to accept new futures during iteration (GH#829) - Add
--nohost
keyword todask-ssh
startup utility (GH#827) - Support scheduler shutdown of remote workers, useful for adaptive clusters (:pr: 811 GH#816 GH#821)
- Add
Client.run_on_scheduler
method for running debug functions on the scheduler (GH#808)
1.15.1 - January 11th, 2017¶
- Make compatibile with Bokeh 0.12.4 (GH#803)
- Avoid compressing arrays if not helpful (GH#777)
- Optimize inter-worker data transfer (GH#770) (GH#790)
- Add –local-directory keyword to worker (GH#788)
- Enable workers to arrive to the cluster with their own data. Useful if a worker leaves and comes back (GH#785)
- Resolve thread safety bug when using local_client (GH#802)
- Resolve scheduling issues in worker (GH#804)
1.15.0 - January 2nd, 2017¶
- Major Worker refactor (GH#704)
- Major Scheduler refactor (GH#717) (GH#722) (GH#724) (GH#742) (GH#743
- Add
check
(default isFalse
) option toClient.get_versions
to raise if the versions don’t match on client, scheduler & workers (GH#664) Future.add_done_callback
executes in separate thread (GH#656)- Clean up numpy serialization (GH#670)
- Support serialization of Tornado v4.5 coroutines (GH#673)
- Use CPickle instead of Pickle in Python 2 (GH#684)
- Use Forkserver rather than Fork on Unix in Python 3 (GH#687)
- Support abstract resources for per-task constraints (GH#694) (GH#720) (GH#737)
- Add TCP timeouts (GH#697)
- Add embedded Bokeh server to workers (GH#709) (GH#713) (GH#738)
- Add embedded Bokeh server to scheduler (GH#724) (GH#736) (GH#738)
- Add more precise timers for Windows (GH#713)
- Add Versioneer (GH#715)
- Support inter-client channels (GH#729) (GH#749)
- Scheduler Performance improvements (GH#740) (GH#760)
- Improve load balancing and work stealing (GH#747) (GH#754) (GH#757)
- Run Tornado coroutines on workers
- Avoid slow sizeof call on Pandas dataframes (GH#758)
1.14.3 - November 13th, 2016¶
1.14.2 - November 11th, 2016¶
1.14.0 - November 3rd, 2016¶
- Add
Client.get_versions()
function to return software and package information from the scheduler, workers, and client (GH#595) - Improved windows support (GH#577) (GH#590) (GH#583) (GH#597)
- Clean up rpc objects explicitly (GH#584)
- Normalize collections against known futures (GH#587)
- Add key= keyword to map to specify keynames (GH#589)
- Custom data serialization (GH#606)
- Refactor the web interface (GH#608) (GH#615) (GH#621)
- Allow user-supplied Executor in Worker (GH#609)
- Pass Worker kwargs through LocalCluster
1.13.3 - October 15th, 2016¶
- Schedulers can retire workers cleanly
- Add
Future.add_done_callback
forconcurrent.futures
compatibility - Update web interface to be consistent with Bokeh 0.12.3
- Close streams explicitly, avoiding race conditions and supporting more robust restarts on Windows.
- Improved shuffled performance for dask.dataframe
- Add adaptive allocation cluster manager
- Reduce administrative overhead when dealing with many workers
dask-ssh --log-directory .
no longer errors- Microperformance tuning for the scheduler
1.13.2¶
- Revert dask_worker to use fork rather than subprocess by default
- Scatter retains type information
- Bokeh always uses subprocess rather than spawn
1.13.1¶
- Fix critical Windows error with dask_worker executable
1.13.0¶
- Rename Executor to Client (GH#492)
- Add
--memory-limit
option todask-worker
, enabling spill-to-disk behavior when running out of memory (GH#485) - Add
--pid-file
option to dask-worker and--dask-scheduler
(GH#496) - Add
upload_environment
function to distribute conda environments. This is experimental, undocumented, and may change without notice. (GH#494) - Add
workers=
keyword argument toClient.compute
andClient.persist
, supporting location-restricted workloads with Dask collections (GH#484) - Add
upload_environment
function to distribute conda environments. This is experimental, undocumented, and may change without notice. (GH#494)- Add optional
dask_worker=
keyword toclient.run
functions that gets provided the worker or nanny object - Add
nanny=False
keyword toClient.run
, allowing for the execution of arbitrary functions on the nannies as well as normal workers
- Add optional
1.12.2¶
This release adds some new features and removes dead code
- Publish and share datasets on the scheduler between many clients (GH#453). See Publish Datasets.
- Launch tasks from other tasks (experimental) (GH#471). See Launch Tasks from Tasks.
- Remove unused code, notably the
Center
object and older client functions (GH#478) Executor()
andLocalCluster()
is now robust to Bokeh’s absence (GH#481)- Removed s3fs and boto3 from requirements. These have moved to Dask.
1.12.1¶
This release is largely a bugfix release, recovering from the previous large refactor.
- Fixes from previous refactor
- Ensure idempotence across clients
- Stress test losing scattered data permanently
- IPython fixes
- Add
start_ipython_scheduler
method to Executor - Add
%remote
magic for workers - Clean up code and tests
- Add
- Pool connects to maintain reuse and reduce number of open file handles
- Re-implement work stealing algorithm
- Support cancellation of tuple keys, such as occur in dask.arrays
- Start synchronizing against worker data that may be superfluous
- Improve bokeh plots styling
- Add memory plot tracking number of bytes
- Make the progress bars more compact and align colors
- Add workers/ page with workers table, stacks/processing plot, and memory
- Add this release notes document
1.12.0¶
This release was largely a refactoring release. Internals were changed significantly without many new features.
- Major refactor of the scheduler to use transitions system
- Tweak protocol to traverse down complex messages in search of large bytestrings
- Add dask-submit and dask-remote
- Refactor HDFS writing to align with changes in the dask library
- Executor reconnects to scheduler on broken connection or failed scheduler
- Support sklearn.external.joblib as well as normal joblib