Making `Cluster`'s `scale_up`/`scale_down` methods coroutines

[jakirkham]

In Distributed's Adaptive class, there are calls to Cluster implementations of scale_up and scale_down from within coroutines. While it would be possible to call these as coroutines in some cases (as is the case in dask-drmaa), this appears to not be part of Adaptive's spec for its cluster instance nor is part of the Cluster's spec. A cursory glance at dask-jobqueue suggests it doesn't use coroutines either. Considering it would be possible to make scale_up and scale_down coroutines and these would fit nicely within the Adaptive model, could we consider migrating in that direction?

[mrocklin]

Currently they can be coroutines or not coroutines, which I like

distributed/distributed/deploy/adaptive.py

Lines 235 to 237 in 34f7cd5

	f = self.cluster.scale_down(workers)
	if gen.is_future(f):
	yield f

[jakirkham]

For context, was attempting to remove a reimplemented Adaptive._retire_workers method from dask-drmaa, but found that we lost logging info about workers being retired. Based on some investigation, it seems to have something to do with scale_down being a coroutine. Though haven't narrowed down a fix as of yet. Hence the question.

ref: dask/dask-drmaa#85 (comment)

[mrocklin]

Closing this for now. Feel free to reopen if the question persists.