Add support for starting plots via the Chia daemon plotter API

[djkaty]

This series of commits enables the plot manager to create plots by calling the Chia daemon (chia run_daemon) plotter API rather than calling chia plots create directly.

Why?

The Chia daemon plotter API allows clients to subscribe to receive periodic plotting progress updates. Existing clients include the Chia GUI Plot pane and Chia-Dashboard. The latter allows Chia farms to monitor plotting progress via the web or mobile device. Swar-Plot-Manager as it exists today does not provide for this possibility, since calling chia directly to create plots does not register their existence with the Chia daemon.

How?

A boolean option use_daemon is added to the manager section of config.yaml. This is set to false by default since setting it to true requires the daemon to be running in the background on a plotting rig. This might not be the case if the rig is not also used as a node or for farming, and I believe that starting plots in this way counts as an advanced usage scenario.

If use_daemon is set to false, the plot manager behaviour is unchanged from the current behaviour.

If use_daemon is set to true, the plotter API will be called to create plots instead of starting chia plots create directly. The plotter API is currently undocumented on the RPC Interfaces wiki page; refer instead to the chia daemon server.py source code file for implementation details.

Implementation details

The Chia daemon creates plots without spacing between the argument switches and their values, eg. it creates arguments such as -k32 -u128 -tFoo. The first commit enables the plot manager to discern these arguments correctly.

The second commit implements the plotter API client. The official source code's daemon proxy class omits wrappers to start_plotting and some related RPCs; I extend the class in this commit to include this. This is a workaround and ultimately, this is something the Chia team should address themselves.

The third commit enables the plot manager's job names to be shown as queue names in the Chia GUI.

Commits 4, 5 and 7 are bug fixes.

Commit 6 ensures the new plot process is nice'd in line with the existing behaviour.

Notes and known issues

First of all, my Python skills are complete garbage so I don't expect this to be approved as is; I don't have much time to work on it unfortunately so please feel free to butcher it and implement your own solution using this as a proof-of-concept :)

The changes add a dependency to chia-blockchain.

The Chia daemon must be running for this to work. The Chia blockchain config.yaml is expected to be in the default root path (~/.chia/mainnet/config/config.yaml - the exact path is fetched from chia.util.default_root.DEFAULT_ROOT_PATH). The hostname, port and SSL certificate for the connection are therefore fetched from the Chia configuration directly and not stored in the plot manager's config.yaml. It may be desirable to allow a custom configuration path in the future. This also means that in theory the plot manager can now create plots on remote machines although there is no point to this right now since we can't query remote processes or log files.

The plot manager will currently hang forever waiting for the daemon to be reachable. We may wish to add a timeout in the future (in which case creating a new plot should fail gracefully, or perhaps fall back to creating a process directly).

The farmer key, pool key and log folder path for plots will be ignored because there is no way to pass this information to the plotter API. The default farmer key, pool key and log folder path will be used. I have modified the plot manager so that it will still find the correct log files for progress tracking.

The plotter API does not return the PID of the created process. We find the process by periodically scanning for new chia processes after the call to the API returns. This can take some seconds. In the event that no processes or more than one process is created in this time window of a few seconds for some reason, the plot manager will hang indefinitely.

I believe those are all the known caveats. I have tested this for several days before submitting the PR and all seems to be working well now. We can see our plot progress via chia-dashboard on our cellphones from plots created by the plot manager, which is extremely useful for my team :)

[swar]

Just a few quick comments. This needs to be formatted for the development branch instead of main. Additionally one quick adaption for this PR will be the need to use four spaces instead of tab to adhere to the conformity with the rest of the library.

Will give a full review when I get a chance.