Skip to content

Commit a8d57a2

Browse files
committed
doc updates; bump to 0.9.2
1 parent b8427e9 commit a8d57a2

4 files changed

Lines changed: 31 additions & 27 deletions

File tree

DESCRIPTION

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
Package: clustermq
22
Title: Evaluate Function Calls on HPC Schedulers (LSF, SGE, SLURM, PBS/Torque)
3-
Version: 0.9.1
3+
Version: 0.9.2
44
Authors@R: c(
55
person('Michael', 'Schubert', email='mschu.dev@gmail.com',
66
role = c('aut', 'cre', 'cph'),

NEWS.md

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,9 @@
1-
# git head
1+
# clustermq 0.9.2
22

33
* Fix a bug where SSH proxy would not cache data properly (#320)
44
* Fix a bug where `max_calls_worker` was not respected (#322)
55
* Local parallelism (`multicore`, `multiprocess`) again uses local IP (#321)
6+
* Pool `info()` now also returns current worker and number of calls
67

78
# clustermq 0.9.1
89

README.md

Lines changed: 10 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -146,14 +146,23 @@ Use [`batchtools`](https://github.com/mllg/batchtools) if you:
146146
* don't mind there's no load-balancing at run-time
147147

148148
Use [Snakemake](https://snakemake.readthedocs.io/en/latest/) or
149-
[`drake`](https://github.com/ropensci/drake) if:
149+
[`targets`](https://github.com/ropensci/targets) if:
150150

151151
* you want to design and run a workflow on HPC
152152

153153
Don't use [`batch`](https://cran.r-project.org/web/packages/batch/index.html)
154154
(last updated 2013) or [`BatchJobs`](https://github.com/tudo-r/BatchJobs)
155155
(issues with SQLite on network-mounted storage).
156156

157+
Questions
158+
---------
159+
160+
You are welcome to ask questions if something is not clear in the [User
161+
guide](https://mschubert.github.io/clustermq/articles/userguide.html).
162+
163+
Please use the [Github
164+
Discussions](https://github.com/mschubert/clustermq/discussions) for this.
165+
157166
Contributing
158167
------------
159168

@@ -162,8 +171,6 @@ to coordinate development of `clustermq`. Contributions are welcome and they
162171
come in many different forms, shapes, and sizes. These include, but are not
163172
limited to:
164173

165-
* Questions: You are welcome to ask questions if something is not clear in the
166-
[User guide](https://mschubert.github.io/clustermq/articles/userguide.html).
167174
* Bug reports: Let us know if something does not work as expected. Be sure to
168175
include a self-contained [Minimal Reproducible
169176
Example](https://stackoverflow.com/help/minimal-reproducible-example) and set

vignettes/userguide.Rmd

Lines changed: 18 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -35,7 +35,7 @@ Install the `clustermq` package in R from CRAN. This will automatically detect
3535
if [ZeroMQ](https://github.com/zeromq/libzmq) is installed and otherwise use
3636
the bundled library:
3737

38-
```r
38+
```{r eval=FALSE}
3939
# Recommended:
4040
# If your system has `libzmq` installed but you want to enable the worker crash
4141
# monitor, set the following environment variable to enable compilation of the
@@ -47,7 +47,8 @@ install.packages('clustermq')
4747
Alternatively you can use the `remotes` package to install directly from
4848
Github. Note that this version needs `autoconf`/`automake` for compilation:
4949

50-
```r
50+
```{r eval=FALSE}
51+
# Sys.setenv(CLUSTERMQ_USE_SYSTEM_LIBZMQ=0)
5152
# install.packages('remotes')
5253
remotes::install_github('mschubert/clustermq')
5354
```
@@ -59,6 +60,7 @@ However, [feedback is very
5960
welcome](https://github.com/mschubert/clustermq/issues/new).
6061

6162
```{r eval=FALSE}
63+
# Sys.setenv(CLUSTERMQ_USE_SYSTEM_LIBZMQ=0)
6264
# install.packages('remotes')
6365
remotes::install_github('mschubert/clustermq', ref="develop")
6466
```
@@ -69,6 +71,7 @@ Choose your preferred parallelism using:
6971

7072
```{r eval=FALSE}
7173
options(clustermq.scheduler = "your scheduler here")
74+
# this may require additional setup, for details see below
7275
```
7376

7477
There are three kinds of schedulers:
@@ -106,7 +109,7 @@ To set up a scheduler explicitly, see the following links:
106109
* [SGE](#SGE) - *should work without setup*
107110
* [SLURM](#SLURM) - *should work without setup*
108111
* [PBS](#PBS)/[Torque](#TORQUE) - *needs* `options(clustermq.scheduler="PBS"/"Torque")`
109-
* if you want another scheduler, [open an
112+
* you can suggest another scheduler by [opening an
110113
issue](https://github.com/mschubert/clustermq/issues/new)
111114

112115
Default submission templates [are
@@ -212,7 +215,7 @@ Q(fx, x=1:3, export=list(y=10), n_jobs=1)
212215
```
213216

214217
If we want to use a package function we need to load it on the worker using the
215-
`pkg` argument or referencing it with `package_name::`:
218+
`pkg` argument or referencing it with `package_name::`.
216219

217220
```{r}
218221
fx = function(x) {
@@ -259,29 +262,22 @@ register(DoparParam()) # after register_dopar_cmq(...)
259262
bplapply(1:3, sqrt)
260263
```
261264

262-
### With `drake`
265+
### With `targets`
263266

264-
The [`drake`](https://github.com/ropensci/drake) package enables users to
267+
The [`targets`](https://github.com/ropensci/targets) package enables users to
265268
define a dependency structure of different function calls, and only evaluate
266269
them if the underlying data changed.
267270

268-
> drake — or, Data Frames in R for Make — is a general-purpose workflow manager
269-
> for data-driven tasks. It rebuilds intermediate data objects when their
270-
> dependencies change, and it skips work when the results are already up to
271-
> date. Not every runthrough starts from scratch, and completed workflows have
272-
> tangible evidence of reproducibility. drake also supports scalability,
273-
> parallel computing, and a smooth user experience when it comes to setting up,
274-
> deploying, and maintaining data science projects.
271+
> The `targets` package is a [Make](https://www.gnu.org/software/make/)-like
272+
> pipeline tool for statistics and data science in R. The package skips costly
273+
> runtime for tasks that are already up to date, orchestrates the necessary
274+
> computation with implicit parallel computing, and abstracts files as R
275+
> objects. If all the current output matches the current upstream code and
276+
> data, then the whole pipeline is up to date, and the results are more
277+
> trustworthy than otherwise.
275278
276-
It can use `clustermq` to perform calculations as jobs:
277-
278-
```{r eval=FALSE}
279-
library(drake)
280-
load_mtcars_example()
281-
# clean(destroy = TRUE)
282-
# options(clustermq.scheduler = "multicore")
283-
make(my_plan, parallelism = "clustermq", jobs = 2, verbose = 4)
284-
```
279+
It can use `clustermq` to [perform calculations as
280+
jobs](https://books.ropensci.org/targets/hpc.html#clustermq).
285281

286282
## Options
287283

0 commit comments

Comments
 (0)