Change the default simply in mpi.parReplicate to “array”.
Fix "warning: format string is not a string literal" in internal.c (remove corresponding codes)
Fix "a function declaration without a prototype is deprecated
in all versions of C".
Fix R man files with Escaped
LaTeX specials: \_
Fix class(x) == "string" type
logic. Use like is.integer(x) instead.
Fix autoconf obsolete bug by using
autoupdate;
autoreconf --warnings=obsolete
suggested by Kurt Hornik. Thanks.
Prof. Ripley fixes a bug related to Mac compiling. Thanks.
In configure.ac, checking for openpty and pthread are removed,
suggested by Prof. Ripley <ripley at stats.ox.ac.uk>. If
necessary, please uncomment
## AC_CHECK_LIB(util, openpty, [
MPI_LIBS="$MPI_LIBS -lutil" ])
##
AC_CHECK_LIB(pthread, main, [ MPI_LIBS="$MPI_LIBS -lpthread"
])
in configure.ac and run autoconf to generate a new configure
file.
Add #include <R.h>, #include <R_ext/Rdynload.h> in internal.c so that Rmpi can be compiled for older versions of R (older than R 3.4.4).
Fix two bugs in R_CallMethodDef callMethods.
URL in DESCRIPTION is changed to http://fisher.stats.uwo.ca/faculty/yu/Rmpi
Found no calls to: ‘R_registerRoutines’, ‘R_useDynamicSymbols’ NOTE is fixed. Many native routines are registered in Rmpi.h.
Add pkg-config detection of OpenMPI in configure.ac, suggested by Prof. Kurt Hornik.
Thanks to Prof Brian Ripley for correcting CRLF line endings in shell scripts configure and cleanup.
The repeat loop in slavedaemon.R is modified so that a specific break is used. Originally the break is sent over by mpi.bcast.cmd and R CMD check will give a warning of using such a break without loop. Also invisible(mpi.comm.disconnect(.comm)) in slavedaemon.R is disabled. It has been reported by many to cause hang when mpi.close.Rslaves() is used. Correspondingly, mpi.close.Rslaves will use mpi.comm.free to free the associated comm.
It seems that there are many issues with OpenMPI version 2 when compiling Rmpi. With OpenMPI version 3, Rmpi can be compiled and run without any issues. So please compile and use Rmpi with OpenMPI version 1 or 3.
Now Rmpi can be run in Rstudio for Windows. Please see updated instructions for Windows.
Warning message in mpi.finalize as well as in Rprofile is removed.
Makevars.win and msmpi.def are updated in src so Rmpi can be installed and run under Microsoft MPI ver 7.
Fix some namespace issues.
To use Rmpi under OpenMPI 1.10.2, if there is loading error, please configure OpenMPI with ./configure --disable-dlopen.
All /dontrun is replaced by /donttest in man pages, suggested by Prof. Brian Ripley.
To close any running slaves left unintentionally by users, the
follow codes
if (mpi.comm.size(1)>0)
mpi.close.Rslaves()
are added to
.onUnload.
Fix a bug in mpi.bcast.Robj2slave and
mpi.bcast.data2slave
libmpi.so.1 is loaded first with dlopen, suggested by Don Armstrong.
Due to discontinuation of MPICH2 for Windows, support of Rmpi under Windows's MPICH2 is dropped. Thanks to Prof. Brian Ripley, Rmpi can be compiled under Microsoft MPI. However, mpi.spawn.Rslaves() is disabled since spawning is not implemented in Microsoft MPI.
MPI_Errhandler_set is replaced by MPI_Comm_set_errhandler due to its deprecation.
A MPI type INTELMPI has been added to configure.ac. This allows
Rmpi to be compiled with Intel's MPI (variation of MPICH). This
modification was supplied by "Boyan Bejanov" <bbejanov
at bank banque canada ca>.
All parallel apply functions
with the arguments x and fun are replaced with X and FUN. This is in
line with R's apply functions.
It has been observed that (un)serialization doesn't release memory properly after use. So the garbage collection (gc()) is used when there is a (un)serialization as along as it is not nonblocking call related.
A new function mpi.bcast.data2slave is added. It transfers data (a double vector or a matrix) natively without (un)serilization. It should be used with a huge vector or matrix. It results less memory usage and faster transmission. Notice that data with missing values (NA) are allowed.
A MPI type CRAY has been added to configure.ac. This allows Rmpi to be compiled on a Cray cluster with Cray's own MPICH2. It should be compiled as
R CMD INSTALL Rmpi_0.6-4.tar.gz --configure-args=" \\ --with-Rmpi-include=/path_to_cray_mpich2_include \\ --with-Rmpi-libpath=/path_to_cray_mpich2_lib \\ --with-Rmpi-type=CRAY" --no-test-load
Please include --no-test-load, otherwise the installation will fail. Use Cray's aprun to launch Rmpi with Rprofile, either interactively or through PBS.
On Mac OS X, there is an error related to loading libmpi.dylib during MPI_Init, an issue found by "Jan de Leeuw" < deleeuw at stat ucla edu >. Under open-mpi 1.6.4, loading libmpi.dylib is not required and hence is removed. The same issue was detected by CRAN, as reported by Prof. Brian Ripley.
The dependence of the package rlecuyer for parallel RNG is removed. Instead the package parallel is used. Accordingly, the function mpi.setup.rngstream is simplified with iseed as input.
Due to difficulty to assign global objects .mpi.applyLB and .mpi.parSim, the option apply.seq in mpi.(i)applyLB related functions is disabled. So is the option sim.seq in mpi.parSim.
Add an option ... to mpi.bcast.cmd. This allows an execution like myfun(x,y) on all slaves if mpi.bcast.cmd(myfun, x=x,y=y) is used, where x and y are objects on the master.
mpi.remote.exec has been modified so that mpi.remote.exec(x<-1:10) will save the object x on all slaves.
Add a MPI API mpi.comm.c2f which converts an Rmpi's communicator into an integer that can be used as the communicator in external FORTRAN code, suggested by "Chris Paciorek" <paciorek at stat.berkeley.edu> and "Benjamin Lipshitz" <lipshitz at berkely.edu>.
Add an option all=FALSE to mpi.bcat.Robj2slave. If all=TRUE, all objects created on master will be transmitted to all slaves. Create a new function mpi.bcast.Rfun2slave which transfers all functions created by master to all slaves.
Core count for Mac OS X is rewritten so that mpi.universe.size will correctly report total number of cores on a single Mac machine. Due to R64 naming on Mac, a script MacR64slaves.sh is created to aid R64 to choose a right slave.
Under Open MPI, add a program check on orted in configure.ac since Rmpi cannot be loaded without orted , an issue found by "Dirk Eddelbuettel" <edd at debian.org>.
Modify mpi.scatter.Robj so that subobj <- lapply(obj,serialize, connection=NULL) is replaced by subobj<-lapply(1:size, function(i) serialize(obj[[i]], NULL)). This results substantial improvement of encode speed. The idea of this modification comes from pdbMPI package.
mpi.bcast.cmd has been rewrittten with two additional options nonblock and sleep. It uses for loop with mpi.send and mpi.recv pair to broadcast a command. The use of mpi.bcast is dropped since it cannot support nonblock calls. If nonblock=TRUE, the receivers will run an infinite repeat loop with nonblock call mpi.iprobe. If there is no message, it will sleep and try again. Accordingly, mpi.spawn.Rslaves has two more options nonblock and sleep. If nonblock=TRUE (default), mpi.spawn.Rslaves() will create slaves with no CPU consumption while waiting. The default value sleep=0.1 (sec) seems using least CPU but still responding to a command quickly. To make slaves more responding, change sleep to a smaller number.
It seems loading libmpi.so with error checking fails Rmpi for Mac. This is because Mac OS doesn't use .so naming. Loading libmpi.dylib has been added. Grab a binary one here. It is compiled under Lion (64bit only)..
Detection of Mac OS is added to mpi.universe.size in case that Rmpi is running under Open MPI and the default value is 1. In this case, the total number of cores is returned instead of 1. This will be useful when spawning slaves on a single Mac. A machine with hyperthreaded cores is not tested though.
Functions bin.nchar, .mpi.serialize and .mpi.unserialize are deprecated.
.First.Lib replaced by .onLoad, .Last.Lib by .onUnload. Fix a unloading warning message for Mac. It looks like that the option --disable-dlopen is not necessary to install Open MPI 1.6, at least on Debian. This might be R's .onLoad correctly loading dynamic libraries and Open MPI is not required to be compiled with static libraries enabled.
Fix all "no visible binding for" NOTE.
inst/Rslaves.bat is removed. Also .bat is renamed to .cmd. Modify src/Makevars.win so that Rmpi can be compiled under both MPICH2 32 and 64bit.
Rmpi has been successfully compiled under 64bit R 2.14.1 on Windows 7 64bit platform. The latest MPICH2 1.4.1p1 or later is needed.
mpi.gather.Robj has been modified to include ... option. Now one can use simplify=FALSE to tell mpi.gather.Robj not to simplify outputs, suggested by "George Ostrouchov" <ostrouchovg at ornl.gov>.
To compile Rmpi against OpenMPI 1.5 or 1.6, you have to enable
static library during compiling OpenMPI:
./configure
--disable-dlopen
This will create static libraries in
additional to shared libraries (not my own idea; coming from here).
Otherwise Rmpi will not be compiled. On the other hand, since OpenMPI
1.5, libmpi.so.0 no longer exists. To accommodate both OpenMPI 1.4 or
older and OpenMPI 1.5 or newer, libmpi.so.0 will be loaded
first. If it fails, libmpi.so wil be loaded next. If either
libmpi.so.0 or libmpi.so is not loaded properly, it will generate an
error like
mca_paffinity_linux.so:
undefined symbol: mca_base_param_reg_int
ERROR: loading failed
Loading libmpi.so with error checking was suggested and patched by "Hugo Mildenberger" <Hugo.Mildenberger at web.de>.
Add –lmpl and –lopa to MPICH2 lib in configure.ac so that Rmpi works under FreeBSD and MPICH(2), suggested by "Rainer Hurling" <rhurlin at gwdg.de>.
An option quiet=FALSE has been added to mpi.comm.spawn and mpi.spawn.Rslaves. If enabled, it avoids printing anything if there
were no errors. This option was suggested and patched by “Manuel López-Ibáñez” <manuel.lopez-ibanez at ulb.ac.be>.
Default CC="gcc -std=gnu99" has been added to configure.ac in case R_HOME is not set properly, suggested by "Tony Dale" <Tony.Dale at canterbury.ac.nz>. mpi.setup.sprng has been removed. Rmpi will reply only on the package rlecuyer for parallel RNG.
slave.hostinfo is modified so that in the default short form with 8 or more slaves, the first 3 and last 2 slaves are shown.
If slaves are not generated through spawning (mpi.comm.size(0) < mpi.comm.size(comm)), mpi.close.Rslaves is not going to delete any log files.
Modify configure.ac so that Rmpi can be compiled under 64bit openSUSE 11.3, suggested by Martin Morgan <mtmorgan at fhcrc.org>.
All LAM related commands are called only under LAM , i.e., if .Call("mpidist",PACKAGE="Rmpi") == 2 is true. Thanks to Prof. Brian Ripley and Martin Morgan <mtmorgan at fhcrc.org> for identifying related bugs.
Add a stream to the master in mpi.setup.rngstream so that master can replicate its own RNG.
Add Rslaves32.bat to work under R-2.12.0 32bit.
mpichhosts has been rewritten to accommondate new changes in MPICH2 v1.2.1. system("smpd -get hosts", intern=TRUE) is used to find MPICH2's hosts information instead of registry query. Due to a bug in R's function system, it returns empty under both Windows 7 32bit and 64bit for R 2.10.1 or lower. This bug is fixed in R 2.11.0. When hosts information is not specified in MPICH2 (default is used), mpichhosts() will find total number of CPUs from registry query.
Add mpi.parMM and mpi.iparMM functions for parallel matrix products. mpi.parMM is a clone of parMM from snow package. It has a load balancing parmeter job.num. Although parallel matrix product can be achieved by mpi.parApply(A, 1, "%*%", B), mpi.parMM is more efficient and faster. Choose job.num to be 2 to 4 times of total number of slaves to futher speed up of huge matrix product. mpi.iparMM is a nonblocking version of mpi.parMM without load balancing feature.
Fix a bug in mpi.iapplyLB.
configure.ac has been completely rewriteen by Prof. Brian Ripley. Now it allows --with-Rmpi-include= and --with-Rmpi-libpath= options so that MPI's include and lib paths don't need to have the same root. The original option --with-Rmpi= is still available. Another feature includes the option --with-Rmpi-type= to specify OPENMPI, LAM, or MPICH. Thanks "Prof. Brian Ripley" <ripley at stats.ox.ac.uk>.
Fix a bug in mpi.comm.dup. Thanks "George Ostrouchov" <ostrouchovg at ornl.gov>.
Fix "Unknown package(s) ‘serialize’ in Rd xrefs" warning, compiled under 2.10.0 or higher.
Add nonblocking mpi.iapply, mpi.iapplyLB, mpi.iparApply, mpi.iparLapply, mpi.iparSapply, mpi.iparCapply, mpi.iparRapply, and mpi.iparReplicate functions. In those functions, mpi.iprobe is used while master is waiting for slave results. After mpi.iprobe returns FALSE, master went sleep with duration specified by sleep argument (default 0.001 sec or 1ms). Now, under OpenMPI, master will use little CPU cycles while waiting for slave results.
Fix a segment fault under OpenMPI 1.3 and Debian 64bit. Thanks "Jeff Squyres" <jsquyres at cisco.com> and "Dirk Eddelbuettel" <edd at debian.org> for the fixing.
Add C level function mpidist. .Call("mpidist",PACKAGE="Rmpi") returns 1 if it is OpenMPI; 2 if LAM and 3 if MPICH.
Modify .First.lib so that dynamic library will be loaded first. Then in case LAM is detected, lamboot might be used.
Add additinoal detections of OpenMPI at various locations, including /usr/lib/openmpi, suggested by "Dirk Eddelbuettel" <edd at debian.org>.
Modify mpi.parApply so that parallel matrix product, like mpi.parApply(A, 1, "%*%", B), can be used. This was suggested by "Martin Morgan" < mtmorgan at fhcrc org>
Modify mpi.apply so that mpi.scatter.Robj is used instead of mpi.send.Robj. This may result some improvements with large datasets.
Fix a bug in mpi.scatter.Robj and add a new function mpi.scatter.Robj2slave which is more efficient than using mpi.parLapply to distribute components of a list to slaves.
sleep.exe is removed from inst directory to avoid the possibility of virus infection. So is the corresponding line in Rslaves.bat.
Modify Rprofile in inst directory so that it will not generate errors under OpenMPI when leaving slaves.
dlopen has been used to load libmpi.so explicitly. This is mainly
useful for Rmpi under OpenMPI where one might see many error
messages:
mca: base: component_find:
unable to open osc pt2pt: file not found (ignored)
if
libmpi.so is not loaded with RTLD_GLOBAL flag.
In Rslave.sh, [ $3 == "needlog" ];
is changed to [ "$3" = "needlog"
]; This might stop those dreadful error messages:
[:
20: ==: unexpected operator
during spawning R slaves,
suggested by "Dirk Eddelbuettel" <edd at debian.org>.
Similarly, in configure.ac,
test "$MPI_ROOT"=="";
is changed to [ [ "$MPI_ROOT" = ""
] ]; Otherwise it will not take --with-mpi argument, reported
by "Shintaro KATAYAMA" <katayama at gsc riken jp>.
All objects in C codes are properly initialized, done by "Dirk Eddelbuettel" <edd at debian.org>. This will stop warning messages during compiling on some Linux distributions such as Fedora.
MPI_Init(&fake_argc, (char ***)(void*)&fake_argv) has been used when compiling Rmpi under MPICH otherwise loading Rmpi in R will result segmentation errors. This was suggested by "Aaron Sethman" <sethmana at mail.nih.gov>. For other systems even with MPICH2, MPI_Init((void *)0,(void *)0); is still used.
configure.ac has been reorganized. Now it will detect OpenMPI first, then LAM and finally MPICH or MPICH2. It can also detect OpenMPI or LAM on both Fedora 32bit or 64bit systems. Users can still overwrite those detections with --with-mpi argument or MPI_ROOT environment variable.
Modify mpi.spawn.Rslaves so that it will not get "System could not find the path" error during spawning under Windows.
The original .splitIndices function (copied from snow) did not split indices evenly. As suggested by my colleague Dr. McLeod <aim at stats.uwo.ca>, I used the algorithm in equal.count from lattice package to split indices evenly.
Transferring in mpi.bcast.cmd has been changed from character to
raw. This might add some overhead but it will avoid any issues
associated with character transfer.
All character
vector transfers have been rewritten since R 2.6.0 does not allow any
modifications under C level. Due to complication of creating multiple
temp buffers for storing receiving buffers, only the first element in
a character vector (R char vector) is transferred. For sending
buffers, #define CHAR2(x) ((char *) CHAR(x)) has been used,
suggested by Prof. Ripley <ripley at stats.ox.ac.uk>. In
addition character receive in mpi.irecv is disabled.
Add
lib64/lam in search path in configure.ac. This might make Rmpi
compile under FC 64bit.
Change libmpi.a to libmpi.so in
configure.ac and add detection of openmpi dir so that Rmpi can be
compiled under OpenMPI with full functionality. R can be started like
R -> library(Rmpi) -> mpi.spawn.Rslaves() without remote nodes.
Or R can be started like "mpirun --host hostfile -np 1 R
--no-save" and library(Rmpi) -> mpi.spawn.Rslaves() bur R
runs in noninteractive mode without command recall. One major problem
associated with OpenMPI is that all R slaves run at 100%CPU while
they are waiting for master instructions. So the implementation of
MPI_BCAST in OpenMPI is different from that in LAM. Hence Rmpi under
OpenMPI is not good for coding and testing. Rather it should be used
for real computations.
Add AC_PROG_CC in configure.ac to
work with SUSE linux, suggested by George Ostrouchov <ostrouchovg
at ornl.gov>.
Modify configure.ac so that LAM on Mac OS
X can be detected, provided that OS X version of LAM at
www.lam-mpi.org is installed.
Change configure.in to
configure.ac and add MPI_INCLUDE, LDFLAGS, CFLAGS, in case MPI_ROOT
is supplied by users, suggested by "Dennis van Dok"
<dennisvd at nikhef.nl>.
R raw data type has been added as type 4. This is used for
transferring R objects, i.e., all R objects are serialized to raw
type for transferring. This is in line with the change of serialize
in R 2.4.0. Wrappers .mpi.serialize and .mpi.unserialize are used in
case Rmpi is running under R 2.2.0 to R.2.3.1.
Add a
few examples to several mpi functions.
Add
mpi.scatter.Robj, mpi.gather.Robj, and mpi.allgather.Robj.
Change "MPI_Comm_free(&master);" in mpi_is_master of
Rmpi.c to "if (!check) MPI_Comm_free(&master);" Thanks
to Martin Siegert <siegert at sfu.ca> for fixing.
Some
patch to configure.in, suggested by Aaron A. King <aaron.king at
umich.edu>, so that if MPI_ROOT is supplied as an environmental
variable or is given by users, mpi.h will be searched within
subfolders of MPI_ROOT.
Rename some internal functions used by slaves. They follow by the
convention .mpi.slave.xxx and hidden.
Automatically
loading SPRNG in mpi.spawn.Rslaves is dropped. Instead one can use
mpi.setup.sprng or mpi.setup.rngstream to setup parallel RNG on
slaves.
Some codes tidy on mpi.remote.exec and mpi.parSim. As a result,
mpi.wrap.fun is dropped.
Add mpichhosts to find host names
on windows platform.
Modify mpi.universe.size to return
length(mpichhosts())-1 under windows platform.
Add
mpi.apply: an array is scattered to slaves and then a fun is
applied.Add mpi.applyLB: a load balancing version of mpi.apply.
Add
mpi.parApply, mpi.parLapply, mpi.parRapply, mpi.parCapply, and
mpi.parSapply: they are clones of those functions in package snow if
job.num=slave.num. They are load balanced if job.num >
slave.num.
Add mpi.parReplicate: wrapper for
mpi.parSapply, mainly used in parallel simulation. It will eventually
replace mpi.parSim due to its efficiency.
Add network map
drive function in inst/Rslaves.bat. May see some error messages if
multiple R slaves are spawned on the same remote node.
Add
nonblocking calls: mpi.isend, mpi.irecv, mpi.isend.Rojb, mpi.iprobe,
mpi.test, mpi.testany, mpi.testsome, mpi.testall, mpi.wait,
mpi.waitany, mpi.waitsome, mpi.waitall, mpi.cancel, and
mpi.test.cancelled.
Add mpi.comm.maxsize,
mpi.request.maxsize, mpi.status.maxsize, mpi.realloc.comm,
mpi.realloc.request, and mpi.realloc.status.
Rmpi can be compiled and run under MPICH2 for Windows.
src/Makevars.win is added so that Rmpi can be compiled without manual
configuration if MPICH2 for Windows is installed at default place.
Instructions to run Rmpi on windows platform are given in
here.
inst/Rslaves.bat windows
batch file is created for spawning R slaves under windows platform
(localhost only, a restriction imposed by MPICH2 for Windows).
Full
sets of creating a Cartesian topology APIs are
implemented.
inst/Rslaves.sh is modified so that the
same R program Master R is ruining will be used by slaves Rs. This
allows co-existences of different versions of R.
inst/Rprofile
used by mpiexec under unix and windows platforms is modified. The
main change is that sys.parent() in eval is removed. This is due to a
unknown reason that the infinite loop cannot be broken if the env
R_RPOFILE is used to launch multiple Rs. Objects created on R slaves
are no longer global unless <<- are used, though they are
accessible during the duration of R slaves. If Rprofile is copied as
.Rprofile at the working directory, sys.parent() can still be used
(need to modify .Rprofile).
To run multiple Rs
without using .Rprofile, add the following line
R_PROFILE=${R_HOME_DIR}/library/Rmpi/Rprofile; export R_PROFILE in
the R shell script (after "export R_HOME_DIR") and rename
it, say, Rmpi. Then one can use mpiexec -n number_of_nodes Rmpi
--no-save -q to launch multiple Rs (assume Rmpi is in the
PATH).
Parts of nonblocking point-to-point MPI APIs are
implemented at C level only. They will be released in the next
version of Rmpi.
mpi.remote.fun is deprecated and mpi.remote.exec
(mpi.remote.slave) has been simplified by using do.call
instead.
replicate is used in mpi.parallel.sim
(mpi.remote.slave) instead of sapply. To be consistent,
mpi.parallel.sim uses cbind to combine all slave results together.
Delete .Random.seed in .First.lib if it is a user-defined RNG. Add needsprng = TRUE argument to mpi.spawn.Rslaves so that SPRNG can be loaded if there is the package rsprng. Implement mpi.parallel.sim for parallel Monte Carlo simulation. Some ideas from student projects of Mr. K. Yu, Ms. Tsai, and Mr. W. Wolterman have been used. Thanks to them for their wonderful works.
Update to require R-1.8.1 and remove the requirement for the serialise package (build into R-1.8.1).
Some minor bugs fix.
Add #ifdef MPI2 to src/Rmpi.c and change MPI_Comm_set_errhandler back to MPI_Errhandler_set for backward compatibility with MPI-1.2, suggested by Ross Boylan.
Add -DMPI2 in configure.in in case lam lib is detected. Otherwise MPI-2 related functions will be disabled.
Add detect MPI in /usr/lib/mpi or /usr/lib/mpich, hopefully to catch MPICH MPI.
Add some lines to configure.in so that configure can detect lam on Debian Linux. Thanks to Kurt Hornik <Kurt.Hornik at wu-wien.ac.at> for Debian instructions.
Internal use.
Add a function string for creating a (binary) character buffer. Remove all C++ style comments. Remove all deprecated functions. Remove mpi.barrier from slavedaemon.R (it causes slaves hanging if master crashed).
Take SPRNG out. Use rsprng package instead. Use mpi.init.sprng to
start SPRNG.
Require serialize package
mpi.send.Robj,
mpi.recv.Robj, mpi.bcast.Robj, and mpi.bcast.Robj2slave use serialize
to transfer R objects.
mpi.bcast.send and mpi.bcast.recv
are deprecated.
mpi.bcast.send.cmd and mpi.bcast.recv.cmd
are replaced by mpi.bcast.cmd
mpi.bcast.send.Robj and
mpi.bcast.recv.Robj are replaced by mpi.bcast.Robj
New
instructions for installing Rmpi for other MPI
implementations.
Thanks to Kurt Hornik <Kurt.Hornik
at wu-wien.ac.at> for Debian instructions.
Some bugs fix. Add Rprofile in inst Add a function mpi.comm.dup
Some bugs fix. Add a function mpi.quit
Initial public release