Timing Comparisons of Mathematica, MATLAB, R, S-Plus, C & Fortran 

Ian McLeod & Hao Yu, March 2, 2002

Purpose

The purpose of these timings is to demonstrate that in certain circumstances the widely held view that you can always dramatically improve on the cpu time required for lengthy computations by using compiled C or Fortran code instead of advanced QPEs such as Mathematica is wrong.

It is frequently stated that interpretative languages are not as useful as compiled languages such as C and Fortran for numerically intensive computations.  It is obviously the case that an well written C program can be expected to outperform Mathematica, R, S-Plus or MATLAB but there are two caveats that need to be understood.  First, if the C program is not efficiently programmed using the best possible algorithm then in fact it may take longer.  Second, by using the Mathematica byte-code compiler the speed can often be within a factor of three of compiled C code.   So if the C code only runs about 3 times as fast, but takes 20 as much time to program and debug as Mathematica code, there is a strong incentive to use Mathematica at least in many situations.  The factor 20 was not made up, but is based on a study done by IBM comparing productivity of Cobol programmers with APL programmers and I believe this would be similar to comparing Mathematica with C or C++,  at least in many situations.  A byte-code compiler is presently being developed for R as well.  MATLAB already has a byte-code compiler.

The Test Problem

The test problem was chosen in order to demonstrate the compiled C or Fortran code does always result in significantly improved timings.  Consider the problem of inverting the covariance matrix of a first-order autoregressive process with dimension n=1000 and correlation parameter 0.5 and unit innovation variance.  The (i,j) entry of this matrix is 0.5^abs(i-j).  

Quantitative Programming Environments (QPEs)

Mathematica

The notebook MatrixInverseTiming.nb contains the compiled and un-compiled version of the test program.  The compiler used in Mathematica is an internal byte-code compiler which optimizes the computation but does not produce native binary code.  In the test example chosen the compiler only produces about a 30% reduction is cpu time but in other situations a much more dramatic reduction can often be possible.  For example,  in the article by McLeod and Quenneville (2000, Mean likelihood estimators. Statistics and Computing 11, 57-65), it was shown that the speed exact computation of the MA(1) likelihood for samples of size n=50 could be improved by a factor of more than 50 by using the Mathematica compiler. 

MATLAB

To get the timing under MATLAB just put the script file, test.m, on the MATLAB path and run the commands,
profile on
test(0.5, 1000)
profile report

R

Just save the R file MatrixInverseTime.R and to run simply input it to R with the R source function.

Fortran

The Fortran source code is in  test0.for.  An MS Windows executable that was built with Compaq Fortran 6.0 using the compiler settings for No debug and Full Optimizations is in test0.exe .  This program can also be compiled and run on linux with f77.  We also experimented with using the LAPACK library available on linux.  The program test1.f uses a general matrix inverse routine while test2.f uses a specialized subroutine for symmetric positive definite matrices.  The linux PC executables are also available as test1 and test2.

C++ Code

Windows source code comprises two files:  test.cpp and test.h.  The MS Windows console executable test.exe  was built using Visual Studio 6.0 with compiler settings for No Debug and Maximum Optimizations.

A similar program was created for linux and compiled with g++,  testThe source code for this program is in test.C and nrutil.h

S-Plus

For S-Plus windows, source the file MatInvTime.S and for S-Plus Linux,  MatInvTimeUnix.S

Timings on Various Machines

We present the  timings on various MS Windows and Linux PCs. 

karl.stats.uwo.ca 

PC with a 1 GHz Xeon processor running Red Hat linux with 4 GB RAM.

Program CPU Time in Seconds
Mathematica, uncompiled 47
Mathematica, compiled 36
R V. 1.3 55
C++ 31
Fortran 31(a)   46(b)   29(c)   9(d)

(a) compiling with f77 -fomit-frame-pointer -O3 -funroll-loops test.f
(b) compiling with f77 -O3   test.f
(c) using LAPACK general matrix inverse
(d) using LAPACK symmetric matrix inverse

arva.stats.uwo.ca 

PC with a 800 MHz  processor running Red Hat linux with 1 GB RAM.

Program CPU Time in Seconds
Mathematica, uncompiled 100
Mathematica, compiled 71
R V. 1.4.1 135
S-Plus, V.6 Linux 253
C++ 76
Fortran 67(a)  24(b)

(a) using LAPACK general matrix inverse
(b) using LAPACK symmetric matrix inverse

aim.stats.uwo.ca 

PC with a 400  MHz Pentium II  processor running WinNT with 256 MB RAM..

Program CPU Time in Seconds
Mathematica, uncompiled 127
Mathematica, compiled 99
R 77
MatLab V.5.3 R11 48
S-Plus Version 5.2 109
C++ 98
Fortran 79

Remark: The MatLab timing seems amazing. On a slightly more powerful system, in the table following,  the MatLab timing is approximately the same as for Mathematica.  

Ian McLeod's Home Computer 

PC with a 732  MHz Pentium III  processor running Win2K with 256 MB RAM.

Program CPU Time in Seconds
Mathematica, uncompiled 51
Mathematica, compiled 36
MATLAB V5.3 R11 34
R 47
S-Plus Version 5.2 66
C++ 66
Fortran 54

arva.stats.uwo.ca

PC with 2 Xeon CPUS at 2 GHZ with 2 Gig RAM

Program CPU Time in Seconds
Mathematica, uncompiled 21.9
Mathematica, compiled 15.8
MATLAB V5.3 R11  
R 99.9
S-Plus Version 6.1 97.2
C++ 12
Fortran 10.1

 

 

 

Last update: Wednesday, June 26, 2002 12:18:31 PM