Variance of the Estimated Efficiency

Introduction

Hoaglin and Andrews (1975) have observed that it is important to give interval estimates or measures of uncertainty to quantities estimated by simulation. Here we derive a general formula for estimating the standard error of the statistical efficiency estimated in a simulation study.

Derivation Using Taylor Series Expansion

Suppose N simulations are to be performed and N is quite large. Let (Subscript[X, i]) and (Subscript[Y, i]), i = 
        1,  ...  , N denote the squared error in the i-th simulation for estimators 𝒳 and 𝒴 respectively. We will assume that before the simulations are run, that the bivariate random variables ((Subscript[X, i]),  (Subscript[Y, i ])) are an independent sequence of random variables with finite variance. Note that  (Subscript[X, i]) and (Subscript[Y, i]) may themselves be correlated and hence not independent of each other but that it is the sequence which is assumed to be independent.

Then  sample average squared errors, \(TraditionalForm\`\((X\&_, \ Y\&_)\)\),  are approximately bivariate normal since by the usual central limit theorem, any linear combination of  (Overscript[X, _ ]) and  \(TraditionalForm\`Y\&_\) is approximately normal.  The estimated relative efficiency of method 𝒴 vs. method 𝒳 is \(TraditionalForm\`e\&^\  = X\&_/\ Y\&_\).  Let \(μ\_x\&_\)=E\({X\&_}\) and \(μ\_y\&_\)=E\({Y\&_}\). Employing a Taylor series linearization of  \(TraditionalForm\`\(e\&^\)\),

\(e\&^\  ≐ μ\_x\&_\/μ\_y\&_ + \((X\&_ - μ\_x\&_)\) 1\/μ\_y\&_ - \ \((\ 
                    Y\&_ - μ\_y\&_\ \ )\) μ\_x\&_\/μ\_\(y\&_\)\%2\)

Hence, after some algebraic simpliciation it can be shown that,

\(Var {e\&^\ } ≐ \(1\/N\) \((σ\_x\/μ\_y\%2 + \(\(σ\_y\) μ\_x\%2\)\/μ\_y\%\(\(\ \)\(4\)\) - \ 
                    2  σ\_\(x, y\)\ \ μ\_x\/μ\_y\%\(\(\ \)\(3\)\))\)\ 2  σ\_\(x, y\)\ \ μ\_x\/μ\_y\%\(\(\ \)\(3\)\)\)

where (Subscript[μ, x])=E{(Subscript[X, i])}, (Subscript[μ, y]) =E{(Subscript[Y, i])},  (Subscript[σ, x]) = Var {(Subscript[X, i])}, (Subscript[σ, y]) = Var {(Subscript[Y, i])} and  (Subscript[σ, x,  y]) = Cov {(Subscript[X, i]),  (Subscript[Y, i])}. These values can then be estimated by the sample moments of the observed (Subscript[X, i])'s and (Subscript[Y, i])'s.

Implementation in Mathematica

The following Mathematica function evaluates the estimated relative efficiency and its standard error for method 𝒴 vs. method 𝒳 given observed data (Subscript[X, i]) and (Subscript[Y, i]), i = 
        1,  ...  , N.

Off[General::spell]; Off[General::spell1];
SDRelativeEfficiencyYvsX[X_, 
          Y_] := 
Module[{μx = Mean[X], μy = 
              Mean[Y], σx, σy, σxy}, 
σx = Mean[(X - μx)^2]; 
σy = 
            Mean[(Y - μy)^2]; 
σxy = 
            Mean[(X - μx)*(Y - μy)]; 
e = μx/μy; 
Sqrt[(σx/μy^2 + (σy*μx^2)/μy^4 - (2*σxy*μx)/μy^3)/
              Length[X]]
];

Numerical Illustration

<< Statistics`

s1 = RandomArray[NormalDistribution[0, 0.01], 500]; 
    d1 = s1 + RandomArray[NormalDistribution[0, 0.0025], 500];

SDRelativeEfficiencyYvsX[(s1 - Mean[s1])^2, (d1 - 
            Mean[d1])^2]

Comparison with Bootstrap

More generally, it is easily seen that if (Subscript[X, i]) and (Subscript[Y, i]), i = 
        1,  ...  , N, denote any random variables with finite variances and \(TraditionalForm\`e\&^\  = X\&_/\ Y\&_\)  then the standard deviation of  (Overscript[e, ^ ]) is given by the above formula.

The bootstrap (Efron & Tibshirani, 1993) provides an alternative but less convenient method of evaluating the standard error.  Efron & Tibshirani show that in most circumstances 200 bootstrap iterations gives sufficient precision for estimating the standard error.

SDRelativeEfficiencyYvsXBootstrap[X_List, Y_List] := 
      Module[
{i}, 
Sqrt[
Variance[
Table[
              Mean[X[[i = 
                      Table[Random[
                          Integer, {1, Length[X]}], {Length[
                            X]}]]]]/
                Mean[Y[[
                    i]]], {200}]
]
]
]

SDRelativeEfficiencyYvsXBootstrap[(s1 - Mean[s1])^2, (d1 - 
            Mean[d1])^2]