Fit Algorithm Performance Comparison (Supporting Information for “Optimal Diffusion Coefficient Estimation in Single-Particle Tracking” by A.J. Berglund 1 & X. Michalet 2 ) 1 Center for Nanoscale Science and Technology National Institute of Standards and Technology, Gaithersburg, MD 20899 [email protected]2 Department of Chemistry and Biochemistry University of California at Los Angeles, Los Angeles, CA 90095. [email protected]The following pages contain a number of graphs which can be examined individually but are best used as follows: ● Fig. S1 shows a comparison of execution time for the MLE and OLSF algorithms applied to simulated trajectories of different length N, for two different values of the reduced square localization error x (the blur factor R being set to 0 in the simulations). ● The next set of graphs is a set of “ Success Maps” for D, σ 2 or D and σ 2 obtained by either the OLSF or MLE algorithm, as described in the article. ● The next 288 pages show comparisons of the MLE and OLSF algorithms used with simulated 1D trajectories for different set of (x, N) parameters. A brief description of these two types of graphs is provided below. Success Maps Success is defined here as percentage of trajectories for which the fitted parameter is (or both parameters are) within 25% of their theoretical values for a given (x, N) pair. The maps are color-coded, with green representing 100% of success (or 1) and red corresponding to complete failure (0). Superimposed to the maps are curves derived from the Cramér-Rao bound for the parameter studied in the map (or quadratic mean of the Cramér-Rao bounds for the “D and σ 2 ” maps). These curves corresponds to contours where an optimal unbiased estimator reaches a given success fraction (0.9, 0.8,…, 0.1). These curves provide a simple way to gauge the performance of each algorithm and/or the practical usefulness of the Cramér-Rao bound. Another way to read these maps is to choose a value of x and read the number of trajectory points N needed to achieve a fitted value of D within 25% of its correct value with some desired probability. Note that the D and σ 2 maps are approximately symmetric with respect to x = 1. A consequence of this is that obtaining a good estimate of D and σ 2 is possible only in a very limited region of the (x, N) space, as shown by the “D and σ 2 ” maps. The two methods appear comparable in terms of their success rate (as defined above). However, it is useful to compare the two methods on a case by case basis (comparing the fitted values obtained by OLSF and MLE on identical simulated trajectories). The maps discussed above are “clickable”: by hovering with the mouse over (x, N) points of the map corresponding to actual data sets, the cursor should change to a finger pointing hand, indicating that this point is linked to another graph. By clicking on one of these points, a new page of the document containing details of the underlying data is opened.
Transcript
Fit Algorithm Performance Comparison
(Supporting Information for “Optimal Diffusion Coefficient Estimation in Single-Particle Tracking”by A.J. Berglund1 & X. Michalet2)
1Center for Nanoscale Science and Technology
National Institute of Standards and Technology, Gaithersburg, MD [email protected]
2Department of Chemistry and BiochemistryUniversity of California at Los Angeles, Los Angeles, CA 90095.
[email protected] The following pages contain a number of graphs which can be examined individually but are best used as follows:
● Fig. S1 shows a comparison of execution time for the MLE and OLSF algorithms applied to simulated trajectories of different length N, for two different values of the reduced square localization error x (the blur factor R being set to 0 in the simulations).
● The next set of graphs is a set of “Success Maps” for D, σ2 or D and σ2 obtained by either the OLSF or MLE algorithm, as described in the article.
● The next 288 pages show comparisons of the MLE and OLSF algorithms used with simulated 1D trajectories for different set of (x, N) parameters.
A brief description of these two types of graphs is provided below. Success Maps Success is defined here as percentage of trajectories for which the fitted parameter is (or both parameters are) within 25% of their theoretical values for a given (x, N) pair. The maps are color-coded, with green representing 100% of success (or 1) and red corresponding to complete failure (0). Superimposed to the maps are curves derived from the Cramér-Rao bound for the parameter studied in the map (or quadratic mean of the Cramér-Rao bounds for the “D and σ2” maps). These curves corresponds to contours where an optimal unbiased estimator reaches a given success fraction (0.9, 0.8,…, 0.1). These curves provide a simple way to gauge the performance of each algorithm and/or the practical usefulness of the Cramér-Rao bound. Another way to read these maps is to choose a value of x and read the number of trajectory points N needed to achieve a fitted value of D within 25% of its correct value with some desired probability.Note that the D and σ2 maps are approximately symmetric with respect to x = 1. A consequence of this is that obtaining a good estimate of D and σ2 is possible only in a very limited region of the (x, N) space, as shown by the “D and σ2” maps.The two methods appear comparable in terms of their success rate (as defined above). However, it is useful to compare the two methods on a case by case basis (comparing the fitted values obtained by OLSF and MLE on identical simulated trajectories).The maps discussed above are “clickable”: by hovering with the mouse over (x, N) points of the map corresponding to actual data sets, the cursor should change to a finger pointing hand, indicating that this point is linked to another graph.By clicking on one of these points, a new page of the document containing details of the underlying data is opened.
MLE & OLSF Algorithms Comparison Each of these “detail” pages contains 3 different graphs representing the following information for a particular (x, N) pair:
o Representation of all OLSF-fitted (D, σ2) pairs for the 1,000 simulated trajectories (black squares), as well as all MLE-fitted (D, σ2) pairs (red circles) and 1-, 2-, and 3-standard deviation ("1CR", "2CR", and "3CR") ellipses (blue curves with decreasing thickness).
❍ Representation of the OLSF- versus MLE-fitted D for each simulated trajectories and 1-, 2-, and 3-standard deviation ("1CR", "2CR", and "3CR") squares for D (they may appear rectangular due to the different ranges covered by the MLE- and OLSF-fitted values).
❍ Representation of the OLSF- versus MLE-fitted σ2 for each simulated trajectories and 1-, 2-, and 3-standard deviation ("1CR", "2CR", and "3CR") squares for σ2.
Note that all values are normalized by the true value of the parameter. For clarity, outliers (5 %) have been removed from these graphs in order to enhance legibility. In other words, only the 950 data points closest to the theoretical values are displayed. The first scatter plots allow a direct comparison of the OLSF and MLE performance in different cases, as well as of each method with the CR bound.It is worth noting that the OLSF algorithm can return negative values of D and/or σ2, while the MLE algorithm is positively biased for D at large x and for σ2 at small x.This is made particularly visible by the two latter types of scatter plots, which compare the D or σ2 values fitted by OLSF and MLE on each simulated trajectory. The gray line indicates identity between the OLSF and MLE parameter. Points above this line correspond to a larger MLE estimate, whereas points below this line correspond to a larger OLSF estimate. The rectangles center indicates the true value of each parameter.It is apparent that in a significant fraction of failure cases, negative OLSF values correspond to almost zero MLE values, showing that neither of the methods performs well in those cases. The user of these algorithms is warned that MLE may thus return positive values that may be artificially larger than the actual D value, or on the contrary, artificially small values, while OLSF might return negative values in both cases. In a sense, the unphysical results of OLSF is a clear warning that the data is probably unfit for either type of analysis, either because of too few trajectory points or too large a reduced square localization error.
Fig. S1: Comparison of MLE (blue curves) and OLSF (green curves) algorithms execution time as a function of number of trajectory points N. Two different situations were simulated (x = 1 and x = 100, R = 0, d = 1 in both cases). The execution time of the OLSF algorithm is dominated by the computation of the fraction of the MSD curve needed for the fit. This increases quadratically with the number of points. In contrast, the MLE algorithm only deals with single step displacements, whose number grows linearly with the trajectory length. To offset these differences, the number of iterations required to converge to the best fir values appears to be larger for the MLE algorithm, making this approach slower for small trajectories (N < 3000). Note however that even in these cases, the execution time is very short. These differences will become irrelevant as computer speed increases.
