Paper Info

Title | ||
---|---|---|

Testing Hypotheses On Simulated Data: Why Traditional Hypotheses-Testing Statistics Are Not Always Adequate For Simulated Data, And How To Modify Them |

Abstract | ||
---|---|---|

To check whether a new algorithm is better, researchers use traditional statistical techniques for hypotheses testing. In particular, when the results are inconclusive, they run more and more simulations (n(2) > n(1), n(3) > n(2), . . . , n(m) > n(m-1)) until the results become conclusive. In this paper, we point out that these results may be misleading. Indeed, in the traditional approach, we select a statistic and then choose a threshold for which the probability of this statistic "accidentally" exceeding this threshold is smaller than, say, 1%. It is very easy to run additional simulations with ever-larger n. The probability of error is still 1% for each n(i), but the probability that we reach an erroneous conclusion for at least one of the values n(i) increases as m increases. In this paper, we design new statistical techniques oriented towards experiments on simulated data, techniques that would guarantee that the error stays under, say, 1% no matter how many experiments we run. |

Year | DOI | Venue |
---|---|---|

2006 | 10.20965/jaciii.2006.p0260 | JOURNAL OF ADVANCED COMPUTATIONAL INTELLIGENCE AND INTELLIGENT INFORMATICS |

Keywords | Field | DocType |

hypothesis testing, simulated data | Data dredging,Alternative hypothesis,Population,Statistic,Computer science,Testing hypotheses suggested by the data,Artificial intelligence,Statistics,Exploratory data analysis,Standard deviation,Machine learning,Statistical hypothesis testing | Journal |

Volume | Issue | ISSN |

10 | 3 | 1343-0130 |

Citations | PageRank | References |

0 | 0.34 | 7 |

Authors | ||

3 |

Authors (3 rows)

Cited by (0 rows)

References (7 rows)

Name | Order | Citations | PageRank |
---|---|---|---|

Richard Aló | 1 | 11 | 1.93 |

Vladik Kreinovich | 2 | 1091 | 281.07 |

Scott A. Starks | 3 | 61 | 12.76 |