Some Advice on Doing High-Quality Computational Experiments

1. Axes of graphs, or tables if you choose to give your data that way,
should be clearly labelled.  Don't just give a jumble of numbers!
Time isn't just a number; you need to give the units (seconds?
microseconds?  clock cycles?)

2. You need to take enough data to convince a skeptical reader that
the behavior really is Theta(n log n) or whatever, and get a
reasonable estimate of the constant.  If n is too small, you'll get a
"time" of zero, so you need to take enough data fo larger values of n.
Typically your values of n should range over at least a factor of 100
(i.e. the largest should be 100 times the smallest).

3. If your data fluctuates wildly, try taking the average over several
independent trials for each value of n.  If your data is so noisy that
its behavior isn't convincing, or if the data is fit just as well by
some form other than the one you're trying to prove, you haven't
passed the "skeptical reader" test.

4. You need to explain enough about how the experiment was performed so
that another person could read your report and reproduce your results.  
For instance, how many trials did you do for each value of n?

5. If you get wierd data, report it honestly.  For instance, many algorithms
start taking longer on really huge lists because we start swapping virtual
memory from the hard drive.  It's ok to not include some data in how you
do your fits, but you should report this and explain it.

I don't need an elaborate report, but I *do* need something which is
readable and convincing.  Knowing how to do good experiments will help
you in a wide range of future careers!

- Cris


_______________________________________________
Cs361 mailing list
Cs361@www.cs.unm.edu
http://www.cs.unm.edu/mailman/listinfo/cs361