Testing stochastic systems presents challenges for the traditional unit testing framework, since in general either the tests are trivial, or else they will sometimes fail. A recent research project developed a methodology to put stochastic unit tests on a firm statistical basis, and this is now used in writing such tests in Opus and UrbanSim. See Hana Ševcíková, Alan Borning, David Socha, and Wolf-Gideon Bleek, ``Automated Testing of Stochastic Systems: A Statistically Grounded Approach,'' in Proceedings of the International Symposium on Software Testing and Analysis, ACM, July 2006 (available from http://www.urbansim.org/papers/sevcikova-issta-2006.pdf).
A result of this is that, indeed, such stochastic unit tests may fail, even though the underlying code is correct. In that case, just run the test again.
Within the UrbanSim project, we have been continuing to gather data on when
the stochastic unit tests fail, as fodder for a possible followup research
paper on this topic. To help with this, UrbanSim project members have been
asked to set the system environment variable
MYSQLSTOCHASTICTESTLOGGER to the
host name for the UrbanSim SQL Server, where a table accumulates this
information. For everyone else, just ignore this environment variable --
if it doesn't exist, the system won't attempt to log the information.