[UrbanSim-Users] managing estimation results

Hello,

Below is an excerpt from the documentation for a little utility we've
developed called "TrapperKeeper." Its purpose is to help us keep
track of estimation results for UrbanSim.

We'd be happy to share this work with all interested. We need to do
some work on our SVN server before we can allow anonymous access. In
the mean time, if you are dying to get your hands on it, I can make a
zip file available.

Best wishes,

Brian Miles
Graduate Assistant
Spatial Analysis Lab - Rubenstein School of Environment and Natural
Resources
University of Vermont

1. Introduction

Documentation for the "TrapperKeeper," a tool that stores the results of
OPUS/UrbanSim model estimations into a MySQL database. This tool was
developed
use on the UrbanSim implementation used in the dynamic land use and
transportation model of Chittenden County, Vermont, at the Spatial
Analysis
Lab and University Transportation Center of The University of Vermont.

Before we began estimating the various sub-models of the land use
model, we
recognized the potential for confusion that could ensue if we had no
way of
tracking and managing the hundreds of estimations we would likely be
performing.
TrapperKeeper was designed and implemented to make it easy to keep
track of
these many model estimations.

This document covers the following aspects of the TapperKeeper:
- Overview
- Data
- Integrating with UrbanSim

2. Overview

TrapperKeeper does its work by:

- parsing the plain text log files generated by UrbanSim estimation
procedures.
- writing the data into a MySQL database.

The models included with UrbanSim can be classified into two broad
categories:

- regression model
- logit model

The estimation attributes and thus the estimation output of each of
these
categories differs. Thus, TrapperKeeper uses a parser and a data
store specific
to each category.

The data are parsed and stored in the MySQL database by Python code
that is
called at some point after an UrbanSim model estimation is run (see
"Integrating
with UrbanSim" below for an example of how to integrate TrapperKeeper
with
UrbanSim).

Estimation data stored in TrapperKeeper's database can later be
referred to using
the MySQL command line client, via a Microsoft Access database that
links to
MySQL using something like ODBC (see "databse/TrapperKeeper.mdb" for
an example of
such a databsae).

3. Data

The TrapperKeeper database is fully described in a file called
"schema.sql", which
is located in the "database" directory). This file can be used to
create the
tables required by TrapperKeeper in a MySQL database using the mysql
command line
client (type "\." from the mysql client prompt for more information).

For each category of model--regression or logit--TrapperKeeper stores
the
attributes of an estimation run in two tables--one for attributes of
the model
estimation at-large and one for the variables of the model. For
example, the
"estimation_regression" table contains one entry for each estimation
run of any
regression-based model, and the "estimation_regression_variables"
table contains
one entry for each variable estimated in a given estimation run of a
regression-
based model.

There is a one-to-many relationship between the "estimation_
regression" table and the "estimation_regression_variables" table--
that is, a
given row in "estimation_regression" can be associated with one or
more rows in
"estimation_regression_variables" (however a given row in
"estimation_regression_variables" can be associated with one and only
one row in
"estimation_regression"). The same holds true for the
"estimation_logit" and
"estimation_logit_variables" tables which are used to store
estimation results for
logit-based models. The relationship between an estimation run and
its variables
is recorded in the "estimation_<MODEL_TYPE>_id" attribute of either
the logit- or
regression-based variable table (where "<MODEL_TYPE>" is one of
"logit" or
"regression").

In addition to the parsed attributes, TrapperKeeper stores the entire
unparsed
plain text log file, in the "raw_log" attribute", for each entry in the
"estimation_regression" and "estimation_logit" tables.

The data model of TrapperKeeper currently supports a number of
attributes that
are not yet used by TrapperKeeper. These include:

- "aiC": A double for storing Akaike information criterion
- plotN: A blob, where N is one of [1-8], for storing graphical plots
of various
attributes of the estimation (e.g. R^2)

These attributes exist because we hope to use R-based estimation
procedures and
to add support for this to TrapperKeeper.

To experiment with storing plots in the database, see the Python program
"testwriteblob2.py" as well as the example data in the "data" directory.

4. Integrating with UrbanSim

The "run_estimation.py" file in the "examples" directory contains
Python code that
demontrates how to run TrapperKeeper's parser and "receipt writer"
components
after an UrbanSim estimation runner. This is done after the call to
"EstimationRunner().run_estimation()".

TrapperKeeper is configured via an estimation configuration, an
example of which
can be found in "examples/estimation_config_services.py". The
following is a
summary of the configuration key/value pairs in this file that
pertain to
TrapperKeeper:

- receipt_configuration: this dictionary specifies the MySQL database
to which
TrapperKeeper will write estimation results.

- base_year: the base year of the model for which the estimation is
being made.
Used to determine the filename of the log file of the estimation
written by the
estimator.

- cache_directory: the directory where the estimator will write the
log file of
the estimation run
Received on Wed Oct 31 2007 - 11:25:53 PDT