Eugene Database
From ExternalWiki
This page details our working plan and notes for completing a distributable Eugene database for users to experiment and learn with.
Contents |
Benchmarks for simulating Eugene
We have tried this on a variety of machines and operating systems, and found that performance varies from a low of 35 seconds for a year on a 64 bit Linux to about 7 minutes for a year on an Windows laptop.
Times per year:
- Desktop, 3Ghz 64 bit Dual-core Xeon, Linux, it runs in ~35 seconds
- MacBook Pro (2.16Ghz dual core), ~52 seconds
- Desktop, 2.8Ghz Dual-core Pentium D, Linux, ~80 seconds
- Laptop, 2.0Ghz Pentium M, Windows XP Pro -- running on AC power -- ~215 seconds
- Desktop, 3.2Ghz Pentium 4, Windows XP Pro, ~215 seconds
- Laptop 1.6GHz Pentium M, Windows XP Pro -- running on AC power -- ~242 seconds
- Laptop 1.6GHz Pentium M, Windows XP Pro -- running on battery -- ~419 seconds
Eugene database release
The Eugene release is out.
Deferred
- Re-run all estimation scripts and update the database?
- Print some kind of loading message to screen before calling up any TraitsUI GUIs.
- Make sure can create an indicator for a dataset defined in an Opus package other than urbansim.
- [Only seems to work if the dataset is also defined in urbansim, e.g. psrc.zone.population_per_acre. -- Daniel]
- Do we need to specify nchunks for loading some datasets?
- Do aggregate and disaggregate indicators work?
- Unroll data, if that is requested. (Or change docs to not say it unrolls???)
- Automatically check all of our indicators (or at least a large chunk of them).
- Call the DB Eugene_1980_baseyear_tutorial in MySQL
- Clean up tables: delete unnecessary tables AND fields in tables [Is this still necessary, since we are distributing the baseyear cache and it only contains the tables needed by UrbanSim?]
- Update database tables documentation (chapter 9) [Joel]
- At first in the manual, to be moved to its own document when expanded
- Get sub-GUI to appear on Mac (and check Linux). Use popen3 (http://docs.python.org/lib/os-newstreams.html#os-newstreams)?
- Allow user comments on wiki page
Done
- Create the www.urbansim.org/tutorials page?
- Remove forced dependencies on MySQLdb.
- Tutorial works with any combination of: MySQL not running/installed, or services database not there, or MYSQL* environment variables not defined.
- Test upgrade script.
- Document the opus_core/tools/gui_export tools.
- Check entire process on a clean machine (Windows, Mac, Linux)
- Fix titles in export tools to not say "baseyear", since they are more general than that.
- Add a field to set number of years to run?
- Review documentation for upgrade scripts for changing interaction set names.
- Add better help strings to all fields of all eugene/tools tools.
- Add opus_upgrade/changes_2006_09_26/readme.html to the release notes.
- Changing names of interaction sets.
- Make sure that the demo works on a machine not running MySQL.
- If output directory already exists, create different directory.
- Document upgrade script (for interaction sets) and what to do if it's not sufficient.
- Automatically build a zip file of the code for each of the "changes" directories in opus_upgrade, so that that code is always the latest.
- Make sure there are no .pyc files in opus_upgrade.
- Make sure that the eugene package zip has eugene in the path. Same for eugene_1980_baseyear_cache.
- Make sure last link on index.html is correct.
- Are assumptions in documentation? Perhaps in a Eugene document?
- Fix all www.urbansim.org links in the html documents to point into demo.zip contents.
- Are doc links to installers correct? Should the installers go to the top-level directory (versus being in "installers")? Or put all installers in the "installers" directory?
- Little GUIs for the tutorial.
- Display error message if problem creating indicator when using GUI.
- Instructions on how to stop simulation started from GUI
- Check that opus-1.0.beta2.tar.gz is in demo.zip.
- Fix demo.zip (it only has one file in it).
- How can user stop simulation? Make it easy, if that's doable. Certainly document it.
- Note in docs that first time launch GUI takes a long time.
- Fix output directory for running simulation
- Change eugene to run 5 years by default.
- Only allow indicators for years that actually exist (e.g., not for 2100).
- Visually indicate when indicator fails to be computed.
- Add validator to ensure that year value is > 0.
- Create new user tutorial documentation [David and Paul]
- Packaging everything in a single zip file. Includes quick install doc, and references to everything installed.
- Release Eugene
- Change installer-creation to also zip eugene's baseyear cache and put it on the web.
- Simplify set of docs shipped. With index.html to refer to them all.
- Ship "uber" package containing all of this stuff.
- Fix datasets to preload design flaw
- Develop tool to take cache and move it into a DB of the users choosing (MySQL, tab, or csv)
- Develop baseline configuration and test run it
- Changed wording from 'stable' to 'snapshot'
- Create a single DB with all models estimated, which can be cached and distributed as a zipped cache
- Zipping a cached DB will become the way we distribute DBs into the future
- Add eugene Opus package to nightly distribution.
- Convert tables from our v3 DB (Eugene_1980_baseyear) to our v4 DB (Eugene_1980_baseyear_opus) [Jesse]
- Estimate Employment Location Choice Model (except home-based)
- Estimate ResidentialLandShareModel
- Estimate Household Location Choice Model [Liming]
- Estimate Developer Model [Liming]
- Estimate Home-Based Employment Location Choice Model
- Estimate LandPriceModel
Historical Validation work
- To Do
- As changes are made to the tutorial DB, think about things that would not work for historical validation
- Paul says: if it seems like too much to get 1980 up and running for historical validation, look at 1994 instead
- Will we switch to the RealEstatePriceModel for this work? We will need new fields in the gridcells table for this (e.g. avg_val_per_unit_industrial, avg_val_per_unit_vacant_land)
