Migrating from UrbanSim 3 to UrbanSim 4

Migrating from UrbanSim 3 to UrbanSim 4

This document describes the steps necessary to migrate an UrbanSim 3 application to work with UrbanSim 4.

If you find any omissions or errors, please report them to bugs at urbansim dot org.  

General Comments

Be careful to use the same version of the documentation as the code, since the documentation and code continue to change daily.

UrbanSim 4 does not have a consistency checker.  In general, the consistency constraints of UrbanSim 3 still apply to UrbanSim 4, unless the relevant database table columns have changed in their semantics (see the list of table changes, below).

The "configurations" mentioned in the manual are similar to the "scenario file" used in UrbanSim 3, though they use different technology.  Both specify how to setup or configure the simulation.  The UrbanSim 4 configuration allows you to configure much more of the simulation than was possible with the scenario file. 

Do not have NULL values in any database column used by UrbanSim 4.  If you do, UrbanSim 4 will fail in confusing ways.

The set of variables used by a model is determined completely by the model's specification.  Thus, it is impossible for our User Manual to document the set of variables used by a model.

The set of variable names implemented in UrbanSim 4's urbansim package is different than those that came with UrbanSim 3.  Some UrbanSim 3 variables have no direct counter-part in UrbanSim 4, and UrbanSim 4 contains many variables not found in UrbanSim 3.

UrbanSim 4 makes it clearer that the set of appropriate accessibility variables depends upon the region being modeled, and the particular implementation of the region's travel model.  Thus, in UrbanSim 4, the accessibility variables are packaged in the psrc package that is used to specialize UrbanSim to run on the Puget Sound Regional Council's metropolitan area.

The use of development type is much less prominent in UrbanSim 4, since the developer model now is a location choice model where projects (agents) choose from gridcells (locations).  

User lower-case for the name of any Python package containing a module defining an Opus variable.  For instance, the Python package 'gridcell' within urbansim must be lower-case, so that Opus can find the Opus variables defined within that package, e.g. 'urbansim.gridcell.acres_of_land'.

We recommend using lower-case for the name of all Python packages.

Use lower-case for any Opus variable names.

Changes to Specific Baseyear Database Tables

This section describes the specific changes needed to be made to the tables in the baseyear database.

accessibilities

• This table is no longer used in UrbanSim 4.  Instead, the data are drawn directly from the travel_data table.

accessibilities_input

• This table is no longer used in UrbanSim 4.  Instead, the data are drawn directly from the travel_data table.

annual_employment_control_totals

• No longer uses column 'total_employment'.

• See the MySQL script annual_employment_control_totals.txt that was used to convert the Eugene 1980 Baseyear database to the UrbanSim4 format

developer_model_coefficients
developer_model_specification

• The developer model as it existed in Java is no longer in UrbanSim 4.  Instead, there is a family of development project location choice models, one for each type of building, that determine where to place new development projects.  The set of development projects to build is computed by randomly sampling from the historical set of development projects until enough development has been scheduled for the current year.  For more details, see the Users Manual.

development_constraints

• The 'plan_type_x' field (that could contain multiple plan types) has been replaced with the 'plan_type_id' field (that contains either a -1 for "don't care" or a single plan type id).  

• The 'devtype_x' field is no longer used.

• This table now has six additional columns, specifying the development constraint for gridcells matching the rest of the field values (except for the constraint_id).

  • min_units

  • max_units

  • min_commercial_sqft

  • max_commercial_sqft

  • min_industrial_sqft

  • max_industrial_sqft
• The conversion process for this table is a very manual process. It would be difficult if not impossible to write a script to do this. In general though, this is what we did to convert the UrbanSim v3 table to v4. The development_constraints, development_types, and plan_types tables were all exported to an Excel document for ease of manipulation. I sorted the development_constraints table by plantype_x then devtype_x. Then for each unique plantype_x, I wrote down which devtype_x numbers were not accounted for. This represented for each plantype_x what devtype's were allowed. I made a new development_constraints table with the appropriate format (see above), and entered for each plan_type_id all of the appropriate numbers in each field (various commercial/industrial sqft and residential unit min/max numbers). This took care of constraining land conversions based on zoning. The next issue was constraining land conversions based on locational/environmental attributes. In the Eugene case it was easy, if a cell was in a wetland, outside the urban growth boundary, in a stream buffer, on a steep slope, or in a floodplain, no development was allowed, so a zero went in each field. Your application may be substantially more complicated than this because various combinations of zoning and environmental attributes may be combined. You may have 2 or more records for a plan_type_id in a case where you may constrain differently depending on combinations of attributes.

development_events_exogenous

This table replaces the development_events table used in UrbanSim 3.  It uses the same schema, except that all of the improvement_value fields must contain 'A' (for adding units or sqft).  

This name change was done since the development_events dataset will contain both prescheduled development events from the development_events_exogenous table as well as development events created by the sequence of development_project_transition model (to create new development projects; not events), the *_development_project_location_choice_models (to place those projects), the development_event_transition_model (to convert the development projects into development events), and the events_coordinator (to process those newly created events).

development_event_history

• Removed the following fields, since they were not being used:
  • commercial_improvement_value_change_type
  • industrial_improvement_value_change_type
  • residential_improvement_value_change_type
  • governmental_improvement_value_change_type
  • fraction_residential_land_value_change_type
  • residential_units_change_type
  • commercial_sqft_change_type
  • industrial_sqft_change_type
  • governmental_sqft_change_type
  • development_type_change_type
• See the MySQL script development_event_history.txt that was used to convert the Eugene 1980 Baseyear database to the UrbanSim4 format

development_types

• The "primary_use_id" column is no longer used, and thus may be dropped from this table.

employment_adhoc_sector_groups

• The "elc_sector" and "scalable_sector" are no longer used to determine which employment location choice model to use for choosing the location for a job.  Thus, they may be deleted from that table. Instead, the model to use for a job is determined as follows:

  • Use the Non-Home-Based Employment Location Choice Model for non-home-based jobs whose employment sector is listed in the non_home_based_employment_location_choice_model_specification table.

  • Use the Home-Based Employment Location Choice Model for home-based jobs whose employment sector is listed in the home_based_employment_location_choice_model_specification table.

  • Use the Scaling Procedure for Jobs Model for all other jobs.

gridcell

• Add three new columns, containing reasonable, positive (greater than zero) values for each record:

  • commercial_sqft_per_job

  • industrial_sqft_per_job

  • governmental_sqft_per_job
• See the MySQL script gridcells.txt that was used to convert the Eugene 1980 Baseyear database to the UrbanSim4 format

household_characteristics_for_ht

• Rename all instances of characteristic “age of head” to be “age_of_head”.  In general, the values of the characteristics field must not have any spaces or tabs.

• The column 'total_number_of_households' is no longer used.

• See the MySQL script household_characteristics_for_ht.txt that was used to convert the Eugene 1980 Baseyear database to the UrbanSim4 format

jobs

• Add a new integer 'building_type' column defining the type of space used by this job.  Currently, the set of allowable values are hard-coded in the code (we plan to move these into the configuration at some point) as follows:

  • 1 = commercial sqft

  • 2 = governmental sqft

  • 3 = industrial sqft

  • 4 = residential unit
• See the MySQL script jobs.txt that was used to convert the Eugene 1980 Baseyear database to the UrbanSim4 format

jobs_for_estimation

• UrbanSim 4 needs three of these tables, one for each of the types of employment location choice model.  Each of these three tables contains only jobs to use for estimating the given type of jobs.

  • jobs_for_estimation_commercial

  • jobs_for_estimation_home_based

  • jobs_for_estimation_industrial

model_variables

• This table is no longer used in UrbanSim 4.  Instead, the variables used by a model are those listed in the model's specification and coefficient tables.

sampling_rates

• This table is no longer used in UrbanSim 4.

urbansim_constants

The following fields are no longer used in UrbanSim 4:

• developer_model_estimation_threshold_count

• logit_choice_set_size_for_estimation

• number_of_developer_model_history_years - instead, the development_event_history table is "unrolled" for every year for which it has data.

• max_persons_per_household_for_control_totals

zones

• Add the column ’faz_id’ to the zones table, indicating the Forecast Analysis Zone (FAZ) for each zone.

zones_in_faz

• This table is no longer used in UrbanSim 4.  Instead, UrbanSim 4 uses the faz_id column in the zones table.

Specification and Coefficients Tables

In UrbanSim 4, the specification and coefficient tables, e.g. land_price_model_specification and land_price_model_coefficients, are intermediary tables produced by the running the integrated estimation process.  Thus, if you need to update the set of variables used by a model, or you need to re-estimate the coefficients, you should do this by re-running the estimation process; you probably should never directly edit the specification or coefficients tables in the database. 

Database Tables no Longer Used

The following tables were used by UrbanSim 3 but are no longer used by UrbanSim 4:

• household_characteristics_for_hlc
• residential_units_for_home_based_jobs
• sqft_for_non_home_based_jobs
• transition_types
• See the MySQL script general_table_cleanup.txt that was used to convert the Eugene 1980 Baseyear database to the UrbanSim4 format