Re: [UrbanSim-Users] my_estimation_config (P.S.)

Siva,
The 'attrtype' may not be the real reason of crashing. It seems to me that
your model won't estimate (probably due to, e.g., multicollinearity, as Hana
speculated), but the code still tries to write the estimation results to
cache. I'd suggest you start with a very simple specification with 1 or 2
variables and see how it goes. At the mean time, in run_estimation.py you
can set "save_estimation_results=False" and should get rid of the 'attrtype'
error.

Liming.

On Wed, Mar 5, 2008 at 5:43 PM, sivakarthik <sivakarthik_at_mail.utexas.edu>
wrote:

> Hi Hana,
> I've made the changes you asked me to make but I am getting the
> following error message:
> C:\opus\eugene\estimation>run_estimation.py
> Cache Directory set to: C:/opus/threecounty/2000/flatfile
> Start simulation run: started on Wed Mar 05 16:31:49 2008
> random seed = 10
> Starting simulation for year 2000: started on Wed Mar 05 16:31:49 2008
> Closing log file:
> C:/opus/threecounty/2000/flatfile\run_multiprocess.log
>
> Logging to file:
> C:/opus/threecounty/2000/flatfile\year_2000_log.txt
> Simulate year 2000: started on Wed Mar 05 16:31:49 2008
> Running Household Relocation Model (from
> urbansim.models.agent_reloc
> ation_model): started on Wed Mar 05 16:31:49 2008
> compute_probabilities ...
> compute_choices ...
> Number of movers: 927
> Running Household Relocation Model (from
> urbansim.models.agent_reloc
> ation_model): completed...0.5 sec
> Unplace 77 agents.
> Estimating Household Location Choice Model (from
> urbansim.models.hou
> sehold_location_choice_model): started on Wed Mar 05 16:31:49 2008
> Number of agents for estimation: 6944
> Sampling locations for estimation ...
> Sampling done in 1 chunk(s).
> Choice set size: 30
> Compute variables ...
> urbansim.household_x_gridcell.cost_to_income_ratio
> urbansim.gridcell.total_annual_rent
>
> urbansim.gridcell.total_residential_value........0.0
> sec
>
> urbansim.gridcell.total_annual_rent:completed.......0.1
> sec
>
> urbansim.household_x_gridcell.cost_to_income_ratio:
> completed...
> 0.1 sec
>
> urbansim.household_x_gridcell.income_and_year_built......0.0
> sec
>
>
> urbansim.household_x_gridcell.percent_high_income_households_wit
> hin_walking_distance_if_low_income
>
> urbansim.gridcell.percent_high_income_households_within_walk
> ing_distance
>
> urbansim.gridcell.number_of_high_income_households_withi
> n_walking_distance
>
> urbansim.gridcell.number_of_high_income_households
>
> urbansim.household.is_high_income........0.0
> sec
>
>
> urbansim.gridcell.number_of_high_income_households:
> completed...0.1 sec
>
> urbansim.gridcell.number_of_high_income_households_withi
> n_walking_distance: completed...0.4 sec
>
> urbansim.gridcell.number_of_households_within_walking_di
> stance
>
> urbansim.gridcell.number_of_households.......0.1
> sec
>
>
> urbansim.gridcell.number_of_households_within_walking_di
> stance: completed...0.4 sec
>
> urbansim.gridcell.percent_high_income_households_within_walk
> ing_distance: completed...0.9 sec
>
> urbansim.household.is_low_income.....................0.0
> sec
>
>
> urbansim.household_x_gridcell.percent_high_income_households_wit
> hin_walking_distance_if_low_income: completed...0.9 sec
>
> urbansim.household_x_gridcell.percent_low_income_households_with
> in_walking_distance_if_high_income
>
> urbansim.gridcell.percent_low_income_households_within_walki
> ng_distance
>
> urbansim.gridcell.number_of_low_income_households_within
> _walking_distance
>
> urbansim.gridcell.number_of_low_income_households...
> .0.1 sec
>
> urbansim.gridcell.number_of_low_income_households_within
> _walking_distance: completed...0.4 sec
>
> urbansim.gridcell.percent_low_income_households_within_walki
> ng_distance: completed...0.5 sec
>
> urbansim.household_x_gridcell.percent_low_income_households_with
> in_walking_distance_if_high_income: completed...0.5 sec
>
> urbansim.household_x_gridcell.percent_minority_households_within
> _walking_distance_if_minority
>
> urbansim.gridcell.percent_minority_households_within_walking
> _distance
>
> urbansim.gridcell.number_of_minority_households_within_w
> alking_distance
> urbansim.gridcell.number_of_minority_households
>
> urbansim.household.is_minority...........0.0
> sec
>
>
> urbansim.gridcell.number_of_minority_households:
> com
> pleted...0.1 sec
>
> urbansim.gridcell.number_of_minority_households_within_w
> alking_distance: completed...0.4 sec
>
> urbansim.gridcell.percent_minority_households_within_walking
> _distance: completed...0.5 sec
>
> urbansim.household_x_gridcell.percent_minority_households_within
> _walking_distance_if_minority: completed...0.5 sec
>
> urbansim.household_x_gridcell.residential_units_when_household_h
> as_children
>
> urbansim.household.is_without_children...............0.0
> sec
>
>
> urbansim.household_x_gridcell.residential_units_when_household_h
> as_children: completed...0.0 sec
>
> urbansim.household_x_gridcell.young_household_in_high_density_re
> sidential
>
> urbansim.gridcell.is_in_development_type_group_SSS...0.0
> sec
>
>
> urbansim.household.is_young..........................0.0
> sec
>
>
> urbansim.household_x_gridcell.young_household_in_high_density_re
> sidential: completed...0.1 sec
> urbansim.household_x_gridcell.young_household_in_mixed_use
>
> urbansim.gridcell.is_in_development_type_group_SSS...0.0
> sec
>
> urbansim.household_x_gridcell.young_household_in_mixed_use:
> comp
> leted...0.1 sec
> Estimate ...
> submodel: -2
> WARNING: Estimation led to singular matrix. No results.
> Estimating Household Location Choice Model (from
> urbansim.models.hou
> sehold_location_choice_model): completed...4.4 sec
> Simulate year 2000:
> completed....................................5.1
> sec
>
> Closing log file:
> C:/opus/threecounty/2000/flatfile\year_2000_log.txt
> Starting simulation for year 2000:
> completed.........................5.2
> sec
>
> Start simulation run:
> completed..........................................5.2
> sec
>
> Closing log file: C:/opus/threecounty/2000/flatfile\run_multiprocess.log
> Traceback (most recent call last):
> File "C:\opus\eugene\estimation\run_estimation.py", line 31, in ?
> estimator.estimate()
> File "c:\opus\urbansim\estimation\estimator.py", line 71, in estimate
> self.save_results(out_storage=out_storage)
> File "c:\opus\urbansim\estimation\estimator.py", line 147, in
> save_results
>
> self.specification.write
> (out_storage=AttributeCache(),out_table_name=specifi
> cation_table)
> File "c:\opus\opus_core\equation_specification.py", line 152, in write
> self.out_storage.write_dataset(local_resources)
> File "c:\opus\opus_core\store\old\attribute_cache.py", line 62, in
> write_datas
> et
> attrtype = write_resources['attrtype']
> File "c:\opus\opus_core\resources.py", line 30, in __getitem__
> return SessionConfiguration().get_dataset_from_pool(key)
> File "c:\opus\opus_core\session_configuration.py", line 113, in
> get_dataset_fr
> om_pool
> return self.get_dataset_pool().get_dataset(dataset_name)
> File "c:\opus\opus_core\dataset_pool.py", line 56, in get_dataset
> self._load_new_dataset(dataset_name, dataset_arguments)
> File "c:\opus\opus_core\dataset_pool.py", line 129, in _load_new_dataset
> raise Exception("Dataset '%s' not found in any of the "
> Exception: Dataset 'attrtype' not found in any of the packages:
> 'urbansim',
> 'opus_core'.
>
>
> When I tried looking for attrtype in the urbansim.org. It is mentioned
> that
> attrtype is a dictionary giving the type for each attribute. Should this
> be
> user specified? Can you please help me with this error. Also you have
> mentioned that my simulation was not successful earlier because a singular
> matrix has been formed. So, does this mean that there is an error in the
> explanatory variables I have chosen to estimate HLCM? I am referring to
> The multi-collinearity problem.
>
> Thanks and Regards,
> Siva
>
>
>
>
>
>
>
>
>
>
>
>
>
> -----Original Message-----
> From: Hana Sevcikova [mailto:hana_at_cs.washington.edu]
> Sent: Monday, March 03, 2008 1:55 PM
> To: sivakarthik
> Cc: users_at_urbansim.org
> Subject: Re: my_estimation_config (P.S.)
>
> Siva, you've got an unwanted line break there when pasting. Check the
> syntax.
>
> Hana
>
> sivakarthik wrote:
> > Hi Hana,
> > I've made the changes you asked me but I am receiving a syntax
> > error.
> >
> > C:\opus\eugene\estimation>python run_estimation.py
> > Traceback (most recent call last):
> > File "run_estimation.py", line 16, in ?
> > from urbansim.estimation.estimation_runner import EstimationRunner
> > File "c:\opus\urbansim\estimation\estimation_runner.py", line 15, in ?
> > from urbansim.estimation.estimator import Estimator
> > File "c:\opus\urbansim\estimation\estimator.py", line 144
> > self.specification.write(out_storage=Attribute Cache(),
> > out_table_name=speci
> > fication_table)
> > ^
> > SyntaxError: invalid syntax
> >
> > Thanks and Regards,
> > Siva
> >
> >
> > -----Original Message-----
> > From: Hana Sevcikova [mailto:hana_at_cs.washington.edu]
> > Sent: Monday, March 03, 2008 12:44 PM
> > To: sivakarthik
> > Subject: Re: my_estimation_config (P.S.)
> >
> > Siva, I just saw, you need to do a little bit more changes:
> >
> > - remove the output_configuration entry from your
> > my_estimation_config.py file
> >
> > - in the 'save_results' method replace the line:
> >
> > raise StandardError, "No output_configuration given."
> >
> > by:
> >
> > pass
> >
> > Let me know if it works,
> >
> > Hana
> >
> >
> > Hana Sevcikova wrote:
> >> Siva,
> >>
> >> You're right - right now the code requires SQL in order to store your
> >> estimation results. But there is no reason for doing so and I will
> >> change it. For now, please make the following changes in the file
> >> urbansim/estimation/estimator.py:
> >>
> >> Comment out the last three lines in the method 'save_results':
> >>
> >>
> >
>
> #self.specification.write(out_storage=out_storage,out_table_name=specificati
> > on_table)
> >>
> >
>
> #self.coefficients.write(out_storage=out_storage,out_table_name=coefficients
> > _table)
> >>
> >
>
> #self.cache_specification_and_coefficients(out_storage,specification_table,c
> > oefficients_table)
> >>
> >> and add these two lines:
> >>
> >>
> >
> self.specification.write
> (out_storage=AttributeCache(),out_table_name=specifi
> > cation_table)
> >>
> >
> self.coefficients.write
> (out_storage=AttributeCache(),out_table_name=coeffici
> > ents_table)
> >>
> >> This will only write results into your cache without accessing the
> >> database. And I will go ahead and make this as an option in the
> > repository.
> >> BTW, your estimation was not successful, since it led to a singular
> >> matrix. Once it is successful, you will see the estimated coefficients
> >> in the output. Check values of your predictors, if some of them has the
> >> same value for all observations.
> >>
> >> Hana
> >>
> >>
> >>
> >> sivakarthik wrote:
> >>> Hello,
> >>>
> >>> From the output below I feel that my estimation was
> >>> successful but I am unable to print the output of it. I am not using
> >>> SQL for running UrbanSim. In my_estimation_config file in Eugeneà
> >>> estimation folder the output_configuration is linked with SQL
> >>> database. I want to know if there is any solution to get the output to
> >>> a particular folder without using SQL database.
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> C:\opus\eugene\estimation>run_estimation.py
> >>>
> >>> Cache Directory set to: C:/opus/threecounty/2000/flatfile
> >>>
> >>> Start simulation run: started on Sat Mar 01 21:45:47 2008
> >>>
> >>> random seed = 10
> >>>
> >>> Starting simulation for year 2000: started on Sat Mar 01 21:45:47
> >>> 2008
> >>>
> >>> Closing log file:
> >>> C:/opus/threecounty/2000/flatfile\run_multiprocess.log
> >>>
> >>>
> >>>
> >>> Logging to file:
> >>> C:/opus/threecounty/2000/flatfile\year_2000_log.txt
> >>>
> >>> Simulate year 2000: started on Sat Mar 01 21:45:47 2008
> >>>
> >>> Running Household Relocation Model (from
> >>> urbansim.models.agent_reloc
> >>>
> >>> ation_model): started on Sat Mar 01 21:45:47 2008
> >>>
> >>> compute_probabilities ...
> >>>
> >>> compute_choices ...
> >>>
> >>> Number of movers: 927
> >>>
> >>> Running Household Relocation Model (from
> >>> urbansim.models.agent_reloc
> >>>
> >>> ation_model): completed...0.6 sec
> >>>
> >>> Unplace 77 agents.
> >>>
> >>> Estimating Household Location Choice Model (from
> >>> urbansim.models.hou
> >>>
> >>> sehold_location_choice_model): started on Sat Mar 01 21:45:48 2008
> >>>
> >>> Number of agents for estimation: 6944
> >>>
> >>> Sampling locations for estimation ...
> >>>
> >>> Sampling done in 1 chunk(s).
> >>>
> >>> Choice set size: 30
> >>>
> >>> Compute variables ...
> >>>
> >>> urbansim.household_x_gridcell.cost_to_income_ratio
> >>>
> >>> urbansim.gridcell.total_annual_rent
> >>>
> >>>
> >>> urbansim.gridcell.total_residential_value........0.0 sec
> >>>
> >>>
> >>>
> >>> urbansim.gridcell.total_annual_rent:
> >>> completed.......0.1 sec
> >>>
> >>>
> >>>
> >>> urbansim.household_x_gridcell.cost_to_income_ratio:
> >>> completed...
> >>>
> >>> 0.1 sec
> >>>
> >>>
> >>> urbansim.household_x_gridcell.income_and_year_built......0.0 sec
> >>>
> >>>
> >>>
> >>>
> >>> urbansim.household_x_gridcell.percent_high_income_households_wit
> >>>
> >>> hin_walking_distance_if_low_income
> >>>
> >>>
> >>> urbansim.gridcell.percent_high_income_households_within_walk
> >>>
> >>> ing_distance
> >>>
> >>>
> >>> urbansim.gridcell.number_of_high_income_households_withi
> >>>
> >>> n_walking_distance
> >>>
> >>>
> >>> urbansim.gridcell.number_of_high_income_households
> >>>
> >>>
> >>> urbansim.household.is_high_income........0.0 sec
> >>>
> >>>
> >>>
> >>>
> >>> urbansim.gridcell.number_of_high_income_households:
> >>>
> >>> completed...0.1 sec
> >>>
> >>>
> >>> urbansim.gridcell.number_of_high_income_households_withi
> >>>
> >>> n_walking_distance: completed...0.4 sec
> >>>
> >>>
> >>> urbansim.gridcell.number_of_households_within_walking_di
> >>>
> >>> stance
> >>>
> >>>
> >>> urbansim.gridcell.number_of_households.......0.1 sec
> >>>
> >>>
> >>>
> >>>
> >>> urbansim.gridcell.number_of_households_within_walking_di
> >>>
> >>> stance: completed...0.4 sec
> >>>
> >>>
> >>> urbansim.gridcell.percent_high_income_households_within_walk
> >>>
> >>> ing_distance: completed...0.9 sec
> >>>
> >>>
> >>> urbansim.household.is_low_income.....................0.0 sec
> >>>
> >>>
> >>>
> >>>
> >>> urbansim.household_x_gridcell.percent_high_income_households_wit
> >>>
> >>> hin_walking_distance_if_low_income: completed...1.0 sec
> >>>
> >>>
> >>> urbansim.household_x_gridcell.percent_low_income_households_with
> >>>
> >>> in_walking_distance_if_high_income
> >>>
> >>>
> >>> urbansim.gridcell.percent_low_income_households_within_walki
> >>>
> >>> ng_distance
> >>>
> >>>
> >>> urbansim.gridcell.number_of_low_income_households_within
> >>>
> >>> _walking_distance
> >>>
> >>>
> >>> urbansim.gridcell.number_of_low_income_households...
> >>>
> >>> .0.1 sec
> >>>
> >>>
> >>> urbansim.gridcell.number_of_low_income_households_within
> >>>
> >>> _walking_distance: completed...0.4 sec
> >>>
> >>>
> >>> urbansim.gridcell.percent_low_income_households_within_walki
> >>>
> >>> ng_distance: completed...0.5 sec
> >>>
> >>>
> >>> urbansim.household_x_gridcell.percent_low_income_households_with
> >>>
> >>> in_walking_distance_if_high_income: completed...0.5 sec
> >>>
> >>>
> >>> urbansim.household_x_gridcell.percent_minority_households_within
> >>>
> >>> _walking_distance_if_minority
> >>>
> >>>
> >>> urbansim.gridcell.percent_minority_households_within_walking
> >>>
> >>> _distance
> >>>
> >>>
> >>> urbansim.gridcell.number_of_minority_households_within_w
> >>>
> >>> alking_distance
> >>>
> >>>
> >>> urbansim.gridcell.number_of_minority_households
> >>>
> >>>
> >>> urbansim.household.is_minority...........0.0 sec
> >>>
> >>>
> >>>
> >>>
> >>> urbansim.gridcell.number_of_minority_households: com
> >>>
> >>> pleted...0.1 sec
> >>>
> >>>
> >>> urbansim.gridcell.number_of_minority_households_within_w
> >>>
> >>> alking_distance: completed...0.4 sec
> >>>
> >>>
> >>> urbansim.gridcell.percent_minority_households_within_walking
> >>>
> >>> _distance: completed...0.5 sec
> >>>
> >>>
> >>> urbansim.household_x_gridcell.percent_minority_households_within
> >>>
> >>> _walking_distance_if_minority: completed...0.5 sec
> >>>
> >>>
> >>> urbansim.household_x_gridcell.residential_units_when_household_h
> >>>
> >>> as_children
> >>>
> >>>
> >>> urbansim.household.is_without_children...............0.0 sec
> >>>
> >>>
> >>>
> >>>
> >>> urbansim.household_x_gridcell.residential_units_when_household_h
> >>>
> >>> as_children: completed...0.0 sec
> >>>
> >>>
> >>> urbansim.household_x_gridcell.young_household_in_high_density_re
> >>>
> >>> sidential
> >>>
> >>>
> >>> urbansim.gridcell.is_in_development_type_group_SSS...0.0 sec
> >>>
> >>>
> >>>
> >>>
> >>> urbansim.household.is_young..........................0.0 sec
> >>>
> >>>
> >>>
> >>>
> >>> urbansim.household_x_gridcell.young_household_in_high_density_re
> >>>
> >>> sidential: completed...0.1 sec
> >>>
> >>>
> >>> urbansim.household_x_gridcell.young_household_in_mixed_use
> >>>
> >>>
> >>> urbansim.gridcell.is_in_development_type_group_SSS...0.0 sec
> >>>
> >>>
> >>>
> >>>
> >>> urbansim.household_x_gridcell.young_household_in_mixed_use: comp
> >>>
> >>> leted...0.1 sec
> >>>
> >>> Estimate ...
> >>>
> >>> submodel: -2
> >>>
> >>> WARNING: Estimation led to singular matrix. No
> results.
> >>>
> >>> Estimating Household Location Choice Model (from
> >>> urbansim.models.hou
> >>>
> >>> sehold_location_choice_model): completed...4.5 sec
> >>>
> >>> Simulate year 2000:
> >>> completed....................................5.5 sec
> >>>
> >>>
> >>>
> >>> Closing log file:
> >>> C:/opus/threecounty/2000/flatfile\year_2000_log.txt
> >>>
> >>> Starting simulation for year 2000:
> >>> completed.........................5.5 sec
> >>>
> >>>
> >>>
> >>> Start simulation run:
> >>> completed..........................................5.5 sec
> >>>
> >>>
> >>>
> >>> Closing log file:
> C:/opus/threecounty/2000/flatfile\run_multiprocess.log
> >>>
> >>> Traceback (most recent call last):
> >>>
> >>> File "C:\opus\eugene\estimation\run_estimation.py", line 31, in ?
> >>>
> >>> estimator.estimate()
> >>>
> >>> File "c:\opus\urbansim\estimation\estimator.py", line 71, in
> estimate
> >>>
> >>> self.save_results(out_storage=out_storage)
> >>>
> >>> File "c:\opus\urbansim\estimation\estimator.py", line 122, in
> >>> save_results
> >>>
> >>> db_server = MysqlDatabaseServer(self.config
> ["output_configuration"])
> >>>
> >>> File "c:\opus\opus_core\store\mysql_database_server.py", line 39, in
> >>> __init__
> >>>
> >>> raise MySqlDbImportException(e)
> >>>
> >>> opus_core.exception.mysqldb_import_exception.MySqlDbImportException:
> >>> Unable to i
> >>>
> >>> mport the MySQLdb package, which is the connection between the MySql
> >>> database an
> >>>
> >>> d Python. The underlying exception was: No module named MySQLdb
> >>>
> >>>
> >>>
> >>> Thanks and Regards,
> >>>
> >>> Siva
> >>>
> _______________________________________________
> Users mailing list
> Users_at_urbansim.org
> http://www.urbansim.org/mailman/listinfo/users
>
Received on Tue Mar 11 2008 - 12:51:20 PDT