Thank you, Jesse and Hana.
We did't try to do tricks with memory. Instead we decreased the number of observations to 10000 in the households_for_estimation table. It's working.
Mark
--- Original Message ---
From: Hana Sevcikova <hana_at_cs.washington.edu>
To: Mark Kryvobokov <k_mark_at_ukr.net>
Date: 31 january, 20:23:33
Subject: Re: [UrbanSim-Users] Memory error in HLCM estimation
Mark,
You need to use considerably less households for estimation. The
estimation process is done on a 3D array of float numbers of the size
number_of_households x 30 x number_of_variables
We usually use 5000 to 10000 agents.
Hana
Mark Kryvobokov wrote:
>
>
> Hello,
>
>
>
> I receive a memory error while trying to run estimation for HLCM. We
> have data on 662249 households, but no data on residential units. I
> experimented with different numbers of residential units, considering
> the ratios of households-to-residential units in each gridcell as 1:1,
> 1:2, 1:3 and even 1:1000 and 3:1. The result is the same memory error:
>
>
>
> Start simulation run: started on Thu Jan 31 16:11:26 2008
>
> random seed = (10, 10)
>
> Starting simulation for year 1980: started on Thu Jan 31 16:11:26 2008
>
> Closing log file: /tmp/opus\run_multiprocess.log
>
> Logging to file: /tmp/opus\year_1980_log.txt
>
> Simulate year 1980: started on Thu Jan 31 16:11:26 2008
>
> Running Household Relocation Model (from
> urbansim.models.agent_relocation_model): started on Thu Jan 31 16:11:26
> 2008
>
> compute_probabilities ...
>
> compute_choices ...
>
> Number of movers: 394660
>
> Running Household Relocation Model (from
> urbansim.models.agent_relocation_model): completed...8.9 sec
>
> Unplace 32888 agents.
>
> Estimating Household Location Choice Model (from
> urbansim.models.household_location_choice_model): started on Thu Jan 31
> 16:11:38 2008
>
> Number of agents for estimation: 662249
>
> Sampling locations for estimation ...
>
> Sampling done in 1 chunk(s).
>
> Choice set size: 30
>
> Compute variables ...
>
> urbansim.gridcell.ln_residential_units...................0.0 sec
>
> urbansim.gridcell.travel_time_to_CBD.....................0.0 sec
>
> Estimate ...
>
> submodel: -2
>
> Estimating Household Location Choice Model (from
> urbansim.models.household_location_choice_model): completed...1 min, 7.5
> sec
>
> Simulate year 1980: completed............................1 min, 19.8 sec
>
> Closing log file: /tmp/opus\year_1980_log.txt
>
> Traceback (most recent call last):
>
> Starting simulation for year 1980: completed.................1 min, 19.9
> sec
>
> Start simulation run: completed..................................1 min,
> 19.9 sec
>
> Closing log file: /tmp/opus\run_multiprocess.log
>
> _ File "C:\MyNew\Opus41\lyonprototype41\estimation\run_estimation.py",
> line 69, in <module> _
>
> estimator.estimate()
>
> _ File "C:\MyNew\Opus41\opus-4.1\urbansim\estimation\estimator.py", line
> 67, in estimate _
>
> self.model_system.run(self.config,
> write_datasets_to_cache_at_end_of_year=False)
>
> _ File
> "C:\MyNew\Opus41\opus-4.1\urbansim\model_coordinators\model_system.py",
> line 137, in run _
>
> write_datasets_to_cache_at_end_of_year=write_datasets_to_cache_at_end_of_year)
>
>
> _ File
> "C:\MyNew\Opus41\opus-4.1\urbansim\model_coordinators\model_system.py",
> line 363, in _run_year _
>
> self.vardict[outputvar] = self.do_process(locals())
>
> _ File
> "C:\MyNew\Opus41\opus-4.1\urbansim\model_coordinators\model_system.py",
> line 441, in do_process _
>
> return eval(ev)
>
> File "<string>", line 1, in <module>
>
> _ File "C:\MyNew\Opus41\opus-4.1\opus_core\model.py", line 60, in
> logged_estimate_method _
>
> results = estimate_method(*req_args, **opt_args)
>
> _ File
> "C:\MyNew\Opus41\opus-4.1\urbansim\models\location_choice_model.py",
> line 195, in estimate _
>
> agents_index, procedure, estimate_config=estimate_config,
> debuglevel=debuglevel)
>
> _ File "C:\MyNew\Opus41\opus-4.1\opus_core\choice_model.py", line 311,
> in estimate _
>
> result = self.estimate_step()
>
> _ File "C:\MyNew\Opus41\opus-4.1\opus_core\choice_model.py", line 339,
> in estimate_step _
>
> result[submodel] = self.estimate_submodel(self.get_all_data(submodel),
> submodel)
>
> _ File "C:\MyNew\Opus41\opus-4.1\opus_core\choice_model.py", line 366,
> in estimate_submodel _
>
> return self.procedure.run(data, upc_sequence=self.upc_sequence,
> resources=self.estimate_config)
>
> _ File "C:\MyNew\Opus41\opus-4.1\opus_core\configurable.py", line 38, in
> config_run_method _
>
> results = run_method(*req_args, **opt_args)
>
> _ File "C:\MyNew\Opus41\opus-4.1\opus_core\bhhh_mnl_estimation.py", line
> 54, in run _
>
> l_2=self.mnl_loglikelihood(data, b2, depm).sum()
>
> _ File "C:\MyNew\Opus41\opus-4.1\opus_core\bhhh_mnl_estimation.py", line
> 115, in mnl_loglikelihood _
>
> self.upc_sequence.compute_utilities(data, b, self.resources)
>
> _ File "C:\MyNew\Opus41\opus-4.1\opus_core\upc_sequence.py", line 58, in
> compute_utilities _
>
> self.utilities = self.utility_class.run(data, coefficients, resources)
>
> _ File "C:\MyNew\Opus41\opus-4.1\opus_core\configurable.py", line 38, in
> config_run_method _
>
> results = run_method(*req_args, **opt_args)
>
> _ File "C:\MyNew\Opus41\opus-4.1\opus_core\linear_utilities.py", line
> 37, in run _
>
> return sum(utility,axis=2)
>
> _ File "C:\Python25\Lib\site-packages\numpy\core\fromnumeric.py", line
> 866, in sum _
>
> return sum(axis, dtype, out)
>
> MemoryError
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Users mailing list
> Users_at_urbansim.org
> http://www.urbansim.org/mailman/listinfo/users
Received on Fri Feb 01 2008 - 02:18:50 PST
This archive was generated by hypermail 2.2.0 : Fri Feb 01 2008 - 02:18:52 PST