Variable Dependencies Tree

Opus implements a mechanism of computing variables structured in a dependencies tree. A variable is only computed if its children have changed since the last computation. Thus, a dataset object keeps a version information about each attribute or variable and each variable class keeps information about which versions of the dependent variables this variable was computed with. This mechanism is automatically invoked by calling the method compute_variables() of the class Dataset.

Figure: Example of a variable dependencies tree. The arrows indicate the dependency direction: $a \longrightarrow b$ means '$a$ is dependent on $b$ '.

As an example, consider Figure 24.1. There are 8 variables/attributes, belonging to 3 datasets, ds1, ds2, and ds3. The arrows show the dependency hierarchy between variables. The hierarchy is defined in the method dependencies() of each variable by listing all direct 'children' variables as expressions, fully-qualified names or dataset-qualified names (in this example the names in the parentheses in the figure). Note that two attributes here are primary, namely ``ds3.var_d'' and ``ds2.var_e'', and thus are defined in their dataset-qualified name (there is no variable implementation for those attributes). For example, the dependencies() method of the root variable ``pkg.ds1.my_variable'' returns a list of three elements: ['pkg.ds1.var_a', 'pkg.ds1.var_b', 'pkg.ds2.var_b'], and dependencies() of the variable ``pkg.ds1.var_a'' returns a list with one expression: ['var_c = log(ds3.var_d)/log(pkg.ds3.var_e)']. As described in Section 24.3.4, expressions do not have user-defined implementations (they are created on the fly). Therefore there is no need to define dependencies for the variable ``ds3.var_c'', Opus determines them automaticaly.

Now suppose we have a system of several models, three of which are invoking the computation of ``my_variable'' of dataset ds1. Suppose also that initially we have created the three datasets and loaded the two primary attributes. Opus assigns to each newly created attribute a version number 0 in its attribute box. This situation is shown in the left upper corner of Figure 24.2. Box number 1 shows that there are only two attributes in the system, both with version number 0.

Figure: Scenario of computing and recomputing variables.

Box 2 shows a situation when the first model invokes the computation of ``my_variable'' . The system works through the defined dependencies and computes all variables needed to compute ``my_variable''. In this process, again, each newly computed variable gets the version number 0 (illustrated by the small boxes above each variable), stored in the attribute box of each variable. Additionally, each Variable instance keeps the version number for each dependent variable, on which this variable was computed. In the figure, this is illustrated by the small boxes bellow each variable. Note that all elements in Figure 24.2 that are created or change their values are shaded.

The box number 3 shows a situation in which another model changes values of variable ``var_e'' of the dataset ds2 which causes an increment of the version number. Therefore, when the next model invokes compute_variables("my_variable"), Opus determines that there is a mismatch between versions of ``ds.var_e'' on which variables ``ds1.var_b'' and ``ds2.var_b'' were computed and its current version (the mismatch is visualized by red crosses). Then all variables above ``ds.var_e'' are recomputed, their version numbers incremented, and the version numbers in the dependencies lists are updated (box 4).

A similar situation arises when another model changes the variable ``ds3.var_e'' (box 5). The next call of compute_variables("my_variable") causes a recomputing of 4 variables (box 6) In the right lower corner of Figure 24.2, the state of the three datasets is shown at the end of the run of our example models.

Note that the dependencies tree is constructed using the dependencies() method of Variable. Therefore a missing item in this method translates to a missing branch of the tree and thus failing to determine a need for recomputing.