Modeling Phase Diagrams
We introduce an analysis technique called modeling phase diagram. The technique supposes to record all modeling activities throughout the creation of a process model in a log. Our technique classies the recorded modeling activities according to cognitive research; the classication, in turn, allows to visualize and analyze the process of modeling itself in a diagram. The technique has been implemented in a graphical modeling tool that logs a user’s modeling activities in the background. We conducted a modeling session with graduate students to demonstrate the feasibility of our approach.
Capturing Events of the Process of Process Modeling
In order to get a detailed picture of how process models are created, we use Cheetah Experimental Platform (CEP). CEP has been specically designed for investigating the process of process modeling in a systematic manner . In particular, we instrumented a basic process modeling editor within CEP to record each user’s interactions together with the corresponding time stamp in an event log, describing the creation of the process model step by step. When focusing on the process modeling environment, the development of process models consists of adding nodes and edges to the process model, naming or renaming these activities, and adding conditions to edges. In addition to these interactions a modeler can influence the process model’s secondary notation, e.g., by laying out the process model using move operations for nodes or by utilizing bendpoints to influence the visualization of edges. A complete overview of the possible interactions is provided in Table 1.
Analyzing the Process of Process Modeling
By capturing all of the described interactions with the modeling tool, we are able to replay a recorded modeling process at any point in time without interfering with the modeler or her problem solving eorts. This allows for observing how the process model unfolds on the modeling canvas (a demonstration of the replay function is available at here). Fig. 1 illustrates the basic idea of our technique. Fig. 1a shows several states of a typical modeling process as it can be observed during replay. Fig. 1c shows the states of a different modeling process that nonetheless results in the same model. This replay functionality of CEP allows to observe in detail how modelers create the model on the canvas. We postulate that observations made for the process of modeling at a syntactic level can be traced back to the various phases of the modeling process (cf. Cognitive Foundations). Clearly, modeling manifests in the creation of model elements.
Hence, (1) a modeling phase consists of a sequence of interactions to create or delete model elements such as activities or edges. (2) A modeler usually does not create a model in a continuous sequence of interactions. She rather pauses after several interactions to inspect the intermediate result of her modeling and to plan the next steps. Syntactically, this manifests in reduced modeling activity or even inactivity. We refer to such a phase as a comprehension phase. (3) Besides modeling and thinking, a modeler also needs to reorganize the model. Reconciliation interactions manifest in moving or renaming model elements to prepare the next modeling interactions or to support her comprehension of the model. A sequence of such interactions is a reconciliation phase.
To obtain a better understanding of the modeling process and its phases, we supplement model replay with a modeling phase diagram. Such a diagram quantitatively highlights the three phases of modeling, comprehension, and reconciliation. It primarily depicts how the size of the model (vertical axis) evolves over time (horizontal axis), as can be seen in Fig. 1b and Fig. 1d for the modeling processes in Fig. 1a and Fig. 1c, respectively. The graph partitions the user interactions into the three phases, based on the kind of interactions and their frequencies in the modeling process.
So, we can read from Fig. 1b that the modeler created the model in a straightforward series of modeling steps interrupted by periods of comprehension. The modeling process in Fig. 1d shows a different approach. After some modeling, the modeler removes parts of the created model and moves an activity to make space for some control-flow constructs, as indicated by the reconciliation phase. Then, several model elements are placed and laid out before the model is completed. Note that the resulting models are identical. Yet, the phase diagrams show significant differences between both modeling processes. This illustrates the value of analyzing the modeling process in the described manner beyond the inspection of the process models themselves.
Measuring the Process of Process Modeling
Based on the theoretical background regarding the process of process modeling, we developed an algorithm for automatically extracting modeling phase diagrams (cf. Fig. 1) from the logs created by CEP. For this purpose, the user interactions depicted in Table 1 are categorized into modeling and reconciliation interactions as listed in Table 2. Comprehension phases are determined by measuring the time when no interaction with the system is recorded. For more information on how modeling phase diagrams are created we refer to .
Modeling Phase Diagram Examples
We present two modeling processes and the corresponding modeling phase diagrams from a modeling session conducted at Eindhoven University of Technology and at Humboldt-Universitat zu Berlin in 2009. Recall that in such a diagram the horizontal axis represents time and the vertical axis the number of elements in the process model. Differences in the number of elements in the process models can be attributed to superfluous activities, missing activities or different usage of gateways among our subjects. We explicitly connect each example to the modeling phase diagram, and to observations that we obtained by replaying each of the process of process modeling in CEP.
Example 1. The modeling phase diagram of Example 1 (cf. Fig. 2) shows a rather long initial comprehension phase after which alternating comprehension and modeling phases can be observed. All modeling phases are very long and steep, i.e., much model content is added per iteration. Virtually no reconciliation can be observed.
Modeling Style. Replaying the modeling process in CEP shows that the modeler appears to have a clear conception of the model to be created. Elements are placed on the canvas in large chunks, while all elements are being placed to appropriate positions so that no movement of elements is required.
Modeling Result. The created process model moderately approximates the expected modeling outcome in terms of graph edit distance (cf. ), due to some superfluous activities. The created model is, however, free of syntax errors and behavioral anomalies, such as deadlocks.
Example 2. The modeling phase diagram of Example 2 is depicted in Fig. 2. This process starts very similar to Example 1 by adding model elements in large chunks after average comprehension phases. At around 800s, the process starts to deviate by a very long comprehension phase. After this phase, modeling continues similarly to the beginning of the process until a large part of the model is removed (falling iteration around 1800s). The modeling process completes in iterations with significantly longer comprehension phases, short modeling phases, and some time spent on reconciliation.
Modeling Style. The replay shows that the modeler started modeling with a clear idea of the model to be created in mind. However, the modeler used some BPMN modeling elements wrongly, i.e., start events as intermediate states. At about 2/3 of the model created, the modeler realizes the mistake, removes all intermediate states, inserts missing gateways and arcs, and completes the model.
Modeling Result. The model shows an above average similarity to the expected modeling result in terms of graph edit distance. While the model is syntactically correct, it contains two deadlocks due to a wrong pairing of gateways.
 Pinggera, J., Zugal, S., Weber, B.: Investigating the process of process modeling with cheetah experimental platform. In: Proc. ER-POIS ’10. (2010) 13–18.
 J. Pinggera, S. Zugal, M. Weidlich, D. Fahland, B. Weber, J. Mendling, H. A. Reijers: Tracing the Process of Process Modeling with Modeling Phase Diagrams. In: Proc. ER-BPM ’11 (accepted for publication), 2011.
 Dijkman, R., Dumas, M., Garca-Banuelos, L.: Graph Matching Algorithms for Business Process Model Similarity Search. In: Proc. BPM ’09. (2009) 48–63.