Evaluation Design
Mixed Methods
The evaluation in the staged test setting is conducted with Mixed Methods, which consists of three different approaches:
- The test facility was equipped with spatial sensors (position, gaze direction, acceleration, distance, contact, etc.) and the garment worn by the test subjects with a biofeedback sensor (skin conductance, GSR) to record somatic, psychophysiological responses. They allow the behavior of the test subjects to be quantitatively evaluated and categorized. This methodological approach corresponds to the «epistemology of sensor systems».
- Since the closed rooms did not allow direct observation, they were equipped with cameras. The recorded videos were displayed together with the biofeedback sensor measurements in real time in the self-developed data visualization software «EvaluationViewer». It allows the researchers to make a qualitative, ethnographic observation and heuristic assessment of the subjects› behavior in different situations and contexts.
- Following the test run, subjects were invited for an interview to capture the subjective experience. The interviews consisted of a qualitative interview and a quantitative questionnaire (semantic differentials). They were audio recorded and transcribed for analysis in the data visualization software «EvaluationViewer». It was also used during the interview to analyze critical situations together with the test persons.
Through precise staging of interactive situations and the help of spatial sensors, it is possible to quantify different behavioral patterns. E.g. in ubiComb3: Did she pick up the phone? (no=0, yes=1). Did he sit down? (no=0, yes=1, immediately=2). Did she interact with the desk drawers? (no=0, yes=1, extensive=2). In the different rooms, quantitative statements about the behavior of the test subjects can thus be derived:
ubiComb1: How does she enter the comb (insecure or decisive)? In which sector does he stay most often (interest, context)? How does she react to the interactive animation (level of participation, acceleration)? Does he take an apple (is he exploratory, understands the interaction)? How does she react to the disruptive change of atmosphere on the video wall? Does he go through the fly curtain to ubiComb2 before the timeout?
ubiComb2: How does she enter the comb (insecure or decisive)? Does he activate the body scans or try to escape (acceptance or refusal)? Does she experiment with the trapezoidal fields (passive, analytical, playful)? How does he respond to the body scans (attention span)? How quickly does he enter ubiComb3 when the door opens?
ubiComb3: How does she enter the comb (insecure or decisive)? Does he pick up the phone? Does she listen to the whole sequence on the telephone (attention span, calmness)? Does he sit down on the chair (involvement)? Does she activate the gadgets (telephone, drawers, typewriter, radio, flashcard box, wastebasket)? Does he write something (participation)? Does she try to go through the door to ubiComb4 before the timeout?
ubiComb4: How does she enter the comb (insecure or decisive)? When does he enter the stage (hesitance)? Does she interact with the microphone? Does he try to go through the revolving door to ubiComb5 prematurely?
ubiComb5: How does she enter the comb (insecure or decisive)? Does he interact with the creatures (which ones)? Does he sit down (involvement)? Does she exit before the timeout?
Through the combination of spatial sensors and biofeedback sensors it is possible to identify psychophysiological reactions in the context of the staged situations. To interpret emotions and sense-making processes, the biofeedback sensor values are therefore measured in the spatio-temporal context of events and activities. E.g. in ubiCom1: In the period of 12 seconds after the first apple is taken out of the apple bowl, high, medium and low GSR values are counted with a histogram. The GSR measurements can also be interpreted heuristically in relation to the synchronized video recording.
Research focus
Three pretests were conducted with 37 subjects to examine the intensity of the user experience, basic understanding of the staged content, navigation and interaction, timing, and other specific factors. Subsequently, evaluations were conducted with 50 subjects from different user groups (taking into account age, gender, and background). The focus was on the following areas of investigation:
- technological perception of the human in the environment: «epistemology of sensor systems»
- human perception of the various staged paradigms of the UbiComp: experience, emotion, state of mind
- human sense-making process in an unfamiliar, technologically enhanced environment: anticipation, interpretation, judgment, appropriation, narrative
- human computer interaction: human perception of agency shifts between humans and machines, and affordances and valences of things and spatial arrangements
- identification of human behavior patterns and attitudes
- comparison and triangulation of qualitative and quantitative data: What can the sensor system capture (technical), what can be observed via the cameras (as objective as possible, ethnographic), and what via the interviews (subjective, ethnographic)?
- design principles for techno-socially hybrid environments (seamlessness – seamfullness)
The findings and conclusions are presented and published at peer-reviewed international conferences.