Evaluation Design

Mixed Methods

The evaluation in the staged test setting is conducted with Mixed Methods, which consists of three different approaches:

  1. The test facility was equipped with spatial sensors (position, gaze direction, acceleration, distance, contact, etc.) and the garment worn by the test subjects with a biofeedback sensor (skin conductance, GSR) to record somatic, psychophysiological responses. They allow the behavior of the test subjects to be quantitatively evaluated and categorized. This methodological approach corresponds to the «epistemology of sensor systems».
  2. Since the closed rooms do not allow direct observation, they were equipped with cameras. The recorded videos are displayed together with the biofeedback sensor measurements in real time in the self-developed data visualization software «EvaluationViewer». It allows the researchers to make a qualitative, ethnographic observation and heuristic assessment of the subjects› behavior in different situations and contexts.
  3. Following the test run, subjects are invited for an interview to capture the subjective experience. The interviews consist of a qualitative interview and quantitative Semantic Differentials. They are audio recorded and transcribed. The data visualization software «EvaluationViewer» is used during the interview to analyze critical situations together with the test persons.

Through precise staging of interactive situations and the help of spatial sensors, it is possible to quantify different behavioral patterns. E.g. in ubiComb3: Did she pick up the phone? (no=0, yes=1). Did he sit down? (no=0, yes=1, immediately=2). Did she interact with the desk drawers? (no=0, yes=1, extensive=2). In the different rooms, quantitative statements about the behavior of the test subjects can thus be derived:

ubiComb1: How does she enter the comb (insecure or decisive)? In which sector does he stay most often (interest, context)? How does she react to the interactive animation (level of participation, acceleration)? Does he take an apple (is he exploratory, understands the interaction)? How does she react to the disruptive change of atmosphere on the video wall? Does he go through the fly curtain to ubiComb2 before the timeout?

ubiComb2: How does she enter the comb (insecure or decisive)? Does he activate the body scans or try to escape (acceptance or refusal)? Does she experiment with the trapezoidal fields (passive, analytical, playful)? How does he respond to the body scans (attention span)? How quickly does he enter ubiComb3 when the door opens?

ubiComb3: How does she enter the comb (insecure or decisive)? Does he pick up the phone? Does she listen to the whole sequence on the telephone (attention span, calmness)? Does he sit down on the chair (involvement)? Does she activate the gadgets (telephone, drawers, typewriter, radio, flashcard box, wastebasket)? Does he write something (participation)? Does she try to go through the door to ubiComb4 before the timeout?

ubiComb4: How does she enter the comb (insecure or decisive)? When does he enter the stage (hesitance)? Does she interact with the microphone? Does he try to go through the revolving door to ubiComb5 prematurely?

ubiComb5: How does she enter the comb (insecure or decisive)? Does he interact with the creatures (which ones)? Does he sit down (involvement)? Does she exit before the timeout?

Through the combination of spatial sensors and biofeedback sensors it is possible to identify psychophysiological reactions in the context of the staged situations. To interpret emotions and sense-making processes, the biofeedback sensor values are therefore measured in the spatio-temporal context of events and activities. E.g. in ubiCom1: In the period of 12 seconds after the first apple is taken out of the apple peel, high, medium and low GSR values are counted with a histogram. The GSR measurements can also be interpreted heuristically in relation to the synchronized video recording.

Research focus

Three pretests were conducted to examine the intensity of the user experience, basic understanding of the staged content, navigation and interaction, timing, and other specific factors. Subsequent evaluations focus on the following areas of investigation:

  • technological perception of the human in the environment: «epistemology of sensor systems»
  • human perception of the various staged paradigms of the UbiComp: experience, emotion, state of mind
  • human sense-making process in an unfamiliar, technologically enhanced environment: anticipation, interpretation, judgment, appropriation, narrative
  • human computer interaction: human perception of agency shifts between humans and machines, and affordances and valences of things and spatial arrangements
  • identification of human behavior patterns and character types
  • comparison and triangulation of qualitative and quantitative data: What can the sensor system capture (technical), what can be observed via the cameras (as objective as possible, ethnographic), and what via the interviews (subjective, ethnographic)?
  • derivation of design principles for techno-socially hybrid environments

During the evaluations, different user groups (age, gender, background, …) are examined and compared.

Based on the findings, design principles for the development of techno-socially hybrid environments will be derived, in which humans are able to perceive the agency of technology and can consciously reject or use it (seamlessness – seamfullness). Likewise, conclusions will be drawn for an interaction design and software development that allows the identification of behavior patterns and human character types by means of spatial sensor systems and biofeedback sensors.