Spatial setup and Sensor Actuator System

The research facility is built in the form of 5 honeycombs and an anteroom. The basic structure consists of an aluminum-steel framework, which can be assembled modularly and is covered with different materials depending on the requirements. The dimensions are 8.70 meters long, 6.50 meters wide and 2.50 meters high. The installation is divided into an experience area for the test persons (also referred to as subjects) and a backstage area for technical control and evaluation. Based on the conception of content and interaction (see Journey/Design) the «Main Engine» was developed with the visual programming software Max/MSP/Jitter. Visitors are guided through the ubiCombs in a time-based manner and are prompted to interact and to make sense in staged scenes. 

The technical basis for the installation is a sensor actuator system: with the help of various sensor techniques, the behavior of the subjects is recorded, interpreted and forwarded to the responsive environment in near real time. The system reacts to the human actions with physically experienceable actuators such as light, sound or automation. The spatial tracking system Pozyx, mounted in the corners above the installation, allows to capture the position, velocity, head movement and direction of view of the subjects. In addition, various sensors are integrated into the scenes in the ubiCombs (e.g. capacitive, touch, distance) and interpreted according to the situation. Cameras are installed in all ubiCombs to allow video recording and real-time observation of situations not captured by sensors. They offer an additional way of  user behavior interpretation through human observation. Using a digital «behavior checklist«, these key moments of behavior are identified, quantified and transferred to the system by hand. Similarly, semantic differentials are used in the subsequent interviews, the quantified data of which can also be transferred to the digital system and thus triangulate qualitative data sets. A Galvanic Skin Response (GSR) sensor in the form of a finger ring is integrated into the garment that the subjects put on for the evaluation. It measures the skin conductivity and thereby psychophysiological reactions of the subjects during the entire run.

In addition to activating the responsive environment, the sensory measurements are also used for evaluation and data analysis. They are stored, categorized and evaluated rule-based or with the help of artificial intelligence. In the Evaluation Viewer, the processed data is visualized and can already be viewed during the tour and used for the subsequent interview.

The following section outlines technical aspects of the various installations and events, as well as the means and ruleset for parametric measurement and data interpretation. 

The Garment:

During the tour, the subjects wear a garment that is equipped with various sensors and actuators. The integrated headband contains the tracking pad and an accelerometer, which are required for spatial tracking. The integrated GSR finger ring provides the biofeedback data, which is transmitted wirelessly (Flora + xBee) to the system together with the spatial tracking data in approximate real time. Physical experiences are provided by the integrated fiber optics, LEDs and vibrators.

For behavioral analysis, the following means are used to measure parameter changes:

• Garment > spatial tracking and timer: activity (position, velocity, head movement, direction of view), dwell time.
• Comb sectors > spatial tracking and timer: spatial position and dwell time in the combs: presence, in center, at edge, close to an interactive object
• Biofeedback finger ring > GSR is measured during the entire experience.

ubiComb1: «Zen Garden»

Two interactive stations are set up in ubiComb1: a video wall and an apple bowl. The video wall is a rear projection that invites visitors to play. Using a camouflaged surveillance camera, subjects are recorded, their appearance aesthetically alienated and projected from behind onto a textile wall of the comb. The apple bowl stands on a pedestal equipped with a scale. The interactive station can thus measure when subjects take apples out of the bowl or put them back in.

For behavioral analysis, the following means are used to measure parameter changes:

• Entire comb > spatial tracking and timer: activity (position, velocity, head movement, direction of view), dwell time.
• Comb sectors > spatial tracking and timer: position and dwell time in context: in front of entrance, in center, directed towards video wall, directed towards apple bowl, directed towards exit.
• Station “video wall” > spatial tracking and timer: level of activity (acceleration and position)
• Station “apple bowl” > activation of balance (weight): number of apples removed, number of changes.
• Event “disturbance” > spatial tracking, timer, GSR: change of behavior after change of video projection (acceleration and position).
• Event “teaser from ubiComb2”: dwell time in ubiComb1 after teaser (acceleration and position).
• Biofeedback finger ring > GSR is measured during the entire experience. 
• Video recording > manual behavior checklist: interaction with tea set and yoga mat (no/ yes/extensive).

ubiComb2: «SciFi»

In the second ubiComb, an interactive floor and hand scanning panels embedded in fake doors invite interaction (see interaction concept). The floor consists of seven floor panels equipped with large-area pressure sensors. Interactivity and feedback are signaled by framing LED strips. Upon entering, assigned video sequences are played on a video wall (rear projection). If the video sequences are watched in full length, the touch sensitive hand scanning panels are activated (iPads). After correct interaction with all interfaces or after a defined time span, an automatic sliding door opens with the help of a stepper motor.

For behavioral analysis, the following means are used to measure parameter changes:

• Entire comb > spatial tracking and timer: activity (position, velocity, head movement, direction of view), dwell time.
• Floor panels > dwell time per panel, number of changes.

• Station “fake doors” > activation of hand scanner, number of attempts.

• Station “sliding door” > activation of hand scanner, number of attempts, time before expiration of defined time span.
• Event “teaser from ubiComb3”: dwell time after teaser (acceleration and position).
• Biofeedback finger ring > GSR is measured during the entire experience. 
• Video recording > manual behavior checklist: interaction game understood? (no/yes/fast).

ubiComb3: «Office Anachronism»

This comb is non-interactive except for the telephone and only played in a temporal succession. First the phone rings (original bell), then the floor lamp turns on, after hanging up the phone (phone speaker plays voice) the desk lamp turns on and typewriter sounds become audible (integrated speaker). Also the radio is activated (original loudspeaker) and sounds of fictitious neighbors become audible (speakers from neighboring combs). Finally, the scene shuts down and the test subject is lured into the fourth comb with a blinking «On Air» sign above the exit door.

For behavioral analysis, the following means are used to measure parameter changes:

• Entire comb > spatial tracking and timer: activity (position, velocity, head movement, direction of view), dwell time.
• Station “telephone” > activation through pick up, duration until hang up (contact)

• Station “door to comb4” > activation time before end of scheduled time (activation floor ubiComb4).

• Event “On Air”: dwell time until opening of door after activation of teaser (activation floor ubiComb4).
• Biofeedback finger ring > GSR is measured during the entire experience. 
• Video recording > manual behavior checklist: interaction with (telephone), typewriter, radio, card index box, lamps, drawers, waste bin (no/ yes/extensive).

ubiComb 4: «Echo Chamber»

The fourth comb invites the subject to enter an interactive stage (equipped with pressure sensors). The scene is animated with lights and speakers play a background beat. A stage microphone invites to sing or speak. When stepping on stage, the curtains are automatically pulled back (solenoid, stepper motors) and a hall of mirrors emerges. The microphone functions as a sensor to determine if the person is participating and records the sounds. If the person dares to sing or speak into the microphone, the sounds get woven into the background beat. After a set period of time, the scene shuts down and a revolving door opens automatically, inviting you to step into the next comb.

• Entire comb > spatial tracking and timer: activity (position, velocity, head movement, direction of view), dwell time.
• Comb sectors > spatial tracking and timer: position and dwell time in context: in front of entrance, in center, directed towards video wall, directed towards apple bowl, directed towards exit.
• Station “stage” > activation floor, activation microphone, recording microphone
• Event “teaser from ubiComb5” > dwell time in ubiComb4 after teaser (acceleration and position).
• Biofeedback finger ring > GSR is measured during the entire experience. 
• Video recording > manual behavior checklist: interaction with curtains, looking for escape (no/ yes/extensive).

ubiComb 5: «Mycelium»

The last ubiComb is enriched with interactive robotic mechanisms: textile anemones (capacitive sensors), pneumatically extendable anemones (with a silent compressor), worms (servo motors mounted on top of each other), hanging tentacles and cross-flow fans. UV light and fiber optics are used for illumination. A sound collage (ambient speakers) runs in the background. The intensity of movements, colours of light and sounds reacts to the movement and touch of the test person. The illumination of the garment (fiber optics) is also animated interactively.

For behavioral analysis, the following means are used to measure parameter changes:

• Entire comb > spatial tracking and timer: activity (position, velocity, head movement, direction of view), dwell time.
• Comb sectors > spatial tracking and timer: position and dwell time in context: in front of entrance, in center, directed towards video wall, directed towards apple bowl, directed towards exit.
• Station “tentacles” > accelerometer: acceleration
• Station “textile anemones” > capacitive textile: touch
• Event “teaser from exit”: dwell time in ubiComb5 after shut down (acceleration and position).
• Biofeedback finger ring > GSR is measured during the entire experience. 
• Video recording > manual behavior checklist: interaction with anemones, worms, tentacles, (no/ yes/extensive).