== How to evaluate Sonifications and Sonic Interaction Designs? == This webpage collects information about the evaluation of SID. This refers both to the design process and the evaluation of the created products or systems that make use of sonic interactions. See also the bibliography section for references to this topic. Bibliography (see also http://trac.sme-ccppd.org/SID/wiki/WG4Sonification/Bibliography) ---- * Ferey N, Nelson J., Martin C., Picinali L., Bouyer G., Tek A., Bourdot P., Burkhardt J. M., Katz B. F. G., Ammi M., Etchebest C., Autin L., « Multisensory VR interaction for protein-docking in the CoRSAIRe project », Virtual Reality (2009) 13, pp. 273–293, 2009. Ergonomic analysis of the protein-protein current docking task. Design of an immersive and multimodal application in Virtual Reality (VR) in order to manipulate two proteins and explore possible docking solutions. During the exploration, visual, audio and haptic feedbacks are combined to render physical or chemical properties of the current docking configuration. Audio rendering uses the « parameter mapping » sonification technique and adds spatialization components. The parameters concerns the proteins mutual interaction :[[BR]] - surface complementarity : each protein’s level of complentarity controls parameters of a granular synthesis process – using the french word « complémentaire »; the more compatible is the protein, the more intelligible is the sound.[[BR]] - hotspot at the interface : the percentage of hotspots controls a low-pass filter cutoff frequency for the two streams generated in the previous process (also rendered stereophonically); the cutoff frequency increases with the number of hotspots present at the interface.[[BR]] - surface collision : the global number of collisions controls phase modulation parameters (carrier and modulation) of a sinusoidal wave; the modulation increases as the collision score gets higher.[[BR]] - electrostatic energy of the complex : the force controls pitch and timbre parameters of two sounds generated by an additive synthesis approach; the « better » this force value is, the more coincident and harmonic the two sounds are.[[BR]] - Van der Waals energy of the complex : the force controls the beating between two sounds frequentially close; the « better » this force value is, the lower the beating is. A multimodal VR-environment is developped and implemented. Global assesment of the project is carried out through an iterative approach of design and evaluation; ongoing evaluations mainly aims at two goals :[[BR]] 1- to evaluate the virtual environment in terms of usability and usefulness, and more formaly the relevance of a concept for the design of work tools.[[BR]] 2- to evaluate the interface in terms of interaction techniques and multimodal representation of information.[[BR]] The first goal – evaluating prototype – requires « real users » (task experts) in a « real world » (realistic scenarios) whereas the second goal – evaluating modal allocation and metaphors’ intelligibility – requires less representative users and contexts. Along the project evaluations have been formative and included in a trial/error loop of product development and quality improvment. At the end of the project, evaluation is considered as summative as it is referred to external norms and performance criteria. ---- * Lemaitre G., Houix O., Visell Y., Franinovic K., Misdariis N., Susini P. « Toward the design and evaluation of continuous sound in tangible interfaces: the Spinotron », International Journal of Human-Computer Studies, 67 (2009), special issue on Sonic Interaction Design (SID), pp. 976–993, 2009. ---- * Martins A. C. G., Rangayyan R. M., « Experimental evaluation of auditory display and sonification of textured images », Proceedings of the 1997 International Conference on Auditory Display (ICAD), Palo Alto, California, USA, November 2-5 1997. Experimental validation of the potential of proposed auditory display and sonification methods for aural analysis of textured images. Analogy is drawn between speech and texture synthesis, as the result of a convolution of an impulse field with a basic wavelet.[[BR]] Mapping between the projections of the image at various angles (Radon transforms or integrals) and audible signals played in sequence. Two kind of textures are considered and respectively linked to unvoiced and voiced speech synthesis :[[BR]] - random, as a result of convolving a random noise filed with a « spot » : spectral envelopes of the projections are related to the filter spot characteristics.[[BR]] - (quasi-)periodic, where basic texture elements – « texton » – are repeated over the image field : the Auditory Display provides timber and pitch related to the texton and the periodicity; in addition, another procedure is based on the cepstral decomposition of the image and a voiced-speech-like encoding approach of the texton and the vertical/horizontal periodicities extracted by the analysis (see [Rangayyan R.M., Martins A.C.G., Ruschioni R.A.,« Aural analysis of image texture via cepstral filtering and sonification », Proc. SPIE: Visual Data Exploration and Analysis III, vol. 2656, pp. 283-294, San Jose, CA, Jan. 1996.], for more details). Evaluation experiments are conducted to test the sonification methods for both textures :[[BR]] - 10 with random texture, to verify the capacity of :[[BR]] . classifying random texture according to the shape of the spot[[BR]] . ordering sounds derived from iamges wth regards to the spot shape[[BR]] . associating a textured image with the sound generated[[BR]] - 5 with periodic texture, to verify the capacity of :[[BR]] . the mapping functions to have a natural association with the image[[BR]] . the mapping functions to allow the ordering of the images with regards to variations in their parameters [[BR]] In these experiments, different parameters of the image processing are tested on the basis of performance tests for a « natural » or logical association between sound and image ; more precisely, differents task groups are defined:[[BR]] - for random texture:[[BR]] . identification by sound of the smooth/sharp nature of the spot[[BR]] . preservation of size/scale ordering[[BR]] . subjective association[[BR]] . discrimination of size[[BR]] - for periodic texture:[[BR]] . subjective association when varying and combining the analysis dimension (horizontal/vertical spacing, size/shape of the texton. The experimental framework is developped and diffused on World-Wide Web : subjects can conduct the experiments on their own via a web browser. No training phase is implemented before the main part of the test. On the whole, the results show a large variation of performance scores among the groups (from 100% to 10%, depending on situations and parameters tested). It is suggested that with a specific training, the results can be further improved. ---- * Nesbitt K. V., Barass S., « Evaluation of a multimodal sonification and visualisation of depth of market stock data », Proceedings of the 2002 International Conference on Auditory Display (ICAD), Kyoto, Japan, July 2-5, 2002. Sonification of depth market stock data, in complement with visual presentation of the data. The challenge for traders is to make prediction and decision concerning the evolution of the market with regards to extra context and history in the display. Currently, traders read depth of market stock data from numeric tables, trying to observe and extract regular patterns in their movements. The study focus on both visual/audio display separately and on their mutual interactions in a multimodal display ; it considers two basic movements: upward and downard. The visualisation is designed with a series of surface formed by the volume of bids, asks and trades at ecah price ; a color chart is complementing the device (bids=yellow, asks=green). The visual metaphor used is stated to be « ecological » and is related to the notion of landscape (hill, valley, river, etc.). The sonification is designed at two levels :[[BR]] - schema level : using the « market place » metaphor (vendors shouting, etc.), by considering this experience as familiar for traders; it is implemented with a male voice saying « buy » (pitched at E5) and a female voice saying « sell » (pitched at G5).[[BR]] - perceptual level : mapping information on perceptually scaled auditory variables; it is implented by defining two notions « price importance » (categorized importance of bid/ask relative to the last trade) and « volume importance » (categorized importance of the volume data relative to the last trade) and assigning them respecitvely to Pitch categories (C5 to B5) for « price » and Loudness categories (low ; mid, high) for « volume ». Morover, a spatialisation parameter is also used for the diffusion of bid and ask price importance.[[BR]] The data are updated every 0.5 seconds, the sounds are scheduled at 0.01 second intervals with a maximum of 50 new entries for each frame. Evaluation is done on the 3 configurations (Visual, Audio, Multimodal) with 15 non-expert subjects. The data used was recorded from a real trading data. The experiment is designed to answer basically 4 questions :[[BR]] - capacity to predict the direction of the next trade in V, A or M configuration ?[[BR]] - difference of performance with V, A or M display ?[[BR]] - capacity to extract consistent patterns in the data ?[[BR]] - capacity to interpret and make decision from V, A or M display ? Main results show that :[[BR]] the 3 configurations (V, A, M) allow significant predictions (better than chance),[[BR]] downward movements are better predicted than upward movements,[[BR]] the mutimodal display is redundant for downward and complementary for upward movements,[[BR]] sonification provides recent trends while visualisation provides context informations.[[BR]] ---- * Noirhomme-Fraiture M., Schöller O., Demoulin C., Simoff S. J., « Complementing visual data mining with the sound dimension: sonification of time dependent data », S.J. Simoff et al. (Eds.): Visual Data Mining, LNCS 4404, pp. 236–247, 2008. Extension of visual data mining : sonification of 2D/3D time series data. Issue of sonification complementarity or alternative to visually displayed graphs.[[BR]] The sound design is assumed by an automated generation of the sound patterns, four methods are considered and combined :[[BR]] . pitch-based mapping on pre-processed data (detection of oultiers and smoothing)[[BR]] . pitch-based mapping on extreme values detection [[BR]] . beat drums mapping on the curve shape (represented by its first derivative)[[BR]] . stereo panning for diffusion. The evaluation part is splitted in two parts and is done either on 2D or 3D data ; in the latter case (3D), a slicing method of the initial surface is used, following 3 canonical directions (horizontal, vertical and diagonal) :[[BR]] - analysis of an evaluation questionnaire ; it contains questions for evaluation of the sonification methods and overall approach together with demographic/skills data about the participant (musical background, etc.) and general informations on data mining techniques.[[BR]] - assessment and evaluation of sound sequences via :[[BR]] . specific questions on the understanding of the data : detection of global/local trends and extreme values or identification of type of trends.[[BR]] . subjective preference on sonifications options (MIDI instrument for pitch or drum mapping, sonification technique)[[BR]] Experiments are conducted on a dedicated website (the subject needs to specify the number of time he had listen to the sound before answering). In terms of performance scores (correct/wrong answers), results are globally encouraging despite some issues outlined : this specific sonification approach does not support very well scaling information (needed to identify and localise a particular point on the graph) ; sonification of 3D graphs seems to be globally worse than in the case of 2D. ---- * Pedersen M., Muhlberger R., « Invisible territory: sonifying the game of Go », Proceedings of the 2005 Australasian Computer Music Conference (ACM), Brisbane, Australy, July, 2005. Sonification for the game of Go (Weiqi, in chinese) for use of an interactive « musification » of the playing. The approach is a mapping from data to sound that preserve symbolism and aesthetics of the game, as well as being as comprehensible as possible. Different reasons to investigate the game of Go in the light of Human-Computer Interaction (HMI) field :[[BR]] - needs for recognition of patterns of play,[[BR]] - large database of computer-readable records of Go games,[[BR]] - complex memorisation of board positions due to large feature space of the game,[[BR]] - pattern-based nature of the game suitable to the linkage with audio/musical motifs.[[BR]] In that case, the sonification of Go described is not for « Play » but for « Replay » where a visual representation of the game’s progress is used for the rational, evaluative cognitive processing by the viewer. For this work, functional and aesthetics requirements are stated, such as : notion of collaborative creativity and ethics, concepts attached to Chinese philosophy (harmony, yin/yang, etc. ), metaphysical origins (« celestial and earthly space »). Transposed to more operational elements these aspects intuitively leads to spatialisation, dualistic tension but also harmony and unity. Two modes of sonification are proposed :[[BR]] - passively to follow the replay of the game,[[BR]] - interactively to allow the viewer to provide audio input as well as listening to the audio output.[[BR]] The sonic interface is a bridge between the « replay » requirements (advancing the game) and the sonification of the game performance, building an audio narrative related to the game positions. Sonification is based on the Shakuhachi, an end blown wind instrument with a distinctly Japanese sound. With regards to the requirements previousy defined, methods for designing the sonification uses the following elements : - time/space : the game grid is taken as the sound stage, each audio event is placed in the position corresponding to the related game event ; a reverberation parameter is also used to figure out the notion of a liberties attached to each board position.[[BR]] - symmetry/duality : symmetry is mapped to timbral parameter of the source (same note, etc.), duality is mapped to pitch (black moves lower than white ones, etc.).[[BR]] - pattern : the repeatedly looping over the last 10 moves assumes the representation of the game patterns (in a lower range than the current moves).[[BR]] The interactive sonification mode of the Go game becomes a « musification » since elements of the mapping between data to sound are modified by a « player » playing a performance. The evaluation is based on the use of the method in its interactive form (« musification »). It is globally informal, on the basis of two main criteria :[[BR]] - how does the performance reflect the symbolism and aesthetics of Go and give also a « valid representation » of the game ?[[BR]] - how does the performance capture attention and convey meaning for both the human performer and the audience ?[[BR]] Others informal observations taken from trial performances tends to confirm the relevancy of the spatialisation component (sensation of being « inside » the game) and the pitch component (comprehension of the game development). The choice of the Shakuhachi is also perceived to be a « natural » association with the Go game. ---- * Rath M., Schleicher D.R., « On the relevance of auditory feedback for quality of control in a balancing task », Acta Acustica united with Acustica 94, 12–20, 2008. ---- * Shelley S., Alonso M., Hollowood J., Pettitt M., Sharples S., Hermes D., Kohlrausch A., « Interactive sonification of curve shape and curvature data », M.E. Altinsoy, U. Jekosch, and S. Brewster (Eds.): HAID 2009, LNCS 5763, pp. 51–60, 2009. ---- * Vezien J. M., Menelas B., Nelson J., Picinali L., Bourdot P., Ammi M., Katz B. F. G., Burkhardt J. M., Pastur L., Lusseyran F., « Multisensory VR exploration for computer fluid dynamics in the CoRSAIRe project », Virtual Reality (2009) 13, pp. 257–271, 2009. ---- * Watson M., Sanderson P., « Sonification supports eyes-free respiratory monitoring and task time-sharing », Human Factors, Vol. 46, No 3, Fall 2004, pp. 497-517, 2004. ---- * Zhao H., Smith B. K., Norman K., Plaisant C., Shneiderman B., « Listening to maps: user evaluation of interactive sonifications of geo-referenced data », IEEE Transactions on Multimedia, http://en.scientificcommons.org/43365038, 2008. ---- * ''(new item)''