A device that combines music, colour, and movement for therapeutic purposes
Cristina G. Pedraz/DICYT Can we hear colours or see sounds? Most of us can't, but synesthetes have this ability together with other capacities. It is estimated that the number of different types of synesthesia is very high, or what amounts to the same thing, some people can experience sensations in a certain sense when another is stimulated. Noted creators such as the painter Wassily Kandinsky and the composer Alexander Scriabin have perceived this crossover of senses (in their case chromesthesia or the association of sounds with colours). But can a person who is not a synesthete experience synesthesia?
This is one of the basic ideas of E-MOCOMU (E-MOtion, COlour and MUsic), a prototype that has been developed by the music and technology PhD student from the University of Valladolid Elena Partesotti. Its objective is to make use of the possibilities of the three dimensions of colour, sound, and movement for therapeutic purposes in order to improve, for example, communication in autistic children.
But this is not the only possibility of this device as it also has pedagogical and creative applications. “E-mocomu allows the user to manipulate sounds, images, and colours by means of their free movement in space. This means that if I move an arm in space there will be sounds that correspond to this movement and also colours and images that I will be able to see on a screen. This technology allows its application in the field of physical rehabilitation, facilitating the carrying out of routine exercises, and also in the field of psychology as an aid to the treatment of various disorders. Moreover, it allows an initial approximation to didactics and pedagogy of music, movement, and creative development, which as we know is vital for the self-expression of an individual at a personal and also artistic level”, explains Partesotti, stressing the importance of expressiveness in therapy.
The idea of developing this prototype arose from the academic and personal experience of this researcher of Italian origin. After studying a biennial master's degree in music therapy at the Universidad Autónoma de Madrid and working in public hospitals and schools, in 2012 she entered the University of Valladolid to study a PhD under Professor Alicia Peñalba and the Professor of the University of Campinas in Brazil, Jônatas Manzolli.
In her opinion, combining creation and technology is a good formula for helping people with psychological and communicative problems and is also a leisure tool for many. “Music therapy affords benefits that can be scientifically measured. For example, in recent years several neuroscience studies have established the connection and the power of music in physiological recovery. Neural correlations due to music modify the activity of a brain structure that is operating abnormally, which is directly relevant to the therapeutic treatment of patients suffering from depression. Moreover, as music helps to strengthen the immune system it also contributes towards the reduction of the factors that accelerate the disease. This being so, why not make use of the creative dimension in a technology?”
According to the researcher, the importance of this union between art and technology is highly relevant in the clinical-therapeutic field. “In music therapy traditional instruments are used and this works very well. Technology is not yet widely used, but we are in the digital era and if a session of music therapy is organised with children, for example, all of them know how to use it and they enjoy it because they can play with it. Technology is a good support and it works”, she emphasises.
Development of the prototype
The development of the first prototype of E-MOCOMU was possible thanks to Elena Partesotti's stay at the Interdisciplinary Sound Communication Nucleus (Núcleo Interdisciplinar de Comunicação Sonora, NICS) Laboratory of the University of Campinas which is coordinated by the professor and composer Jônatas Manzolli. This initial prototype was mainly aimed at musical education and has been expanded and improved by the addition of new functions thanks to the collaboration of the audiovisual designer Vj Kalma.
The device is based on Kinect, a technology that allows users to control and interact with the console without the need to maintain physical contact with a traditional videogames control, as it has a sensor that maps movements (which is known as a MOCAP or Motion Capture System).
“E-MOCOMU has been designed as an accessible low-cost technology that anyone can have at home, as it can be used for play in a family context. We have created different programmes to integrate the visual and sound elements", she points out.
The system is in the process of being patented thanks to a grant from the Prometeo programme for the development of market-orientated prototypes which is released annually by the General Foundation of the University of Valladolid (Funge); this has also allowed its development to continue.
“Our idea is to make it more detailed and in 3D, which would give rise to further possibilities. I would like it to be more visual and for any movement, however small, to become a specific sound. This would require the development of rather more detailed and faster technology so that the device itself could carry out all tasks. To do this we need more resources”, the researcher concludes.