Coenesthesia: An Aesthetic of Healing through Hybrid Reality Art ©

An excerpt from my thesis

3.2      Creating a Sense of Body Ownership Over Virtual Body Parts.

First I want to clarify the use of ownership.  Here I do not write in socio-economic terms of ownership. Ownership in this context comes from the language used in neuro-prosthetics research, to establish things that are included as being part of the makeup of one’s physical body. 

Whereas amenable objects are the realm of ‘me and not me extension’ somewhere between subjective and objective, so too are prosthetic limbs. I look to the field of cognitive neuro- prosthetics to establish techniques for creating a sense of body ownership over virtual representations of body parts. This field builds, “models of self-consciousness in order to project them onto artificial limbs, avatars and robots.” (Blanke, 2012) I look at a few contributions to this field that seem relevant to my mission.

Hugh Herr, renowned neuro-prosthetics researcher who revolutionized prosthetic limbs after losing his own legs in a climbing incident at the age of 17 says "Technology has the power to heal, to rehabilitate and to even extend human experience and capability" (Hugh Herr, 2015)

In 1998 Botvinick and Cohen established the ‘the rubber arm illusion’.  This is an experiment where one arm of the participant is hidden from view and a rubber arm is positioned where the participant’s real arm would typically be. The experimenter then simultaneously brushes both the real arm of the participant and the rubber arm for length of time. This synchronization creates a sense of body ownership in the participant for the rubber arm and when it is threatened with pain, they jump to protect it.  This experiment shows how sight, touch and proprioception combine to create a convincing feeling of body ownership, one of the foundations of self-consciousness (Nature, vol 391, p 756).  In 2007, Frank H Durgin et al, expanded on the rubber arm experiment[1]. They replaced the brush with a laser and discuss the results in terms of multisensory integration[2]. Vision, Touch and Synchronisation are all key elements.

Experiments along similar lines, established the mirror therapy method (Ramachandran, 1994), to relieve pain in phantom limbs. Using a mirror to recreate a mental image of the missing limb was found to alleviate pain in the phantom limb and over time to remap the mental image of that limb in the brain of the amputee having a lasting effect on the pain relief. With the limb missing, confusion develops in the signals sent between the limb and the brain, forming in some patients a mental image of a cramped up, clenched hand. When a mirror is used to substitute an image of the opposite arm there is instant pain relief when they see the hand stretched and relaxed.  The mental image of the limb is key to the experience of it.

Heautoscopy is a phenomenon where patients have a sensation of being reduplicated and to exist at two or even more locations. It has been found that self-identification with two virtual bodies was stronger during synchronous stroking (Heydrich, 2013).  This proposes that having more than one version of organs in the virtual space I am creating will allow the user to identify with more variations of organ representation.

To make effective use of the sound component in the research creation VR project, the sound component incorporation is influenced by the findings of Noel et al, in which they consider acoustic stimuli, in the mix of synchronous stroking during the rubber arm illusion, “the distance at which acoustic stimuli are presented may alter the balance between self- and non-self, biases.” (Noel et al, 2017). 

Insights gained from this research include the fact that light when employed through synchrony with a physical sensation can contribute to a sense of body ownership and of heautoscopy, actively engaging motor neurons and the position of acoustic stimuli are also of key importance to the mix. In Coenesthesia I create a synchronous link between the immersant’s real organs and the virtual representations of the body organs and virtual objects through synchronous biometric data.

 

[1] Two experiments involving a total of 220 subjects are reported.  The experiments document that “stroking”a false hand with the bright beam of light from a laser pointer can produce tactile and thermal sensations when the hand can be seen as one’s own. Overall, 66% of subjects reported somatic sensations from the light. Felt hand location was recalibrated toward the location of the false hand for those subjects who felt the light. Moreover, the proprioceptive recalibration from the laser experience was comparable to that produced by actual coordinated brushing of the false hand and of the unseen heal hand after 2 min of stimulation. The illusion may be experienced on one’s real hand as well. (Durgin,)

 

[2] We interpret this touch-from-light illusion in terms of a multisensory-integration theory wherein perceptual signals of highcertainty from one sense modality can produce perceptualconsequences that influence the experience of a second modality (De Gelder & Bertelson, 2003; Driver & Spence, 2000; Ernst & Banks, 2002; Gibson, 1966; McGurk & MacDonald, 1976; Shimojo & Shams, 2001). For example, an insect crawling on the skin would not normally produce tactile sensations if the mechanical disturbances to the skin are below sensory threshold. Once the insect is seen, however, a vivid experience of

tactile sensations could arise from the combination of the visual localization evidence with sensory noise from the tactile sensors (see Durgin, 2002). In the present case, sensory integration depends on the ease with which a rubber hand can be incorporated

into the body schema or body image (Head et al., 1920;Schilder, 1938).

 

Consciousness as Holographic Drawing Device.

A bit of a ramble on consciousness

By: Wendy Whaley

The author is never wholly dead as suggested by Roland Barthes (“The Death of the Author”1977). The author’s consciousness is fixed in position - time being a component of position, and words and perspective being components of consciousness - to be encountered from other positions following the birth of the work. If consciousness is a person’s awareness or perception of something (Oxford English Dictionary,1933) and perception is a kind of additive accumulation of all of inputs and thoughts derived from those inputs, then a holographic snapshot of the author’s consciousness becomes part of the reader’s consciousness and symbolically creates new silver halide crystals for each moment that the work is considered, thus drawing consciousness through space and time.

 Under all circumstances of being alive, we experience sub consciousness, by virtue of our autonomic nervous system. We cannot be conscious without simultaneously being subconscious but we can be subconscious without being conscious.  The voice in one’s head is usually the conscious, the subconscious is quieter and typically presents as a gut feeling, an instinct, it is the signal sent from the brain to tell a foot to move, it is the electrical signal beating one’s heart. When conscious one tends to tune out the majority of one’s subconscious operations. I am looking for the situations in which one tunes into them, synchronising and balancing the systems of consciousness, attenuating signals, gaining symbiotic control between the conscious and subconscious.

At its basest level, consciousness is made up of the fundamental constituents of matter and the forces acting between them, our minds are grown from this and store our memories, our experiences in an intricate network of what Max Tegmark would call the 5th state of matter (Tegmark, 2014). As liquid, gas, solid and plasma are states of matter, so too could consciousness be.  But is it something physical and tangible or might it be purely an invisible force? There are various schools of thought on what exactly consciousness is?  Many people in the arts and sciences, have cited an ethereal consciousness plane that they tap in to, a place where they go and are fed ideas. Prince, Einstein, David Lynch, Floria Sigismondi, are but a handful with this experience.  Daniel Dennett is a neuroscientist who suggests that consciousness is simply a physical by-product of evolution (Rothman, 2017). There are neurological structures thought to house memories and cognitive functions, but consciousness has not been proven as solely a combination of these things. Alan Watts speaks of consciousness as resonance, and that if you are happy and you know that you are happy then that is resonance. Philosopher Nick Bostrom published a thought experiment exploring the possibility that we live in a simulation which if true suggests that we are some form of Artificial Intelligence, programed and run as a simulation (Bostrom, 2016). 

 I speculate that consciousness is fractal memory bank of catalogued experiences of phenomenological existence that forms an ego, combined with our symbiotic connection to the whole of reality: a state of matter that allows for reception and transmission of waveform data.  Allan Watts says that each one of us is an “aperture by which the universe looks back at itself” (Watts, 1960). The scientific guru of the Dali Lama, David Bohm suggests consciousness as holographic (capable of projecting what we perceive as reality) (Bohm,2008).  Consciousness then, is perspective of thought. I wonder what happens if we proprioceptively shift our perspective, maybe we just become a little more conscious by occupying more space and gaining a higher resolution. Rather than one single perspective, if we think of pixels as perspectives then perhaps the greater the resolution the more conscious we become. This would most certainly apply to how conscious a person is who sees things from other people’s perspectives (although one can never truly see things from another’s perspective since their perspective is dependent on their lived experience leading up to any given moment in time, we can only proprioceptively shift our own consciousness to attempt things from another’s perspective), and   Perhaps more accurately though, if we think of perspectives from a fractal perspective, this holds the greatest potential for increased consciousness.  Perhaps if we apply this concept to looking inward at our bodies from more perspectives, the perspectives of our subconscious, our autonomic systems, we can gain an amplification of consciousness and of self.

 

A Look at Hito Steyerl's "In Free Fall: A Thought Experiment on Vertical Perspective"

This piece takes us on a journey through advancements in our use of perspective from classic linear perspective through to the common areal perspectives of today’s drones and satellite imagery. Hito suggests that William Turner broke us out of our traditional linear perspective by taking into account the perspectives relevant to his works (climbing the mast of a ship to get the feel and tilted angle of viewing things from above while at sea for the slave ship, or hanging his head out of a moving train for Rain, Steam and Speed – The Great Western Railway).

 

Hito moves on to speak of Theodor Adorno scoffing at “philosophy’s obsession with earth as origin” (Steyerl, 2011) and sums the whole thought experiment up in the following 3 paragraphs. First:

A fall toward objects without reservation, embracing a world of forces and matter, which lacks any original stability and sparks the sudden shock of the open: a freedom that is terrifying, utterly deterritorializing, and always already unknown. Falling means ruin and demise as well as love and abandon, passion and surrender, decline and catastrophe. Falling is corruption as well as liberation, a condition that turns people into things and vice versa.

 

At the core of this sentiment I feel as though a good way to visualize this is to think of how aerial and space photography separates us from the earth which we have been attached to for most of our lives and so it is a kind of leaving the womb, so we could either feel excited to get out and explore or afraid to leave our safe home with nothing to hold on to. The final statement makes us pause to contemplate “Falling is corruption as well as liberation, a condition that turns people into things and vice versa”. This makes me think of the film Falling Down with Michael Douglas (Joel Schumacher,1993). Michael Douglas’s character is fired but he doesn’t tell his family, this begins both his liberation and his decent into madness.  With injustices as targets and himself labeled as a terrorist, both he and his targets become things from his perspective but people from the publics perspective.

She continues with one sentence that stands alone as an entire paragraph:

It takes place in an opening we could endure or enjoy, embrace or suffer, or simply accept as reality.

Here she is saying that your mental perspective is the lens with which you choose to see anything and everything.

Finally, the perspective of free fall teaches us to consider a social and political dreamscape of radicalized class war from above, one that throws jaw-dropping social inequalities into sharp focus. But falling does not only mean falling apart, it can also mean a new certainty falling into place. Grappling with crumbling futures that propel us backwards onto an agonizing present, we may realize that the place we are falling toward is no longer grounded, nor is it stable. It promises no community, but a shifting formation.

Reading this really gives wonderful perspective to life in general and reinforces the need to embrace a path of excitement toward movement and change. True happiness lies in the ability to adapt. Here I will quote one of my favorite tunes (Eric Idle, 1983)

Whenever life gets you down, Mrs. Brown,
And things seem hard or tough,
And people are stupid, obnoxious or daft,

And you feel that you've had quite eno-o-o-o-o-ough,

Just remember that you're standing on a planet that's evolving
And revolving at 900 miles an hour.
It's orbiting at 19 miles a second, so it's reckoned,
The sun that is the source of all our power.
Now the sun, and you and me, and all the stars that we can see,
Are moving at a million miles a day,
In the outer spiral arm, at 40,000 miles an hour,
Of a galaxy we call the Milky Way.

Our galaxy itself contains a hundred billion stars;
It's a hundred thousand light-years side to side;
It bulges in the middle sixteen thousand light-years thick,
But out by us it's just three thousand light-years wide.
We're thirty thousand light-years from Galactic Central Point,
We go 'round every two hundred million years;
And our galaxy itself is one of millions of billions
In this amazing and expanding universe.

Field Observations

By: Wendy Whaley

I chose a perhaps disputable gesture study for my field observations.  I say that it could be a disputable subject just based on the reading of Gesture and Affect: The Practice of a Phenomenology of Gesture. In this text, Vilém Flusser attempts to rather allusively, define gesture in such a way that I cannot even pull out one quote from the text that would get at the core of what he is saying. I am not even sure that I don’t just think this article is just a bit of mental Ashtanga that leaves the interpretation as open as it begins.  Therefore, I will interpret gesture based on a mix of experience and instinct.

I was going to say that gesture is any movement that is meant as a means of communication, but one can gesture without meaning to, and so, to me a gesture is any movement that elicits emotion.  I will use the internet as my field and explore gestures that elicit frisson.

The piece that I will discuss is the giffy.gif that was spread around Reddit and Facebook about a year ago.  The piece came with a headline that read “A Microtubule Carrying Happiness to Your Brain”, and in the small blurb underneath said that it was a microtubule carrying dopamine to receptors in the brain.  I have kept this gif on my desktop to access at a moment’s notice to help flood my brain with happiness whenever I need it. I have been trying to pinpoint the elements within this moving image that make me feel as though happiness is flooding my brain. Perhaps it was just the suggestion from the headline, and I do think that has played a part, but there are also visual clues. When the image is still and not moving, it has no affect.  I found what I believe to be the source of this image, a Harvard documentary The Inner Life of a Cell.  The definition of what is actually happening in the scene that contains this clip is that 'micro tubules provide tracks for membrane bound vesicles to travel, to and from, the plasma membrane', not the same as was suggested with the gif.  The footage in the documentary also has no effect on me, and so I have looked at the differences in the documentary footage to the perfectly looping gif that I found on the internet.  The footage in the documentary is lit very neutrally, just a minimal diffuse ambient overall, whereas the footage in the gif has a bright light source creating shadows that accentuate the movement.  The gif contains more particle detail in the surface of the ball that the microtubule is carrying, the waves of interacting particle dynamics cause me frisson. The other notable gesture that causes frisson is the motion of the microtubule. The anthropomorphic shape that it has mimics that of a person carrying the ball with fuzzy slippers on its feet.  The feet appear to have this wonderful clicking into place each time they take a step. The looping animation of the entire clip is seamless, I could watch it until my eyes get tired

Flusser's piece actually provides a clue to the effectiveness of this piece. He speaks of 'movements of the body',  and alludes to gestures being a human phenomenon. I propose that the anthropomorphic qualities of the vesicle contribute to the reading of this gif as a gesture. 

 

references:

Flusser, Vilém. “Gesture and Affect: The Practice of a Phenomenology of Gesture.” Gestures.

Minneapolis: University of Minnesota Press, 2014. 1-9. Print.

 

https://www.youtube.com/watch?v=FzcTgrxMzZk&t=5s

 

Holography, Stereoscopic 3D & the Struggle to be Taken Seriously as Art Forms.

Written By: Wendy S. Whaley

Within moments of embarking on my journey to become a legitimate artist, by attending what was then known as the Ontario College of Art (now OCADU), I discovered Holography. “Wow” I thought, “Art and Science together, a perfect pair. I can paint in space with pure light.”

I studied the works of Margaret Benyon, Bruce Evans, Al Razutis, Claudette Abrams, Michael Page, the massive full scale bar of Paula Dawson (she actually made a full scale holographic neighborhood pub). These are some truly inspiring artists. As I became more involved with this world of holography, it became quickly evident that there was snobbery when it came to the holographic medium. As the school was going through restructuring, at every stage, there was a constant struggle to keep the Holography lab and courses available to students. The idea that holography is not really art, it’s a science, spread like an infection.  Michael Page should be commended for his Herculean efforts to keep it alive.  I suppose this concept stems from the large amount of purely industrial or trinket holograms in existence, but the same can be said of any medium. As a Rembrandt that stares back at you with emotion and meaning, and the wall of your apartment painted blue, are, at their core, different creatures, so too, are a Margaret Benyon and a rotating hologram of a car. So long as there is meaning behind the medium used, art is not about the medium, but about the message one wants to convey through that medium and the connection the artist can form with the viewer.

After Graduating OCA, I met Noel Archambault (A pioneer IMAX DOP and Stereographer) who adapted my knowledge of holography to stereo 3D and I became deeply embroiled in the resurgence of stereoscopic cinema.

As holography and stereoscopic 3D have been accused of being 'gimmicky' and 'passing fads', there has been a stronghold of artists, DOP's and Directors who have held fast to the knowledge that these are great tools with which to connect with the viewer on a primal level.

Stereoscopic cinema found its popularity in the wonderfully campy science fiction films of the 50's but took a dive in popularity due to technical standards not being strong enough to keep the experience comfortable for audiences. The resurgence of stereoscopic cinema in the 21st century has solved many of the technical issues revolving around projection, but with the majority of studios not willing to pay for true stereo productions and not willing to pay to do 3D conversion to real world standards, the industry in North America is again shrinking due to technical and artistic insufficiency. Only fully 3D animated films take advantage of stereo 3D and many children are uncomfortable wearing 3D glasses.

This industry suffers from the same snobbery as the holographic industry but more from the perspective that the 3D is a nuisance that does nothing to contribute to the experience. This is partly due to the aforementioned issues but also due to the fact that very few productions actually use 3D depth as an artistic means of connecting with the audience. Whether it is gimmicky or not, the 'Despicable Me' films are some the only ones in recent years to successfully use 3D to help connect with the audience, by means of using it to accentuate comedic punctuation. Back in the day, Andy Warhol's “Frankenstein” stands out for the final scene, when Udo Kier is stabbed and his liver is poking out on a spear. Parallax and convergence are pulled simultaneously to emphasis the moment.

In an over enthusiastic effort to keep the experience comfortable for the audience, the modern 3D cinema revolution has really missed the boat on the usefulness of stereo 3D. The industry's fear of losing a profitable revenue stream by pushing the 3D too far has left them with a shallow, minimally effective product that has little appeal to anyone truly interested in an immersive experience.

For 3D to be used effectively in cinema, it needs to be treated like the number one supporting actor. It is a tool with which to more deeply connect with the viewer. Imagine that you are a film maker. As you show a close up on an actor, consider the scale and space between you. You the film maker, now have a tool with which to enter the viewer's personal space. Before filming it is important to do a complete script and storyboard analysis and tweak to determine when to call upon the tools of enhanced perception afforded to you by stereo 3D. These tools include:

  • Scale. The ability to make the viewer seem small or large in a scene through interocular distance adjustments.

  • Distance. The ability to enter the viewer's personal space.

  • P.O.V. is more effective in stereo 3D.

  • The use of Stereo Window can enhance the feeling of objects entering the theatre space. Through use of the false frame that separates objects behind the screen from those in the theatre space.

  • 2D and 3D depth cues are symbiotic and enhance each other. So be liberal with linear and aerial perspective, texture, parallax and occlusion.

  • Stereo Pyramid. By keeping objects from breaking frame, keep them inside the pyramid between the back of the viewer’s head and the screen for more effective stereo.

  • Limitation: Infinite Depth of Field (this is a bone of contention with most DOP's. To effectively and comfortably create 3D, it is best to adhere to infinite depth of field for all shots that have significant depth and minimal camera moves. Use light and motion to steer the viewer’s eye rather than focus. This is due to the simple fact that in order to view stereo 3D our natural mechanism of having our focus and our convergence connected is unnaturally forced apart, it hurts less if everything is in focus.

In conclusion, both Holography and Stereo 3D are effective tools for connecting with the viewer at a more primal level. If they are to be taken seriously as artist media, then artists need to use them effectively toward accentuating moments of impact and occupying the space directly around the viewer, with specific artistic intent. These tools are not effective in a vacuum and must work in conjunction with two dimensional depth cues such as linear and aerial perspective, texture, parallax and occlusion. Most importantly of all, to be effective, there must be motivation for their use, through depth of story and meaning.

I am a part of the stronghold who believes in the power of 3D because it immediately makes the viewer integral to the work, which inherently creates a greater connection to it. I look forward to what artists might do with Google Glass and other examples of exciting, emerging technologies.

 

IMAX Cliffhanger

 

"IMAX approached me to be the VFX Supervisor for their new trailer.  I worked closely with Director Jackson Myers to establish the look of the piece.  Being attached to Method Studios at this time, we brought it there where I Supervised a team of talented artists, to bring this epic piece to life.  This is a fully stereoscopic 4k piece.  We shot in Toronto at Pinewood Studios on a Red Epic Dragon. 
It was an honor working with Jackson, who's Mother Tony Myers I have worked with many times or the years, and an honor to finally have the chance to work with Dylan Reade who trained and worked with my mentor Noel Archambault for years. 
To see the entire trailer, please go see a film in IMAX."WW