बची - तर्गेतेद इन्दिविदुअल कंप्यूटर ब्रेन interface

fyi:

Mind Monitoring via Mobile
Brain-Body Imaging
Scott Makeig
Human-Computer Interface International 2009
San Diego, CA, 2009
Mind Monitoring via Mobile Brain-body Imaging
Scott Makeig1
1Swartz Center for Computational Neuroscience, Institute for Neural
Computation, University of California, San Diego, USA
scott@sccn.ucsd.edu
Abstract. Current brain-computer interface (BCI) research attempts to estimate
intended operator body or cursor movements from his/her
electroencephalographic (EEG) activity alone. More general methods of
monitoring operator cognitive state, intentions, motivations, and reactions to
events might be based on continuous monitoring of the operator’s (EEG) as well
as his of her body and eye movements and, to the extent possible, her or his
multisensory input. Joint modeling of this data should attempt to identify
individualized modes of brain/body activity and/or reactivity that appear in the
operator’s brain and/or behavior in distinct cognitive contexts, if successful
producing, in effect, a new mobile brain/body imaging (MoBI) modality. Robust
MoBI could allow development of new brain/body-system interface (BBI)
designs performing multidimensional monitoring of an operator’s changing
cognitive state including their movement intentions and motivations and (‘topdown’)
apprehension of sensory events.
Key words: cognitive monitoring; electroencephalography (EEG); motion
capture; independent component analysis (ICA); brain-computer interface
(BCI), mobile brain/body imaging (MoBI); human-computer interface (HCI)
1 Introduction
Over the last decade there has been an explosion of interest in using EEG to monitor
selected movement intentions of an operator trained to produce changes in the
amplitude of one or more EEG measures that are mechanically associated by a braincomputer
interface (BCI) system with two or more intended external actions (in
simplest form, moving a screen cursor up or down). BCI research was first funded to
construct systems allowing communication by a relatively few cognitively intact but
totally paralyzed or ‘locked-in’ subjects though, naturally, first exploratory phases of
BCI research use normal test subjects. To insure the possibility that the methods
developed in these phases might be usable by the target locked-in subjects, it was
important to establish that the EEG changes used to detect movement intentions were
not based on non-brain contributions to EEG signals recorded on the scalp, e.g.,
activity arising from subject eye movements or scalp muscle activities. Thus, for
many researchers the BCI concept became identified with the goal of using ‘pure’
EEG, apart from non-brain ‘artifacts,’ to convey and decipher a subject’s stereotyped
cursor (or body) movement intentions.
2 Scott Makeig1
The goal of providing a useful, non-invasive communication system for ‘lockedin’
subjects is surely laudable, and actual demonstrations that both a few ‘locked-in’
and many normal subjects can communicate (albeit quite slowly) via learned control
of their macroscopic brain activity patterns, without involvement of direct motor
control, are novel and intriguing. However, unnecessary adherence to this limited BCI
goal could slow development of more general classes of human-system interfaces
involving continuous monitoring of non-invasively recorded brain activity.
1.1 Unexplored problems in BCI research
As a new subject, at least four fundamental questions about the operation,
limitations, and effects of EEG-based BCI operation remain unexplored:
1. Key obstacles to widespread acceptance and application of non-invasive EEGbased
BCI systems are the need for a long training regimen, and the failure of a
significant fraction of subjects to achieve stable, non-invasive BCI control even
after intensive training. Finding specific reasons for these difficulties, and methods
around them, are fundamental if BCI or more general ‘neurotechnology’ or
‘neuroergonomic’ HCI research is to have broad applications.
2. When a subject in a BCI experiment learns to move a computer screen cursor by
increasing or reducing the amplitude of a selected brain rhythm – whether a mu
rhythm, near-DC potential, or other phenomenon – what ‘body part’ (or brain
system) do they use to willfully effect the modulation? While this is a fundamental
issue for BCI research, it is one that has so far been nearly ignored.
3. Although achieving volitional control of a BCI system through EEG modulation
alone is an intriguing goal, more general questions for HCI systems involving EEG
monitoring are how to combine EEG analysis with concurrent recording and
analysis of subject behavior, eye and muscle activities, and multisensory input to
monitor and adapt to changing human cognitive state, intent, and reactivity.
4. Another relatively unexplored question is whether there are psychobiological
effects of training and performing volitional control of natural brain rhythms.
These effects might either be phasic (affecting the operator only during BCI
operation) or tonic (also affecting their behavior and/or brain activity at other
times); they might be positive (for example producing a useful strengthening of
attentional control), or negative (some unforeseen consequence of disrupting
natural, non-conscious modes of dynamic brain regulation).
All these questions should and must eventually be addressed by the advancing
fields of human neuroscience and neurotechnology. This paper discusses a general
plan of approach to the first three questions above – How can learning of EEG-based
volitional control be made quicker and more universal? What EEG modulatory
systems do successful BCI subjects use to learn and to effect volitional control of
their EEG activity? And, how can EEG be combined with other information about
operator behavior and sensation to allow human-system interactions to estimate and
use information about operator mental state and cognitive reactions to events?
3
1.2 EEG Modulation
EEG dynamics have long been characterized by their diverse spectral profiles. For
example, slow semi-rhythmic activity is characteristic of EEG in deep sleep, while
awake/alert EEG contains more high-frequency activity. Narrow-band brain rhythms
appear most predominantly in the (8-12 Hz) alpha band, but also at somewhat higher
and lower frequencies. Spectral modulations of EEG activity at lower and higher
frequencies affect broader frequency bands. A considerable (if insufficient) amount is
known about several brain systems that modulate the spectrum of local field activity
in the brain’s cortex, the brain source of most scalp-recorded EEG. A number of these
systems are the brainstem-centered ‘evaluation’ systems labeled by the specific
neurotransmitter they project quite widely (acetylcholine, dopamine, norepinephrine,
serotonin, or etc.). However, evidence for the involvement of these or other systems
in successful BCI control has not been presented.
1.3 Mobile brain/body imaging (MoBI)
The fundamental purpose of the brain is to control behavior or more exactly, to
optimize the outcome of behavior – maximizing its ensuing rewards and/or
minimizing ensuing penalties as per subject purposes, needs, and desires. It is now
possible to record brain activity at relatively high bandwidth – a Mbit/sec or more of
EEG, MEG, BOLD, single-cell spike/field data, etc. Surprisingly, however, there has
been little serious effort to concurrently record the behavior the brain is controlling
with anything near the same bandwidth. In human brain experiments, behavior is
most often recorded only in the form of a sparse series of minimal finger button
presses – giving an effective rate of behavioral data collection near 1 bit/sec. Simply
from this ~1,000,000:1 mismatch, it is no wonder that recent progress in human
psychophysiology has been relatively slow.
The obvious remedy for this oversight is to simultaneously record as much
behavioral information as possible in paradigms including some range of natural
behavior. It should be desirable to record as wide and natural a range of behavior as
possible, in as physically free and natural a behavioral environment as possible.
Currently, this goal can only be approached only using EEG brain imaging, since its
sensors, alone among current high-bandwidth brain imaging modalities, are light
enough that its recording does not require major restriction on subject head or body
movements.
Recently, I have proposed the combination of wearable, high-density EEG and
body motion capture (combined, ideally, with eye gaze and audiovisual scene
recording) may constitute a new brain imaging modality, ‘Mobile Brain/Body
Imaging’ or MoBI [1]. Once successfully developed and demonstrated, MoBI could
allow, for the first time, the study of macroscopic brain dynamic patterns supporting
natural and naturally motivated actions (and interactions) in normal 3-D
environments.
4 Scott Makeig1
A key first problem to be overcome in realizing the promise of mobile brain
imaging is the problem of separating the activities of brain EEG sources from nonbrain
artifacts, particularly head and neck muscle activities and artifacts induced in
the EEG by eye movements. A workable solution to this problem, at least, is the
introduction of independent component analysis (ICA) of EEG data [2-6]. Under
favorable circumstances, ICA cleanly separates brain and non-brain data source
activities that are mixed by volume conduction in scalp EEG recordings, a process for
which much open-source software is now available [7]. A second problem is to model
the muscular forces producing the observed motor behavior; for this, open-source
biomechanical modeling software is also becoming available [8]. Finally, adequate
statistical signal processing or machine learning methods are required to discover
dynamic links between concurrent brain source activities, muscle activations, and
other classes of MoBI data.
Supposing the near-future availability of viable MoBI recording and analysis
methods, we can ask how the concept of BCI can be expanded to consider brain/body
interface (BBI) designs that acquire and continuously update information about the
cognitive state, reactions, intentions, and motivations of the system operator from
joint MoBI recording.
2 Brain/body Interface (BBI) Methods
For a BBI system to be maximally effective, it would seem wise to consider and
test two design principles:
a) To best understand the complex associations of ongoing multidimensional
changes in EEG dynamics with cognitive state, perceptual events, and movement
intentions and motivations, the analysis should both observe and take into
account the subject’s movements (including limb, body, and eye movements),
and any other available physiological signals. In other words, to optimally model
brain activity it is important to take in to account, as much as possible, the
behavior the brain is controlling. This suggests the potential importance of the
development of concurrent brain/body imaging recording and analysis, as in the
MoBI concept.
b) The information about cognitive state and action motivations and intentions that
may be most robustly decoded from joint EEG and behavioral information should
concern distinctions between circumstances and events in which EEG dynamics
exhibit spontaneous differences. In particular, it is likely that learned control of
EEG signals will be most successful when the learned repertoire of EEG
modulations used to decode subject control intentions are identical or close to the
subject’s repertoire of spontaneous EEG modulations.
The identified EEG dynamics used in BBI monitoring and control may either
index brain dynamics that play supporting roles in these circumstances, or their
cortical source activities may also play a direct role in shaping the joint timing of
5
distributed neural activities, a concept that is gradually being re-introduced into
neuroscience by new evidence and by theoretical considerations of the utility of mass
action in the central nervous system for controlling behavior and its outcomes.
2.1 EEG Modulators
Standard methods for analyzing EEG data are based on averaging measures of
EEG dynamics across trials or time windows, thereby collapsing the continuously
time-varying signals into a average representation of activity time-locked to one or
more classes of events. Further, most EEG analyses focuses on the individual scalp
channel signals, though they are each differently weighted mixtures of many brain
and non-brain source signals. Independent component analysis attempts to locate
discrete sources of information in multidimensional data in which several independent
information streams are linearly mixed in sensor data. However, the spectrum of each
identified brain source component signal, like every recorded scalp signal, varies
irregularly over time. Standard methods for analyzing either independent component
or scalp channel signals during a period of continued subject task performance
typically model the exhibited variability as noisy deviations from a stable mean
spectrum or stable event-related spectral perturbation (ERSP) time/frequency mean,
variation noise in which spectral power at each frequency is implicitly assumed to
vary independently.
An alternate approach assumes that the observed power spectral variability sums
variations in several to many modes of spectral variability (and co-variability) that are
characteristic of the component source process. Earlier, we introduced the device of
converting component spectrograms to log power while positing that the motive force
behind these modal modulations are processes that modulate spectral activity
multiplicatively, at characteristic frequencies, with independent or near-independent
time courses or effect distributions across trials [9]. Recently, we have tested the use
of ICA decomposition the ongoing log power spectrograms of a number of
independent component processes from single subjects performing eyes-closed
imagination exercises 1 . Log spectral decomposition separated second-to-second
variations in the log spectrogram into a log sum of multiplicative modulator
processes, each with a fixed spectral and spatial component effect template whose
effect on the affected spatial component log spectra is determined by multiplication
by a single log amplitude time series. This approach gave a number of interesting
results including alpha band processes at different frequencies plus harmonics,
broader beta and theta band processes, and very broadband shifts in power
distribution.
We have also experimented with adding information to the analysis about the
time locking and other experimental events and the context in which they occur. The
goal of this analysis approach is to avoid so far as possible the method of planned
comparisons, the basis for most experimental data analysis, in which measures for
1 Onton, J. and Makeig, S., unpublished data
6 Scott Makeig1
pairs of conditions are compared, each measure an identically weighted average of
measurements characterized by one (or sometimes more) key variable value.
For example, there have been thousands of EEG studies that compared the
average responses (typically called ‘P300’) to ‘target’ and ‘non-target’ stimuli in a
simple attention task. The underlying assumption here is that the brain emits identical
responses to each ‘target’ or ‘non-target’ stimulus, respectively, regardless of the local
event context. The hope is that the effect of the ‘target/non-target’ variable is
separable from other variables, and essentially stable across time. Unfortunately, this
is not the case. P300 ‘target’ responses vary widely in amplitude and scalp
distribution from trial to trial, and this type of variability limits the performance of
simple BCI systems, for example one that might attempt to set a fixed threshold to
identify the appearance of a ‘target’ response, regardless of event context [10].
I propose that BBI research explore an alternative approach in which multiple
characteristic relationships between EEG dynamics and single events in context are
determined directly from the joint EEG, stimulus, and behavioral data. Some facts
concerning the nature of individual events may be available to at BBI system in real
time, for example the moment and screen on which a piece of information is
presented, or the screen to which the subject is directing their gaze.
An example of an unavailable context variable might be the interpretation of the
subject of a visual event as representing a challenge or threat. In pilot data recorded to
build an individualized (or collective) BBI model, the level of threat could be varied
systematically and the level of perceived threat might be estimated from the subject’s
brain and behavioral responses. In subsequent real-time operation, other variables
defining the current event and event context may be available from the system event
log and subject behavioral record.
Combined with direct observation of the EEG and subject motor behavior, these
available context variables, combined, may allow estimation of the unavailable
variable – here, whether and to what extent the subject perceives a visual event to
signal a threat to the operation of the system. This information might be used to
immediately deploy available additional countermeasures whenever a genuine
perceived system threat is estimated to occur, or possibly to monitor the state of
responsiveness of the subject when false indications of (test) threats are delivered to
the subject, probing their advancing level of expertise in recognizing a threatening
event, or for estimating their current cognitive fitness for duty.
If the system response to the operator’s appraisal of a threatening event helps the
operator mount an adequate and timely response, then the system response will serve
as a powerful reward, and naturally over time and use the operator’s EEG pattern
should be expected to adapt to give a more distinct perceived-threat signal to the
system. Thus, a natural cognitive response monitoring system could easily become an
interactive learned BCI/BBI system. Further, it is natural to hypothesize that when the
system is based on the operator’s natural brain response modes, it may also be natural
7
and relatively easy for the operator to learn to produce the EEG patterns that are most
distinctly and reliably detected by the system.
Fig. 1. Schematic model diagram for a non-invasive brain/behavior system interface (BBI)
design. Concurrent scalp EEG, behavior, and event/context data are collected in a Mobile
Brain/body Imaging (MoBI) paradigm (thick ovals). In most currently proposed BCI systems
(dotted arrows), selected EEG data are processed in near-real time to estimate or predict some
behavioral or event/context parameter (‘BCI’). In the proposed BBI, the EEG data are first
separated into cortical EEG source processes (upper middle oval) (plus non-brain artifact
processes, not shown). Then the time/frequency behaviors of the source processes are further
separated into effects of a number of maximally distinct EEG source modulator processes
(upper right oval). In the BBI model, both selected EEG time-domain source and frequencydomain
source modulator data may be integrated with the behavioral and other event/context
data to estimate or predict selected behavioral and/or event/context parameters (broad light blue
arrows).
Fig. 1 gives the gist of the concept in graphic form. Three types of MoBI data
may be recorded concurrently to run a brain/body interface (BBI): high-density EEG
data, behavioral data, and context data (event information, event, EEG, and
behavioral history, etc.). Standard BCI systems (dotted arrows) attempt to estimate
some parameter of the behavioral and/or event/context data directly from the scalp
EEG using a machine learning approach. In the proposed BBI model (wide light blue
arrows), the EEG data are first separated into cortical (and non-brain) EEG source
processes (thin blue arrows), the spectral modulator processes operating on these
source processes are estimated from the EEG source data, and the linkage of the EEG
source and source modulator processes to the behavioral and/or event/context data are
8 Scott Makeig1
determined. When one or more parameters of the event/context data are unavailable
(e.g., in real-time operation), any of the available MoBI data may be used in a BBI to
estimate the unavailable parameter. The estimation process might be designed to
perform well even when additional data variables are missing. The MoBI data used
for this estimation might include available behavioral data (body motion capture, eye
gaze tracking, etc.) and event/context information as well as EEG dynamics.
3 DISCUSSION
The model of an EEG-based BBI system shown in Fig. 1 has the advantage of
involving volitional control of spatiotemporal EEG dynamic patterns most
specifically associated with the operator’s spontaneous EEG responses in the targeted
event categories [11]. While it is natural to hypothesize that strengthening and
controlling spontaneously active EEG patterns may be more easily and quickly
learned, this assumption may prove incorrect in some or many cases, and thus basic
experiments (and adequate analyses) are needed to test it. Earlier, we showed that
applying even highly overlearned BCI control of a single pre-defined EEG feature
may involve complex and asymmetric EEG changes in and among many cortical
regions [12]. Thus, gaining a basic understanding of the nature and learning of
volitional EEG control may in many cases prove to be a complex and difficult
process.
How may we determine which brain modulatory systems are involved in
spontaneous and learned control of particular EEG or behavioral/EEG dynamics? A
full answer to this question may require invasive experiments (potentially involving
patient volunteers who have been implanted with cortical electrodes for clinical
purposes), positron emission tomography (PET) experiments that can assess
neurotransmitter distributions in the brain, various psychopharmacological
manipulations, combined with carefully selected behavioral paradigms, for example
those directly manipulating reward levels known to be linked to dopamine release
[13]. However, a number of brain modulatory systems may be involved in most state
changes and event responses of interest, so this investigation should be expected to be
involved.
A possible objection to the model shown in Fig. 1 is that if an adequate BCI
function linking the recorded EEG signal to the target behavioral or event/context
parameter(s) of interest proves to be linear, then constructing a more elaborate BBI
function linking EEG data first to EEG sources, then to their natural modes of spectral
modulation, and finally to the estimated event/context or behavioral measure may not
give a better-performing estimator. The proposed EEG source modulator model,
however, is nonlinear as it operates on source (log) power spectra. Linear or other
functions of the estimated source and source modulator time courses, therefore,
involve additional information and might well have advantages over direct (and
particularly, linear) BCI estimation. However, use of power spectral estimates ignore
source signal phase and with it, precise latency information available in the time9
domain data. Thus, applying a joint linear (or other) estimator to combine timedomain
and time/frequency-domain data could improve performance over a timedomain
estimator alone.
Recently Bigdely Shamlo and colleagues demonstrated a successful application of
such an approach [14]. We reported a method for estimating the probability that a
briefly presented visual image contained a rare target feature – an airplane feature in a
stream of satellite ground images presented to the subject at a rate of 12 images per
second. Near-real time performance in correctly detecting the presentation of single
target-bearing images solely from high-density EEG (by combining source timedomain
and source spectral modulator information in a linear estimator) was high,
giving an area under the ROC curve of over 90% for most subjects.
Like most BCI projects, this project did not expressly capture subject behavioral
information. However, it did allow use of maximally independent EEG sources
capturing potentials induced by characteristic subject eye gaze behavior following
target appraisal, unlike BCI systems built to serve completely paralyzed subjects.
Although the very rapid serial visual presentation (RSVP) did not reward normal
saccadic eye movements, independent components accounting for eye movements
following target perception was found to carry some target classification information
(though of less value compared to several brain EEG source responses).
The BBI model shown schematically in Fig. 1 does not propose a method for
combining EEG and behavioral data, in particular body motion capture data. This is a
topic that both requires and deserves much attention and exploration. Of particular
interest is to determine the extent to which it is desirable to solve the biomechanical
inverse problem, estimating which muscle actions produce the observed sequence of
body movements, before estimating links between EEG source activities, body
movements, and operator mental state or reactions [8].
Finally, can the proposed MoBI-based BBI systems be practical for widespread
application, or must they remain basic research tools? EEG spatial filtering requires
the availability of a relatively high number of scalp EEG recording channels.
Typically, BCI designers have attempted to maximize signal to noise ratio by
restricting the number of channels used in the classifier, an approach that might also
lower the cost of the system, if realized using currently available technology. To date,
body motion capture (mocap) systems also remain quite expensive. Thus, can the
proposed MoBI-based BBI systems become practical for routine application, even in
(e.g.) high-value military or civilian environments? Here, the rapid progress of
electronic fabrication methods, allow microminiaturized data acquisition and
processing units based on flexible thin-film technologies should allow development
and relatively low-cost deployment of wearable high-density EEG and behavioral
monitoring systems within a few years [15]. Such systems should be readily
applicable to some important problems, for example alertness monitoring of shiftwork
operators of high-value, high-risk systems [16]. Full realization of the MoBIbased
BBI concepts discussed here will likely require a great deal more basic and
10 Scott Makeig1
applied research in many laboratories combining expertise in several fields of
neuroscience, mathematics, and engineering.
References
1. Makeig, S., Gramann, K., Jung, T-P., Sejnowski, T.J., Poizner, H.: Linking brain, mind
and behavior: The promise of mobile brain/body imaging (MoBI). Int. J. Psychophysiolog.
(accepted for publication)
2. Makeig, S., Jung, T-P., Ghahremani, D., Bell, A.J., Sejnowski, T.J.: Blind separation of
auditory event-related brain responses into independent components. Proc. Natl. Acad.
Sci. USA. 94, 10979—10984 (1997)
3. Makeig, S., Westerfield, M., Jung, T-P., Covington, J., Townsend, J., Sejnowski, T.J.,
Courchesne, E.: Functionally independent components of the late positive event-related
potential during visual spatial attention. J. Neurosci. 19, 2665—2680 (1999)
4. Jung, T-P., Makeig, S., Humphries, C., Lee, T-W., McKeown, M.J., Iragui, V., Sejnowski,
T.J.: Removing electroencephalographic artifacts by blind source separation.
Psychophysiolog. 37, 163—178 (2000)
5. Jung, T-P., Makeig, S., Westerfield, M., Townsend, J., Courchesne, E., Sejnowski, T.J.:
Analysis and visualization of single-trial event-related potentials. Human Brain Mapping
14, 166—85 (2001)
6. Makeig, S., Westerfield, M., Jung, T.P., Townsend, J. Courchesne, E. Sejnowski, T.J.:
Dynamic brain sources of visual evoked responses. Science 295, 690—694 (2002)
7. Delorme, A., Makeig, S.: EEGLAB: An open source toolbox for analysis of single-trial
EEG dynamics including independent component analysis. J. Neurosci. Meth. 134, 9—21
(2004)
8. Delp, S.L., Anderson, F.C., Arnold, A.S., Loan, P., Habib, A., John, C.T., Guendelman,
E., Thelen, D.G.: OpenSim: Open-source software to create and analyze dynamic
simulations of movement. IEEE Trans. Biomed. Eng. 54, 1940—50 (2007)
9. Makeig, S., Delorme, A., Westerfield, M., Jung, T-P., Townsend, J., Courchesne, E.,
Sejnowski, T.J.: Electroencephalographic brain dynamics following manually responded
visual targets. PLOS Biol. 2, 747—762 (2004)
10. Onton, J., Delorme, A., Makeig, S.: Frontal midline EEG dynamics during working
memory. NeuroImage 27, 341—356 (2005)
11. Makeig, S., Enghoff, S., Jung, T-P., Sejnowski, T.J.: A natural basis for efficient brainactuated
control. IEEE Trans. Rehab. Eng. 8, 208—211 (2000)
12. Delorme A, Makeig S. EEG changes accompanying learned regulation of 12-Hz EEG
activity. IEEE Trans. Neural Sys. Rehab. Eng. 2, 133—136 (2003)
13. Arias-Carrión, O., Pöppel, E.: Dopamine, learning and reward-seeking behavior. Act.
Neurobiol. Exp. 67, 481—488 (2007)
14. Bigdely-Shamlo, N., Vankov, A., Ramirez, R.R., Makeig, S.: Brain activity-based image
classification from rapid serial visual presentation. IEEE Trans. Neural Syst. Rehabil. Eng.
16, 432—441 (2008)
15. Chin-Teng Lin, C-T., Ko, L-W.; Chiou, J-C., Duann, J-R., Huang, R-S., Liang, S-F., Chiu,
T-W., Jung, T-P.: Noninvasive neural prostheses using mobile and wireless EEG. Proc.
IEEE 96, 1167—1183 (2008)
16. Makeig, S., Inlow, M.: Lapses in alertness: Coherence of fluctuations in performance and
the EEG spectrum. Electroencephalogr. Clin. Neurophysiol. 86, 23—35 (1993)