Measures and Metrics for Assessing Detection Algorithms in Brain Machine Interfaces

Ameer Mohammed, Ashraf Adam Ahmad, Auwalu M. Abdullahi, Mahmood T. Kabir

Abstract


A major challenge in the analysis of detection algorithms for brain machine interface (BMI) is the availability of objective methods for comparing various technologies. The absence of standard measures for BMI detection algorithms prompted the need for a thorough review and commentary on performance measures for BMI algorithms. This was necessary so that suitably tailored measures are selected that are fully representative and compatible with set performance targets. The review includes measures that could be used to evaluate efficacy, complexity and efficiency of detection algorithms; as well as their strengths and weaknesses. At the end, a commentary is provided on the need to provide either standard or custom performance measures for BMI algorithms, such that the algorithms can be assessed accurately.


Full Text:

PDF

References


Aho, K., Derryberry, D., & Peterson, T. (2014). Model selection for ecologists: the worldviews of AIC and BIC. Ecology, 95(3), 631–636.

Akce, A., Norton, J. J. S., & Bretl, T. (2015). An SSVEP-Based Brain-Computer Interface for Text Spelling With Adaptive Queries That Maximize Information Gain Rates. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 23(5), 857–866. https://doi.org/10.1109/TNSRE.2014.2373338

Baizabal-Carvallo, J. F., & Jankovic, J. (2016). Movement disorders induced by deep brain stimulation. Parkinsonism & Related Disorders. https://doi.org/10.1016/j.parkreldis.2016.01.014

Baldi, P., Brunak, S., Chauvin, Y., Andersen, C. A., & Nielsen, H. (2000). Assessing the accuracy of prediction algorithms for classification: an overview. Bioinformatics (Oxford, England), 16(5), 412–424.

Battiti, R. (1994). Using mutual information for selecting features in supervised neural net learning. IEEE Transactions on Neural Networks, 5(4), 537–550. https://doi.org/10.1109/72.298224

Biederman, W., Yeager, D. J., Narevsky, N., Leverett, J., Neely, R., Carmena, J. M., … Rabaey, J. M. (2015). A 4.78 mm 2 Fully-Integrated Neuromodulation SoC Combining 64 Acquisition Channels With Digital Compression and Simultaneous Dual Stimulation. IEEE Journal of Solid-State Circuits, 50(4), 1038–1047. https://doi.org/10.1109/JSSC.2014.2384736

Chen, W.-M., Chiueh, H., Chen, T.-J., Ho, C.-L., Jeng, C., Ker, M.-D., … Wu, C.-Y. (2014). A Fully Integrated 8-Channel Closed-Loop Neural-Prosthetic CMOS SoC for Real-Time Epileptic Seizure Control. IEEE Journal of Solid-State Circuits, 49(1), 232–247. https://doi.org/10.1109/JSSC.2013.2284346

Elhosary, H., Zakhari, M. H., Elgammal, M. A., Abd El Ghany, M. A., Salama, K. N., & Mostafa, H. (2019). Low-Power Hardware Implementation of a Support Vector Machine Training and Classification for Neural Seizure Detection. IEEE Transactions on Biomedical Circuits and Systems, 13(6), 1324–1337. https://doi.org/10.1109/TBCAS.2019.2947044

Fatourechi, M., Ward, R. K., & Birch, G. E. (2008). A self-paced brain–computer interface system with a low false positive rate. Journal of Neural Engineering, 5(1), 9–23. https://doi.org/10.1088/1741-2560/5/1/002

Fawcett, T. (2006). An introduction to ROC analysis. Pattern Recognition Letters, 27(8), 861–874. https://doi.org/10.1016/j.patrec.2005.10.010

Fisher, J. W., Siracusa, M., & Tieu, K. (2009). Estimation of Signal Information Content for Classification. 2009 IEEE 13th Digital Signal Processing Workshop and 5th IEEE Signal Processing Education Workshop, 353–358. https://doi.org/10.1109/DSP.2009.4785948

French, D. D., Campbell, R. R., Sabharwal, S., Nelson, A. L., Palacios, P. A., & Gavin-Dreschnack, D. (2007). Health care costs for patients with chronic spinal cord injury in the Veterans Health Administration. The Journal of Spinal Cord Medicine, 30(5), 477–481. Retrieved from http://www.ncbi.nlm.nih.gov/pubmed/18092564

Fukami, T., Shimada, T., Forney, E., & Anderson, C. W. (2012). EEG character identification using stimulus sequences designed to maximize mimimal hamming distance. 2012 Annual International Conference of the IEEE Engineering in Medicine and Biology Society, 1782–1785. https://doi.org/10.1109/EMBC.2012.6346295

Goñi, J., van den Heuvel, M. P., Avena-Koenigsberger, A., Velez de Mendizabal, N., Betzel, R. F., Griffa, A., … Sporns, O. (2014). Resting-brain functional connectivity predicted by analytic measures of network communication. Proceedings of the National Academy of Sciences of the United States of America, 111(2), 833–838. https://doi.org/10.1073/pnas.1315529111

Hacker, M. L., Tonascia, J., Turchan, M., Currie, A., Heusinkveld, L., Konrad, P. E., … Charles, D. (2015). Deep brain stimulation may reduce the relative risk of clinically important worsening in early stage Parkinson’s disease. Parkinsonism & Related Disorders, 21(10), 1177–1183. https://doi.org/10.1016/j.parkreldis.2015.08.008

Hebb, A. O., Zhang, J. J., Mahoor, M. H., Tsiokos, C., Matlack, C., Chizeck, H. J., & Pouratian, N. (2014). Creating the Feedback Loop: closed-loop neurostimulation. Neurosurgery Clinics of North America, 25(1), 187–204. https://doi.org/10.1016/j.nec.2013.08.006

Kassiri, H., Bagheri, A., Soltani, N., Abdelhalim, K., Jafari, H. M., Salam, M. T., … Genov, R. (2016). Battery-less Tri-band-Radio Neuro-monitor and Responsive Neurostimulator for Diagnostics and Treatment of Neurological Disorders. IEEE Journal of Solid-State Circuits, 51(5), 1274–1289. https://doi.org/10.1109/JSSC.2016.2528999

Kassiri, H., Tonekaboni, S., Salam, M. T., Soltani, N., Abdelhalim, K., Velazquez, J. L. P., & Genov, R. (2017). Closed-Loop Neurostimulators: A Survey and A Seizure-Predicting Design Example for Intractable Epilepsy Treatment. IEEE Transactions on Biomedical Circuits and Systems, 1–15. https://doi.org/10.1109/TBCAS.2017.2694638

Kostoglou, K., Michmizos, K. P., Stathis, P., Sakas, D., Nikita, K. S., & Mitsis, G. D. (2016). Classification and Prediction of Clinical Improvement in Deep Brain Stimulation from Intraoperative Microelectrode Recordings. IEEE Transactions on Biomedical Engineering, 1–1. https://doi.org/10.1109/TBME.2016.2591827

Lee, K. H., Kung, S.-Y., & Verma, N. (2012). Low-energy Formulations of Support Vector Machine Kernel Functions for Biomedical Sensor Applications. Journal of Signal Processing Systems, 69(3), 339–349. https://doi.org/10.1007/s11265-012-0672-8

Lemm, S., Blankertz, B., Dickhaus, T., & Müller, K.-R. (2011). Introduction to machine learning for brain imaging. NeuroImage, 56(2), 387–399. https://doi.org/10.1016/j.neuroimage.2010.11.004

Liu, X., Zhang, M., Richardson, A. G., Lucas, T. H., & Van der Spiegel, J. (2016). Design of a Closed-Loop, Bidirectional Brain Machine Interface System With Energy Efficient Neural Feature Extraction and PID Control. IEEE Transactions on Biomedical Circuits and Systems, 1–14. https://doi.org/10.1109/TBCAS.2016.2622738

Marceglia, S., Rossi, E., Rosa, M., Cogiamanian, F., Rossi, L., Bertolasi, L., … Priori, A. (2015). Web-based telemonitoring and delivery of caregiver support for patients with Parkinson disease after deep brain stimulation: protocol. JMIR Research Protocols, 4(1), e30. https://doi.org/10.2196/resprot.4044

Marković, D., & Brodersen, R. (2012). DSP Architecture Design Essentials. New York, NY: Springer.

Marshland, S. (2015). Machine Learning : an algorithmic perspective (2nd ed.). Boca Raton, FL: CRC Press.

Mason, S. G., & Birch, G. E. (2003). A general framework for brain-computer interface design. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 11(1), 70–85. https://doi.org/10.1109/TNSRE.2003.810426

Miyawaki, Y., Uchida, H., Yamashita, O., Sato, M., Morito, Y., Tanabe, H. C., … Kamitani, Y. (2008). Visual Image Reconstruction from Human Brain Activity using a Combination of Multiscale Local Image Decoders. Neuron, 60(5), 915–929. https://doi.org/10.1016/j.neuron.2008.11.004

Mohammed, A., & Demosthenous, A. (2018). Complementary Detection for Hardware Efficient On-Site Monitoring of Parkinsonian Progress. IEEE Journal on Emerging and Selected Topics in Circuits and Systems, 8(3), 603–615. https://doi.org/10.1109/JETCAS.2018.2830971

Mohammed, A., Zamani, M., Bayford, R., & Demosthenous, A. (2017). Toward On-Demand Deep Brain Stimulation Using Online Parkinson’s Disease Prediction Driven by Dynamic Detection. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 25(12), 2441–2452. https://doi.org/10.1109/TNSRE.2017.2722986

Muraskin, J., Sherwin, J., Lieberman, G., Garcia, J. O., Verstynen, T., Vettel, J. M., & Sajda, P. (2017). Fusing Multiple Neuroimaging Modalities to Assess Group Differences in Perception–Action Coupling. Proceedings of the IEEE, 105(1), 83–100. https://doi.org/10.1109/JPROC.2016.2574702

Pisotta, I., Perruchoud, D., & Ionta, S. (2015). Hand-in-hand advances in biomedical engineering and sensorimotor restoration. Journal of Neuroscience Methods, 246, 22–29. https://doi.org/10.1016/j.jneumeth.2015.03.003

Rabaey, J. M., Chandrakasan, A. P., & Nikolic, B. (2002). Digital Integrated Circuits - A Design Perspective (2nd ed.). Pearson.

Rhew, H., Jeong, J., Fredenburg, J. A., Member, S., Dodani, S., Patil, P. G., … Member, S. (2014). A Fully Self-Contained Logarithmic Closed-Loop Deep Brain Stimulation SoC With Wireless Telemetry and Wireless Power Management. 1–15.

Rouse, A. G., Stanslaski, S. R., Cong, P., Jensen, R. M., Afshar, P., Ullestad, D., … Denison, T. J. (2011). A chronic generalized bi-directional brain–machine interface. Journal of Neural Engineering, 8(3), 036018. https://doi.org/10.1088/1741-2560/8/3/036018

Schalk, G., Brunner, P., Gerhardt, L. A., Bischof, H., & Wolpaw, J. R. (2008). Brain–computer interfaces (BCIs): Detection instead of classification. Journal of Neuroscience Methods, 167(1), 51–62. https://doi.org/10.1016/j.jneumeth.2007.08.010

Secretary, W. H. P. (2013). Fact sheet: BRAIN Initiative. Retrieved February 1, 2017, from https://obamawhitehouse.archives.gov/the-press-office/2013/04/02/fact-sheet-brain-initiative

Sellers, E. W., Krusienski, D. J., McFarland, D. J., Vaughan, T. M., & Wolpaw, J. R. (2006). A P300 event-related potential brain–computer interface (BCI): The effects of matrix size and inter stimulus interval on performance. Biological Psychology, 73(3), 242–252. https://doi.org/10.1016/j.biopsycho.2006.04.007

Shanechi, M. M. (2019). Brain–machine interfaces from motor to mood. Nature Neuroscience, 22(10), 1554–1564. https://doi.org/10.1038/s41593-019-0488-y

Shulyzki, R., Abdelhalim, K., Bagheri, A., Salam, M. T., Florez, C. M., Velazquez, J. L. P., … Genov, R. (2015). 320-Channel Active Probe for High-Resolution Neuromonitoring and Responsive Neurostimulation. IEEE Transactions on Biomedical Circuits and Systems, 9(1), 34–49. https://doi.org/10.1109/TBCAS.2014.2312552

Skutkova, H., Vitek, M., Sedlar, K., & Provaznik, I. (2015). Progressive alignment of genomic signals by multiple dynamic time warping. Journal of Theoretical Biology, 385, 20–30. https://doi.org/10.1016/j.jtbi.2015.08.007

Vrieze, S. I. (2012). Model selection and psychological theory: a discussion of the differences between the Akaike information criterion (AIC) and the Bayesian information criterion (BIC). Psychological Methods, 17(2), 228–243. https://doi.org/10.1037/a0027127

Yargholi, E., & Hossein-Zadeh, G.-A. (2016). Brain Decoding-Classification of Hand Written Digits from fMRI Data Employing Bayesian Networks. Frontiers in Human Neuroscience, 10, 351. https://doi.org/10.3389/fnhum.2016.00351

Yoo, S.-S., Kim, H., Filandrianos, E., Taghados, S. J., & Park, S. (2013). Non-Invasive Brain-to-Brain Interface (BBI): Establishing Functional Links between Two Brains. PLoS ONE, 8(4), e60410. https://doi.org/10.1371/journal.pone.0060410


Refbacks

  • There are currently no refbacks.