Artificial Intelligence in Anatomic Pathology

  • Joshua J. Levy
    Corresponding author.
    Program in Quantitative Biomedical Sciences, Geisel School of Medicine at Dartmouth, Hanover, NH 03755, USA

    Department of Epidemiology, Geisel School of Medicine at Dartmouth, Hanover, NH 03755, USA

    Emerging Diagnostic and Investigative Technologies, Department of Pathology, Dartmouth Hitchcock Medical Center, 1 Medical Center Drive, Borwell Building Floor 4th, Lebanon, NH 03756-1000, USA
    Search for articles by this author
  • Louis J. Vaickus
    Emerging Diagnostic and Investigative Technologies, Department of Pathology, Dartmouth Hitchcock Medical Center, 1 Medical Center Drive, Borwell Building Floor 4th, Lebanon, NH 03756-1000, USA
    Search for articles by this author
Published:September 01, 2021DOI:
      Whole slide images (WSIs) are large digitized representations of tissue/cytology specimen and are of prohibitive dimensionality, costly to store and integrate with the clinical workflow in high throughput.


      To read this article in full you will need to make a payment


      Subscribe to Advances in Molecular Pathology
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect


        • Serag A.
        • Ion-Margineanu A.
        • Qureshi H.
        • et al.
        Translational AI and deep learning in diagnostic pathology.
        Front Med. 2019; 6: 185
        • Fischer A.H.
        • Jacobson K.A.
        • Rose J.
        • et al.
        Hematoxylin and eosin staining of tissue and cell sections.
        Cold Spring Harb Protoc. 2008; (pdb.prot4986 (2008))
        • Levy J.J.
        • Salas L.A.
        • Christensen B.C.
        • et al.
        PathFlowAI: a high-throughput workflow for preprocessing, deep learning and interpretation in digital pathology.
        Pac Symp Biocomput. 2019; 25: 403-414
        • Wei J.W.
        • Tafe L.J.
        • Linnik Y.A.
        • et al.
        Pathologist-level classification of histologic patterns on resected lung adenocarcinoma slides with deep neural networks.
        Sci Rep. 2019; 9: 3358
        • Jordan M.I.
        • Mitchell T.M.
        Machine learning: trends, perspectives, and prospects.
        Science. 2015; 349: 255-260
        • Chan S.
        • Reddy V.
        • Myers B.
        • et al.
        Machine learning in dermatology: current applications, opportunities, and limitations.
        Dermatol ther (Heidelb). 2020; 10: 365-386
        • Quinlan J.R.
        Induction of decision trees.
        Mach Learn. 1986; 1: 81-106
        • Breiman L.
        Random forests.
        Mach Learn. 2021; 45: 5-32
        • Chen T.
        • Guestrin C.
        XGBoost: a scalable tree boosting system.
        in: Proceedings of the 22Nd ACM SIGKDD international conference on knowledge discovery and data mining. Association for Computing Machinery, San Francisco2016: 785-794
        • Cover T.
        • Hart P.
        Nearest neighbor pattern classification.
        IEEE Trans Inf Theor. 1967; 13: 21-27
        • Lachenbruch P.A.
        • Sneeringer C.
        • Revo L.T.
        Robustness of the linear and quadratic discriminant function to certain types of non-normality.
        Comm Stat. 1973; 1: 39
        • Hesterberg T.
        • Choi N.H.
        • Meier L.
        • et al.
        Least angle and angle and nd certain types of non.
        Stat Surv. 2008; 2: 61-93
        • Hearst M.
        • Dumais S.T.
        • Osman E.
        • et al.
        Support vector machines.
        IEEE Intell Syst App. 1998; 13: 18-28
        • Wold S.
        • Esbensen K.
        • Geladi P.
        Principal component analysis.
        Chemometr Intell Lab Syst. 1987; 2: 37-52
        • Becht E.
        • McInnes L.
        • Healy J.
        • et al.
        Dimensionality reduction for visualizing single-cell data using UMAP.
        Nat Biotechnol. 2018; 37: 38
        • McInnes L.
        • Healy J.
        • Saul N.
        • et al.
        Uniform manifold approximation and projection.
        J Open Source Softw. 2018; 3: 861
        • Likas A.
        • Vlassis N.
        • Verbeek J.
        • et al.
        The global k-means clustering algorithm.
        Pattern Recognition. 2003; 36: 451-461
        • Reynolds D.
        Gaussian mixture models.
        in: Li S.Z. Jain A. Encyclopedia of biometrics. Springer, New York2009: 659-663
        • von Luxburg U.
        A tutorial on spectral clustering.
        Stat Comput. 2007; 17: 395-416
        • McInnes L.
        • Healy J.
        • Astels S.
        hdbscan: hierarchical density based clustering.
        J Open Source Softw. 2017; 2: 205
        • LeCun Y.
        • Bengio Y.
        • Hinton G.
        Deep learning.
        Nature. 2015; 521: 436-444
        • Esteva A.
        • Chou K.
        • Yeung S.
        • et al.
        Deep learning-enabled medical computer vision.
        NPJ Digital Med. 2021; 4: 1-9
        • Krizhevsky A.
        • Sutskever I.
        • Hinton G.E.
        ImageNet classification with deep convolutional neural networks.
        in: Advances in neural information processing systems. Twenty-sixth Conference on Neural Information Processing Systems, Lake Tahoe, Nevada. vol. 25. 2012: 1097-1105
        • Ching T.
        • Chou K.
        • Yeung S.
        • et al.
        Opportunities and obstacles for deep learning in biology and medicine.
        J R Soc Interf. 2018; 15: 20170387
        • Lo S.-C.B.
        • Chan H.P.
        • Lin J.S.
        • et al.
        Artificial convolution neural network for medical image pattern recognition.
        Neural Networks. 1995; 8: 1201-1214
        • Zhang A.
        • Lipton Z.C.
        • Li M.
        • et al.
        Dive into deep learning.
        2020 (Available at:)
        • Ronneberger O.
        • Fischer P.
        • Brox T.
        U-Net: convolutional networks for biomedical image segmentation.
        in: Navab N. Hornegger J. Wells W.M. Medical image computing and computer-assisted intervention al Networks. Springer Science+Business Media, New York2015: 234-241
        • Vuola A.O.
        • Akram S.U.
        • Kannala J.
        Mask-RCNN and U-Net ensembled for nuclei segmentation.
        in: 2019 IEEE 16th international Symposium on biomedical imaging (ISBI 2019). International Symposium on Biomedical Imaging, Venice, Italy2019: 208-219
        • Redmon J.
        • Farhadi A.
        YOLOv3: an incremental improvement.
        2018 (arXiv:1804.02767 [cs])
        • Bankhead P.
        • Loughrey M.B.
        • Fernández J.A.
        • et al.
        QuPath: open source software for digital pathology image analysis.
        Sci Rep. 2017; 7: 1-7
        • Perez L.
        • Wang J.
        The effectiveness of data augmentation in image classification using deep learning.
        2017 (arXiv:1712.04621 [cs])
        • Wei J.
        • Suriawinata A.
        • Vaickus L.
        • et al.
        Generative image translation for data augmentation in colorectal histopathology images.
        ML4H@NeurIPS MLR Press, Vancouver, Canada2019
      1. Macenko M. et al. A method for normalizing histology slides for quantitative analysis. In 2009 IEEE international symposium on biomedical imaging: from nano to macro 1107-1110. Boston, Massachusetts, 28 June-1 July, 2009.

        • Shallu
        • Mehra R.
        Breast cancer histology images classification: training from scratch or transfer learning?.
        ICT Express. 2018; 4: 247-254
        • Zkowski R.R.
        • Moowski M.
        • Zambonelli J.
        • et al.
        Accurate, reliable and active histopathological image classification framework with Bayesian deep learning.
        Sci Rep. 2019; 9: 14347
        • Heinemann F.
        • Birk G.
        • Stierstorfer B.
        Deep learning enables pathologist-like scoring of NASH models.
        Sci Rep. 2019; 9: 18454
        • Ramot Y.
        • Zandani G.
        • Madar Z.
        • et al.
        Utilization of a deep learning algorithm for microscope-based fatty vacuole quantification in a fatty liver model in mice.
        Toxicol Pathol. 2002; 48: 702-707
        • Davison B.A.
        • Harrison S.A.
        • Cotter G.
        • et al.
        Suboptimal reliability of liver biopsy evaluation has implications for randomized clinical trials.
        J Hepatol. 2020; 73: 1322-1332
        • Stringer C.
        • Wang T.
        • Michaelos M.
        • et al.
        Cellpose: a generalist algorithm for cellular segmentation.
        Nat Methods. 2021; 18: 100-106
        • Ziemys A.
        • Kim M.
        • Menzies A.M.
        • et al.
        Integration of digital pathologic and transcriptomic analyses connects tumor-infiltrating lymphocyte spatial density with clinical response to BRAF inhibitors.
        Front Oncol. 2020; 10: 757
        • Jackson C.R.
        • Sriharan A.
        • Vaickus L.J.
        A machine learning algorithm for simulating immunohistochemistry: development of SOX10 virtual IHC and evaluation on primarily melanocytic neoplasms.
        Modern Pathol. 2020; 33: 1638-1648
        • Hou L.
        • Gupta R.
        • Van Arnam J.S.
        • et al.
        Dataset of segmented nuclei in hematoxylin and eosin stained histopathology images of ten cancer types.
        Sci Data. 2020; 7: 185
        • Vaickus L.J.
        • Suriawinata A.A.
        • Wei J.W.
        • et al.
        Automating the paris system for urine cytopathologytopathologyancer types. C and evaluation on primar.
        Cancer Cytopathol. 2019; 127: 98-115
        • Karim M.R.
        • Beyan O.
        • Zappa A.
        • et al.
        Deep learning-based clustering approaches for bioinformatics.
        Brief Bioinform. 2021; 22: 393-415
        • Kingma D.P.
        • Welling M.
        Auto-encoding variational bayes.
        2014 (arXiv:1312.6114 [cs, stat])
        • Yamamoto Y.
        • Tsuzuki T.
        • Akatsuka J.
        • et al.
        Automated acquisition of explainable knowledge from unannotated histopathology images.
        Nat Commun. 2019; 10: 5642
        • Feng Y.
        • Zhang L.
        • Mo J.
        Deep manifold preserving autoencoder for classifying breast cancer histopathological images.
        IEEE/ACM Trans Comput Biol Bioinform. 2020; 17: 91-101
        • Hu B.
        • et al.
        Unsupervised learning for cell-level visual representation in histopathology images with generative adversarial networks.
        IEEE J Biomed Health Inform. 2018; 23: 1316-1328
        • Lu M.Y.
        • Chen R.J.
        • Mahmood F.
        Semi-supervised breast cancer histology classification using deep multiple instance learning and contrast predictive coding (Conference Presentation).
        in: Tomaszewski J. Ward A. Medical imaging 2020: digital pathology. vol. 11320. International Society for Optics and Photonics, Houston, TX2020: 113200J
        • Chen R.J.
        • et al.
        Pathomic fusion: an integrated framework for fusing histopathology and genomic features for cancer diagnosis and prognosis.
        • Ianni J.D.
        • Soans R.E.
        • Sankarapandian S.
        • et al.
        Tailored for real-world: a whole slide image classification system validated on uncurated multi-site data emulating the prospective pathology workload.
        Sci Rep. 2020; 10: 3217
        • Campanella G.
        • Hanna M.G.
        • Geneslaw L.
        • et al.
        Clinical-grade computational pathology using weakly supervised deep learning on whole slide images.
        Nat Med. 2019; 25: 1301-1309
        • Lu M.Y.
        • Williamson D.F.K.
        • Chen T.Y.
        • et al.
        Data-efficient and weakly supervised computational pathology on whole-slide images.
        Nat Biomed Eng. 2021; 5: 555-570
        • Lu M.Y.
        • Chen T.Y.
        • Williamson D.F.K.
        • et al.
        AI-based pathology predicts origins for cancers of unknown primary.
        Nature. 2021; 594: 106-110
        • Tomita N.
        • Abdollahi B.
        • Wei J.
        • et al.
        Attention-based deep neural networks for detection of cancerous and precancerous esophagus tissue on histopathological slides.
        JAMA Netw Open. 2019; 2: e1914645
        • Levy J.
        • Haudenschild C.
        • Barwick C.
        • et al.
        Topological feature extraction and visualization of whole slide images using graph neural networks.
        Pac Symp Biocomput. 2020; : 285-296
        • Yamashita R.
        • Levy J.
        • Haudenschild C.
        • Barwick C.
        • et al.
        Deep learning model for the prediction of microsatellite instability in colorectal cancer: a diagnostic study.
        Lancet Oncol. 2021; 22: 132-141
        • Adnan M.
        • Kalra S.
        • Tizhoosh H.R.
        Representation learning of histopathology images using graph neural networks. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops..
        2020: 988-989
        • Ash J.T.
        • Darnell G.
        • Munro D.
        • et al.
        Joint analysis of expression levels and histological images identifies genes associated with tissue morphology.
        Nat Commun. 2021; 12: 1609
        • Carmichael I.
        • et al.
        Joint and individual analysis of breast cancer histologic images and genomic covariates.
        2019 (arXiv:1912.00434 [eess, q-bio, stat])
        • Zheng H.
        • Momeni A.
        • Cedoz P.-L.
        • et al.
        Whole slide images reflect DNA methylation patterns of human tumors.
        NPJ Genomic Med. 2020; 5: 11
        • Hao J.
        • Kosaraju S.
        • Tsaku N.
        • et al.
        Interpretable and integrative deep learning for survival analysis using histopathological images and genomic data.
        Pac Symp Biocomputing. 2020; 25: 355-366
        • Zhan Z.
        • Jing Z.
        • He B.
        • et al.
        Two-stage Cox-nnet: biologically interpretable neural-network model for prognosis prediction and its application in liver cancer survival using histopathology and transcriptomic data.
        NAR Genom Bioinform. 2021; 3: lqab015
        • Cheerla A.
        • Gevaert O.
        Deep learning with multimodal representation for pancancer prognosis prediction.
        Bioinformatics. 2019; 35: i446-i454
        • de Vries N.L.
        • Mahfouz A.
        • Koning F.
        • et al.
        Unraveling the complexity of the cancer microenvironment with multidimensional genomic and cytometric technologies.
        Front Oncol. 2020; 10: 1254
        • Zhang M.
        • Sheffield T.
        • Zhan X.
        • et al.
        Spatial molecular profiling: platforms, applications and analysis tools.
        Brief Bioinform. 2021; 22: bbaa145
      2. Van TM, Blank CU. A user’s perspective on GeoMxTM digital spatial profiling. Immuno-Oncology Technol 2019;1:11-8.

        • Goytain A.
        • Ng T.
        NanoString nCounter technology: high-throughput RNA validation.
        in: Li H. Elfman J. Chimeric RNA: methods and protocols. Springer Nature, New York2020: 125-139
        • Tan X.
        • Su A.
        • Tran M.
        • et al.
        SpaCell: integrating tissue morphology and spatial gene expression to predict disease cells.
        Bioinformatics. 2020; 36: 2293-2294
        • He B.
        • Bergenstråhle L.
        • Stenbeck L.
        • et al.
        Integrating spatial gene expression and breast tumour morphology via deep learning.
        Nat Biomed Eng. 2020; 4: 827-834
        • Levy J.J.
        • Jackson C.R.
        • Haudenschild C.C.
        • et al.
        PathFlow-mixmatch for whole slide image registration: an investigation of a segment-based scalable image registration method.
        bioRxiv. 2020;
        • Paknezhad M.
        • et al.
        Regional registration of whole slide image stacks containing highly deformed artefacts.
        2020 (arXiv:2002.12588 [cs, eess])
        • Senior A.W.
        • Evans R.
        • Jumper J.
        • et al.
        Improved protein structure prediction using potentials from deep learning.
        Nature. 2020; 577: 706-710
        • Levy J.
        • Jackson C.
        • Sriharan A.
        • et al.
        Preliminary evaluation of the utility of deep generative histopathology image translation at a Mid-sized NCI Cancer Center.
        in: Proceedings of the 13th International Joint Conference on Biomedical Engineering Systems and Technologies (BIOSTEC 2020). vol. 3. Bioinformatics SCITEPRESS, Valletta, Malta2021: 30
        • Pichat J.
        • Iglesias J.E.
        • Yousry T.
        • et al.
        A survey of methods for 3D histology reconstruction.
        Med Image Anal. 2018; 46: 73-105
        • Goodfellow I.
        • Pouget-Abadie J.
        • Mirza M.
        • et al.
        Generative adversarial networks.
        Advances in Neural Information Processing Systems. 2014; 3
      3. Zhu JY, Park T, Isola P, et al. Unpaired image-to-image translation using cycle-consistent adversarial networks. in Proceedings of the IEEE international conference on computer vision. Venice, Italy: IEEE; 2017;2223–2232.

        • Levy J.J.
        • Azizgolshani N.
        • Andersen M.J.
        • et al.
        A large-scale internal validation study of unsupervised virtual trichrome staining technologies on nonalcoholic steatohepatitis liver biopsies.
        Mod Pathol. 2021; 34: 808-822
      4. Ghazvinian Zanjani F, Zinger S, Ehteshami Bejnordi B, et al. Stain normalization of histopathology images using generative adversarial networks. In 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018), Washington DC.

      5. Sundararajan M, Taly A, Yan Q. Axiomatic attribution for deep networks. in Proceedings of the 34th International Conference on Machine Learning. Sydney, Australia: Association for Computing Machinery; 017;70: 3319–28

        • Lundberg S.M.
        • Erion G.
        • Chen H.
        • et al.
        From local explanations to global understanding with explainable AI for trees.
        Nat Machine Intell. 2020; 2: 56-67
        • Devlin J.
        • Chang M.-W.
        • Lee K.
        • et al.
        BERT: pre-training of deep bidirectional transformers for language understanding.
        in: Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies. vol. 1. Association for Computational Linguistics, Minneapolis, Minnesota2019: 4171 (Long and short papers)
        • Levy J.
        • Vattikonda N.
        • Haudenschild C.
        • et al.
        Comparison of machine learning algorithms for the prediction of current procedural terminology (CPT) codes from pathology reports.
        medRxiv. 2021;
        • Tosun A.B.
        • Pullara F.
        • Becich M.J.
        • et al.
        Explainable AI (xAI) for anatomic pathology.
        Adv Anat Pathol. 2020; 27: 241-250
        • Bürkner P.C.
        Advanced Bayesian Multilevel Modeling with the R Package brms.
        R J. 2018; 10: 395
        • Bürkner P.C.
        brms: an R package for bayesian multilevel models using Stan.
        J Stat Softw. 2017; 80: 1-28
        • McElreath R.
        Statistical rethinking: a Bayesian course with examples in R and Stan.
        CRC Press, Boca Raton, Florida2020
        • Young A.T.
        • Fernandez K.
        • Pfau J.
        • et al.
        Stress testing reveals gaps in clinic readiness of image-based diagnostic artificial intelligence models.
        NPJ Digit Med. 2021; 4: 10
        • Kompa B.
        • Snoek J.
        • Beam A.L.
        Second opinion needed: communicating uncertainty in medical machine learning.
        NPJ Digital Med. 2021; 4: 4
        • Cabitza F.
        • Ciucci D.
        • Rasoini R.
        A giant with feet of clay: on the validity of the data that feed machine learning in medicine.
        in: Cabitza F. Batini C. Magni M. Organizing for the digital world. Springer International Publishing, New York2019: 121-136
        • Djulbegovic B.
        • Paul A.
        From efficacy to effectiveness in the face of uncertainty: indication creep and prevention creep.
        JAMA. 2011; 305: 2005-2009
        • Begoli E.
        • Bhattacharya T.
        • Kusnezov D.
        The need for uncertainty quantification in machine-assisted medical decision making.
        Nat Mach Intell. 2019; 1: 20-23
        • Pasetto S.
        • Gatenby R.A.
        • Enderling H.
        Bayesian framework to augment tumor board decision making.
        JCO Clin Cancer Inform. 2021; 5: 508-517
        • Gerke S.
        • Minssen T.
        • Cohen G.
        Ethical and legal challenges of artificial intelligence-driven healthcare.
        Artif Intell Healthc. 2020; : 295-336
        • Razavian N.
        Augmented reality microscopes for cancer histopathology.
        Nat Med. 2019; 25: 1334-1336
        • Grote T.
        • Berens P.
        On the ethics of algorithmic decision-making in healthcare.
        J Med Ethics. 2020; 46: 205-211
        • Rigby M.J.
        Ethical dimensions of using artificial intelligence in health care.
        AMA J Ethics. 2019; 21: 121-124
        • Jackson B.R.
        • Ye Y.
        • Crawford J.M.
        • et al.
        The Ethics of Artificial Intelligence in Pathology and Laboratory Medicine: Principles and Practice..
        Acad Pathol. 2021; 8
        • Abras C.
        • Maloney-krichmar D.
        • Preece J.
        User-centered design.
        in: Bainbridge W. Encyclopedia of human-computer interaction. Sage Publications (Publications), Thousand Oaks2004
        • Lu M.Y.
        • et al.
        Federated learning for computational pathology on gigapixel whole slide images.
        2020 (arXiv:2009.10190 [cs, eess, q-bio])
        • Warnat-Herresthal S.
        • Schultze H.
        • Shastry K.L.
        • et al.
        Swarm learning for decentralized and confidential clinical machine learning.
        Nature. 2021; 594: 265-270