ECML 2023 Tutorial -- Revolutionizing Anomaly Detection: Approaches and Guidelines Generation
Anomaly detection is a significant task of data mining, and also a hot research topic in various fields of artificial intelligence in recent years. It has a wide range of applications, such as extreme climate event detection, mechanical fault detection, terrorist detection, fraud detection, malicious URL detection etc.
This tutorial aims to present a comprehensive review of both shallow and deep-learning-based anomaly detection with an explanation. We first introduce the key intuitions, objective functions, underlying assumptions, and advantages and disadvantages of state-of-the-art anomaly detection methods. We also introduce several principled approaches used to provide anomaly explanations for deep detection models. Furthermore, we will discuss the connections between classic shallow and novel deep methods and provide a practical guide on how to select an outlier detector in different applications.
Background
There are many kinds of classic shallow anomaly detection methods proposed to solve the problem of anomaly detection in various scenarios. However, the explosive growth of databases in both size and dimensionality is challenging for anomaly detection methods in two important aspects: the requirement of low computational cost and the susceptibility to high-dimensionality issues. Efficient methods are in high demand for time-critical applications ranging from network intrusion detection to credit card fraud detection.
Recently, deep learning has shown Phenomenal success in tackling these complexities in a wide range of applications, but popular deep learning techniques are inapplicable to anomaly detection due to some unique characteristics of anomalies, e.g., rarity, heterogeneity, unbounded nature, and prohibitively high cost of collecting large-scale anomaly data. A large number of studies, therefore, have been dedicated to deep methods specifically designed for anomaly detection. These studies demonstrate great success in addressing some major challenges to which shallow anomaly detection methods fail in different application contexts.
Specific goals and content
This tutorial presents a comprehensive coverage of both shallow and deep learning-based anomaly detection and explanation, including hands-on practices and a practitioner's guide on how to select suitable outlier detectors in various real-world applications.
Through this tutorial, we aim to promote the research development in algorithms, theories and evaluation of explainable shallow and deep anomaly detection in the data mining and machine learning community.
Expected background of the audience
The tutorial has no prerequisites but general knowledge of data analytics is desired. This tutorial is intended to draw the attention of students and researchers who are interested in anomaly detection and have basic machine-learning skills, and statistical and mathematic backgrounds.
Researchers and practitioners in finance, cybersecurity, and healthcare would also find the tutorial helpful in practice.
You can preview and download the tutorial slides here
Tutorial Outline
1. Overview of challenges and methods (1 hour)
• Problem definition and applications
• Challenges
• Overview of anomaly detection approaches
• Shallow vs Deep methods
2. Shallow anomaly detection models (1 hour)
• Probabilistic models
• Distance/Density-based models
• Isolation-based models
• Hands-on Activity (Code Demonstration)
Break for 0.5 hour
3. Deep anomaly detection models (1 hour)
• The modelling and supervision information
• Anomaly explanation in deep detectors
• Hands-on Activity (Code Demonstration)
4. Conclusions and Future opportunities (0.5 hours)
• Summary and Practical Guide
• Directions for future research
• Q & A
Tutorial Slides
You can preview and download the tutorial slides here
Code for demonstration
The TAD's code for Tutorial is available here
The Isolation-Kernel's code for Tutorial is available here
The Pang's code for Tutorial is available here
The pyod's code for Tutorial is available here
Tutorial Presenters
Dr. Ye Zhu (primary contact, will present Part 2)
A/Prof. Mark Carman (will present Part 1)
Prof. Gang Li (will present Part 3 & 4)
Dr. Guansong Pang
Dr. Xuyun Zhang
Presenters' CV
Dr. Ye Zhu
Senior Lecturer of Computer Science
School of Information Technology
Deakin University, Burwood, VIC Australia 3125
Email: [email protected]
Phone: +61 3 924 68039
Website: https://www.deakin.edu.au/about-deakin/people/ye-zhu
Dr. Ye Zhu is a Senior Lecturer of Computer Science at the School of Information Technology at Deakin University. He is an IEEE senior member and ACM member. He received a PhD degree in Artificial Intelligence with a Mollie Holman Medal for the best doctoral thesis of the year from Monash University in May 2017. Dr Zhu joined Deakin University as a post-doc research fellow in complex system data analytics in July 2017 and then became a lecturer in Feb 2019. He was promoted to Senior Lecturer in May 2022. His research works mainly focus on clustering analysis, anomaly detection, similarity learning and their applications for pattern recognition. Dr Zhu has published over 40 papers in top-tier conferences and journals, including SIGKDD, IJCAI, VLDB, AAAI, TKDE, AIJ, ISJ, PRJ, JAIR and MLJ. He has served as Program Chair and Program Committee for many prestigious international conferences, such as SIGKDD, AAAI, IJCAI and PAKDD. He was invited to give a talk and participate in a one-week intensive discussion at NII Shonan Meeting #123 in Japan on the topic of “data-dependent dissimilarity” in October 2018. He organised and chaired the workshop “Anomaly and Novelty Detection, Explanation and Accommodation” at SIGKDD in August 2022. He also conducted a tutorial focusing on “How to Select Shallow and Deep Methods for Anomaly Detection” at the 2022 International Conference on Advanced Data Mining and Applications (ADMA) in November 2022. Furthermore, he has secured several large grants for interdisciplinary and industrial research. He recently obtained both an Early Career Researcher Award and a Teaching and Learning Award from Deakin University.
A/Prof. Mark Carman
Associate professor of Data Science and Artificial Intelligence
Dipartimento di Elettronica, Informazione e Bioingegneria (DEIB)
Politecnico di Milano (The Polytechnic University of Milan)
Office 023, Building 20, Via Ponzio 34/5, 20133 Milano, Italia
Email: [email protected]
Phone: +39 02 2399 3628
Website: https://www.deib.polimi.it/eng/people/details/1439980
A/Prof. Carman's research interests lie in the areas of information retrieval and machine learning. His specific areas of expertise include rank learning techniques for Web search engines, quality control techniques for crowdsourcing, statistical modelling of text for sentiment and sarcasm detection, models of user expertise online, topic models for personalising search results, product recommendation, clustering & anomaly detection techniques, text extraction from images, and digital forensics. Moreover, A/Prof. Carman has experience working with and/or extending a large variety of machine learning techniques including Bayesian networks, graphical and topic modelling, non-parametric models and Bayesian statistics, inductive logic programming, regression tree ensembles, large scale optimisation/learning problems, time series models, self-exciting processes, bandit algorithms and various areas of deep learning (applied to both images and text).
Prof. Gang Li
Professor of Computer Science
School of Information Technology
Deakin University, Burwood, VIC Australia 3125
Email: [email protected]
Phone: +61 3 925 17434
Website: https://www.deakin.edu.au/about-deakin/people/gang-li
Prof. Gang Li is currently a Professor at the School of IT, Deakin University, Australia. He is an internationally recognized expert in data mining and business intelligence. He has supervised and co-supervised 18 PhD students to completion, and published over 200 papers in the project area, with Google scholar citations of 6,000+ and an h-index of 35. CI Li holds one international patent and has substantial research experience in graphical models and privacy-aware data science. Her proposed preference model method won the IEEE/ACM ASONAM 2012 Best Paper Award, and his series work on trajectory analysis won the IFITT award for Journal Paper of the Year (2015 and 2017). He has developed a mechanism for privacy preservation that retrieves user rating information while retaining anonymity in product and tag recommendation systems. We have also investigated information loss in coupled datasets and proposed an improved differential privacy mechanism to accommodate coupled datasets. These works have been presented at prestigious conferences and published in journals, including IEEE Transactions on Information Forensics and Security, and IEEE Transactions on Data and Knowledge Engineering, and have won the best student award at PAKDD. He is an associate editor for Decision Support Systems (Elsevier), the Journal of Information & Knowledge Management (World Scientific) and Cyber Security (Springer).
Dr. Guansong Pang
Assistant Professor
School of Computing and Information Systems
Singapore Management University
Room 5035, 80 Stamford Rd, Singapore 178902
Email: [email protected]
Phone: +65 6826 4864
Website: https://faculty.smu.edu.sg/profile/guansong-pang-6271
Dr. Guansong Pang obtained his PhD degree in Data Mining at the University of Technology Sydney in 2019. He was a Research Fellow in the Australian Institute for Machine Learning at the University of Adelaide, and now is an Assistant Professor at Singapore Management University. His research interests lie in data mining, machine learning and their applications; he has been dedicated to the research on anomaly detection for over six years. He has published more than 50 papers (most of them are on (deep) anomaly detection) in refereed conferences and journals, such as KDD, AAAI, IJCAI, CVPR, ACM MM, ICDM, CIKM, IEEE Transactions on Knowledge and Data Engineering, and Data Mining and Knowledge Discovery Journal. He is one of the key presenters of the KDD17's tutorial on "Non-IID Learning", the KDD18's tutorial on “Behavior Analytics: Methods and Applications” and the KDD21’s tutorial on “Toward Explainable Deep Anomaly Detection”. He also gives many oral representations of his papers at top conferences such as IJCAI16, IJCAI17, CIKM17, KDD18, and KDD19 and invited talks at various universities.
Dr. Xuyun Zhang
Senior Lecturer
School of Computing
Macquarie University, Sydney, NSW Australia 2109
Email: [email protected]
Phone: +61 2 9850 8229
Website: https://researchers.mq.edu.au/en/persons/xuyun-zhang
Dr Xuyun Zhang is currently working as a senior lecturer and a DECRA research fellow in the School of Computing at Macquarie University (Sydney, Australia). Besides, he has working experience at the University of Auckland and NICTA (now Data61, CSIRO). He received his PhD degree in Computer and Information Science from the University of Technology Sydney (UTS) in 2014, and his MEng and BSc degrees from Nanjing University. His research interests include scalable anomaly detection and data mining, private and secure machine learning, data privacy and cyber security, edge-cloud computing, etc. He has so far published 100+ high-quality internationally refereed publications in the relevant prestigious journals and conferences such as ACM Computing Surveys, IEEE TPDS, TKDE, TII, TDSC, TBD, ICDE, ICDM, AAAI, IJCAI, SIGIR, WSDM, CIKM, etc. He has been listed as one of the Clarivate 2021 Highly Cited Researchers. He has proposed to investigate isolation forest-based anomaly detection from the perspective of similarty hashing to understand the effectiveness of isolation forest mechanism for anomaly detection and to enhance the scalability and applicability of isolation forest-based anomaly detection.
• Bandaragoda, T. R., Ting, K. M., Albrecht, D., Liu, F. T., Zhu, Y., & Wells, J. R. (2018). Isolation-based anomaly detection using nearest-neighbor ensembles. Computational Intelligence, 34(4), 968-998.
• Liu, F. T., Ting, K. M., & Zhou, Z. H. (2008, December). Isolation forest. In 2008 Eighth IEEE International Conference on Data Mining, Pisa, Italy, 2008, pp. 413-422.
• Pang, G., Li, J., van den Hengel, A., Cao, L., & Dietterich, T. G. (2022, August). ANDEA: anomaly and novelty detection, explanation, and accommodation. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (pp. 4892-4893).
• Pang, G., Shen, C., Cao, L., & Hengel, A. V. D. (2021). Deep learning for anomaly detection: A review. ACM computing surveys (CSUR), 54(2), 1-38.
• Pang, G., van den Hengel, A., Shen, C., & Cao, L. (2021, August). Toward deep supervised anomaly detection: Reinforcement learning from partially labeled anomaly data. In Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining (pp. 1298-1308).
• Pang, G., & Aggarwal, C. (2021, August). Toward explainable deep anomaly detection. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining (pp. 4056-4057).
• Pang, G., Shen, C., & van den Hengel, A. (2019, July). Deep anomaly detection with deviation networks. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining (pp. 353-362).
• Qin, X., Ting, K. M., Zhu, Y., & Lee, V. C. (2019, July). Nearest-neighbour-induced isolation similarity and its impact on density-based clustering. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 33, No. 01, pp. 4755-4762).
• Ruff, L., Kauffmann, J. R., Vandermeulen, R. A., Montavon, G., Samek, W., Kloft, M., ... & Müller, K. R. (2021). A unifying review of deep and shallow anomaly detection. Proceedings of the IEEE, 109(5), 756-795.
• Ting, K. M., Liu, Z., Zhang, H., & Zhu, Y. (2022a). A new distributional treatment for time series and an anomaly detection investigation. Proceedings of the VLDB Endowment, 15(11), 2321-2333.
• Ting, K. M., Xu, B. C., Washio, T., & Zhou, Z. H. (2021). Isolation Distributional Kernel A New Tool for Point & Group Anomaly Detection. IEEE Transactions on Knowledge and Data Engineering. DOI: 10.1109/TKDE.2021.3120277.
• Zhu, Y., Ting, K. M., Carman, M. J., & Angelova, M. (2021). CDF Transform-and-Shift: An effective way to deal with datasets of inhomogeneous cluster densities. Pattern Recognition, 117, 107977.
• Zhang. X., Dou, W., He, Q., Zhou, R., Leckie, C., Kotagiri, R., & Salcic, Z. (2017). LSHiForest: a generic framework for fast tree isolation based ensemble anomaly analysis. In Proceedings of the IEEE 33rd International Conference on Data Engineering (ICDE) (PP. 983-994).
• Xiang, H., Hu, H., & Zhang, X., (2022) DeepiForest: A deep anomaly detection framework with hashing based isolation forest. In Proceeding of the 22nd IEEE International Conference on Data Mining (ICDM) (pp. 1251-1256).
• Salehi, M., Leckie, C., Bezdek, J. C. , Vaithianathan, T. & Zhang, X. (2016). Fast Memory Efficient Local Outlier Detection in Data Streams. IEEE Transactions on Knowledge and Data Engineering, 28(12), 3246-3260.
