Ensemble learning involves combining the predictions from multiple machine learning models.

The effect can be both improved predictive performance and lower variance of the predictions made by the model.

Ensemble methods are covered in most textbooks on machine learning; nevertheless, there are books dedicated to the topic.

In this post, you will discover the top books on the topic of ensemble machine learning.

After reading this post, you will know:

• Books on ensemble learning, including their table of contents and where to learn more about them.
• Sections and chapters on ensemble learning in the most popular and common machine learning textbooks.
• Book recommendations for machine learning practitioners interested in ensemble learning.

Let’s get started.

Ensemble Learning Book List

The books dedicated to the topic of ensemble learning that we will cover are as follows:

1. Supervised and Unsupervised Ensemble Methods and their Applications, 2008.
2. Pattern Classification Using Ensemble Methods, 2010.
3. Ensemble Learning, 2019.
4. Ensemble Methods in Data Mining, 2010.
5. Ensemble Methods, 2012.
6. Ensemble Machine Learning, 2012.

There are also some books from Packt, but I won’t be reviewing them; they are:

Did I miss a book on ensemble learning?
Let me know in the comments below.

Have you read any of these books on ensemble learning?
What did you think? Let me know in the comments.

Let’s take a closer look at these books, including their author, table of contents, and where to learn more.

Supervised and Unsupervised Ensemble Methods and their Applications

The full title of this book is “Supervised and Unsupervised Ensemble Methods and their Applications” and it was edited by Oleg Okun and Giorgio Valentini and published in 2008.

Supervised and Unsupervised Ensemble Methods and their Applications

This book is a collection of academic papers by a range of different authors on the topic of applications of ensemble learning.

The book includes nine chapters divided into two parts, assembling contributions to the applications of supervised and unsupervised ensembles.

— Page VIII, Supervised and Unsupervised Ensemble Methods and their Applications, 2008.

Table of Contents

• Part I: Ensembles of Clustering Methods and Their Applications
  • Chapter 01: Cluster Ensemble Methods: From Single Clusterings to Combined Solutions
  • Chapter 02: Random Subspace Ensembles for Clustering Categorical Data
  • Chapter 03: Ensemble Clustering with a Fuzzy Approach
  • Chapter 04: Collaborative Multi-Strategical Clustering for OBject-Oriented Image Analysis
• Part II: Ensembles of Classification Methods and Their Applications
  • Chapter 05: Intrusion Detection in Computer Systems Using Multiple Classifier Systems
  • Chapter 06: Ensembles of Nearest Neighbors for Gene Expression Based Cancer Classification
  • Chapter 07: Multivariate Time Series Classification via Stacking of Univariate Classifiers
  • Chapter 08: Gradient Boosting GARCH and Neural Networks for Time Series Prediction
  • Chapter 09: Cascading with VDM and Binary Decision Trees for Nominal Data

I generally would not recommend this book to machine learning practitioners unless one of the applications covered by the book is directly related to your current project.

You can learn more about this book here:

Pattern Classification Using Ensemble Methods

The full title of this book is “Pattern Classification Using Ensemble Methods” and it was written by Lior Rokach and published in 2010.

Pattern Classification Using Ensemble Methods

This book provides a technical introduction to the topic of ensemble machine learning written for students and academics.

Throughout the book, special emphasis was put on the extensive use of illustrative examples. Accordingly, in addition to ensemble theory, the reader is also provided with an abundance of artificial as well as real-world applications from a wide range of fields. The data referred to in this book, as well as most of the Java implementations of the presented algorithms, can be obtained via the Web.

— Page viii, Pattern Classification Using Ensemble Methods, 2010.

Table of Contents

• Chapter 01: Introduction to Pattern Classification
• Chapter 02: Introduction to Ensemble Learning
• Chapter 03: Ensemble Classification
• Chapter 04: Ensemble Diversity
• Chapter 05: Ensemble Selection
• Chapter 06: Error Correcting Output Codes
• Chapter 07: Evaluating Ensembles of Classifiers

I like the level of this book. It is technical, but not overly so, and stays grounded in the concerns of using ensemble algorithms on supervised predictive modeling projects. I think it is a good textbook on ensemble learning for practitioners.

You can learn more about this book here:

Ensemble Learning

The full title of this book is “Ensemble Learning: Pattern Classification Using Ensemble Methods” and it was written by Lior Rokach and published in 2019.

Ensemble Learning Pattern Classification Using Ensemble Methods

This is a direct update to the book “Pattern Classification Using Ensemble Methods” and given a different title.

The first edition of this book was published a decade ago. The book was well-received by the machine learning and data science communities and was translated into Chinese. […] The second edition aims to update the previously presented material on the fundamental areas, and to present new findings in the field; more than a third of this edition is comprised of new material.

— Page vii, Ensemble Learning: Pattern Classification Using Ensemble Methods, 2019.

Table of Contents

• Chapter 01: Introduction to Machine Learning
• Chapter 02: Classification and Regression Trees
• Chapter 03: Introduction to Ensemble Learning
• Chapter 04: Ensemble Classification
• Chapter 05: Gradient Boosting Machines
• Chapter 06: Ensemble Diversity
• Chapter 07: Ensemble Selection
• Chapter 08: Error Correcting Output Codes
• Chapter 09: Evaluating Ensemble Classifiers

This is a great textbook on ensemble learning for students and practitioners and is preferred over “Pattern Classification Using Ensemble Methods” if you must choose between the two.

You can learn more about this book here:

Ensemble Methods in Data Mining

The full title of this book is “Ensemble Methods in Data Mining: Improving Accuracy
Through Combining Predictions” and it was written by Giovanni Seni and John Elder and published in 2010.

Ensemble Methods in Data Mining

This is a technical book on ensembles, although concepts are demonstrated with full examples in R.

This book is aimed at novice and advanced analytic researchers and practitioners – especially in Engineering, Statistics, and Computer Science. Those with little exposure to ensembles will learn why and how to employ this breakthrough method, and advanced practitioners will gain insight into building even more powerful models. Throughout, snippets of code in R are provided to illustrate the algorithms described and to encourage the reader to try the technique.

— Page i, Ensemble Methods in Data Mining, 2010.

Table of Contents

• Chapter 01: Ensembles Discovered
• Chapter 02: Predictive Learning and Decision Trees
• Chapter 03: Model Complexity, Model Selection and Regularization
• Chapter 04: Importance Sampling and the Classic Ensemble Methods
• Chapter 05: Rule Ensembles and Interpretation Statistics
• Chapter 06: Ensemble Complexity
• Appendix A: AdaBoost Equivalence to FSF Procedure
• Appendix B: Gradient Boosting and Robust Loss Functions

I believe this is the first book I purchased on ensemble learning years ago. It is a good crash course in ensemble learning for practitioners, especially those already using R. It may be a little too mathematical for most practitioners; nevertheless, I think it might be a good smaller substitute for the above textbook on ensemble methods.

You can learn more about this book here:

Ensemble Methods

The full title of this book is “Ensemble Methods: Foundations and Algorithms” and it was written by Zhi-Hua Zhou and published in 2012.

Ensemble Methods: Foundations and Algorithms

This is another focused textbook on the topic of ensemble learning targeted at students and academics.

This book provides researchers, students and practitioners with an introduction to ensemble methods. The book consists of eight chapters which naturally constitute three parts.

— Page vii, Ensemble Methods: Foundations and Algorithms, 2012.

Table of Contents

• Chapter 01: Introduction
• Chapter 02: Boosting
• Chapter 03: Bagging
• Chapter 04: Combination Methods
• Chapter 05: Diversity
• Chapter 06: Ensemble Pruning
• Chapter 07: Clustering Ensembles
• Chapter 08: Advanced Topics

This book is well written and covers the main methods with good references. I think it’s another great jump-start on the basics of ensemble methods as long as the reader is comfortable with a little math. I liked the algorithm descriptions and worked examples.

You can learn more about this book here:

Ensemble Machine Learning

The full title of this book is “Ensemble Machine Learning: Methods and Applications” and it was edited by Cha Zhang and Yunqian Ma and published in 2012.

Ensemble Machine Learning

This book is a collection of academic papers written by a range of authors on the topic of applications of ensemble machine learning.

Despite the great success of ensemble learning methods recently, we found very few books that were dedicated to this topic, and even fewer that provided insights about how such methods shall be applied in real-world applications. The primary goal of this book is to fill the existing gap in the literature and comprehensively cover the state-of-the-art ensemble learning methods, and provide a set of applications that demonstrate the various usages of ensemble learning methods in the real world.

— Page v, Ensemble Machine Learning: Methods and Applications, 2012.

Table of Contents

• Chapter 01: Ensemble Learning
• Chapter 02: Boosting Algorithms: A Review of Methods, Theory, and Applications
• Chapter 03: Boosting Kernel Estimators
• Chapter 04: Targeted Learning
• Chapter 05: Random Forests
• Chapter 06: Ensemble Learning by Negative Correlation Learning
• Chapter 07: Ensemble Nystrom
• Chapter 08: Object Detection
• Chapter 09: Classifier Boosting for Human Activity Recognition
• Chapter 10: Discriminative Learning for Anatomical Structure Detection and Segmentation
• Chapter 1: Random Forest for Bioinformatics

Like other collections of papers, I would generally not recommend this book unless you are an academic or one of the chapters is directly related to your current machine learning project. Nevertheless, many of the chapters provide a solid and compact introduction to ensemble methods and how to use them on specific applications.

You can learn more about this book here:

Book Chapters

Many machine learning textbooks have sections on ensemble learning.

In this section, we will take a quick tour of some of the more popular textbooks and the relevant sections on ensemble learning.

The book “An Introduction to Statistical Learning with Applications in R” published in 2016 provides a solid introduction to boosting and bagging for decision trees in chapter 8.

• Section 8.2: Bagging, RandomForests, Boosting

The book “Applied Predictive Modeling” published in 2013 covers the most popular ensemble algorithms with examples in R, with a focus on the ensembles of decision trees.

• Chapter 8: Regression Trees and Rule-Based Model
• Chapter 14: Classification Trees and Rule-Based Models

The book “Data Mining: Practical Machine Learning Tools and Techniques” published in 2016 provides a chapter dedicated to ensemble learning and covers a range of popular techniques, including boosting, bagging, and stacking.

• Chapter 12: Ensemble Learning.

The book “Machine Learning: A Probabilistic Perspective” published in 2012 provides a number of sections on algorithms that perform ensembling, as well as a dedicated section on the topic focused on stacking and error-correcting output codes.

• Section 16.2: Classification and regression trees (CART)
• Section 16.4: Boosting
• Section 16.6: Ensemble learning

The book “The Elements of Statistical Learning” published in 2016 covers the key ensemble learning algorithms as well as the theory for ensemble learning generally.

• Chapter 8: Model Inference and Averaging
• Chapter 10: Boosting and Additive Trees
• Chapter 15: Random Forests
• Chapter 16: Ensemble Learning

Did I miss your favorite machine learning textbook that has a section on ensemble learning?
Let me know in the comments below.

Recommendations

I have a copy of each of these books as I like to read about a given topic from multiple perspectives.

If you are looking for a solid textbook dedicated to the topic of ensemble learning, I would recommend one of the following:

A close runner-up is “Ensemble Methods in Data Mining” that mixes theory and examples in R.

Also, I recommend “Pattern Classification Using Ensemble Methods” if you cannot get your hands on the more recent “Ensemble Learning: Pattern Classification Using Ensemble Methods“.

Summary

In this post, you discovered a suite of books on the topic of ensemble machine learning.

Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.