Joint learning and pruning of decision forests

Decision forests, such as Random forest (Breiman, 2001) and Extremely randomized trees (Geurts et al., 2006), are state-of-the-art supervised learning methods. Unfortunately, the size of the trees increases (at worst linearly) with the size of the training set, leading to substantial memory footprints when storing the forests. Several post-processing methods have been proposed to tackle this problem (Meinshausen et al., 2009; Joly et al., 2012; De Vleeschouwer et al., 2015; Ren et al., 2015). Although very e↵ective, they require building the whole forest first and then solving a highdimensional optimization problem to (post-)prune it. In this research, we would like to investigate whether a similar compression e↵ect could be achieved directly during tree growing, without ever requiring the development of the whole forest. For that purpose, we propose a joint learning and pruning (JLP) method which iteratively and greedily deepens the trees by optimizing globally the sequence of nodes to develop, while still choosing locally the splitting variables and cut points at all tree nodes based on the standard score criterion. The algorithm is presented in Section 2 and some results and comparisons with more naive and post-pruning methods are reported in Section 3.