Boosting is a celebrated machine learning approach which is based on the idea of combining weak and moderately inaccurate hypotheses to a strong and accurate one. We study boosting under the assumption that the weak hypotheses belong to a class of bounded capacity. This assumption is inspired by the common convention that weak hypotheses are "rules-of-thumbs" from an "easy-to-learn class". (Schapire and Freund '12, Shalev-Shwartz and Ben-David '14.) Formally, we assume the class of weak hypotheses has a bounded VC dimension. We focus on two main questions: (i) Oracle Complexity: How many weak hypotheses are needed in order to produce an accurate hypothesis? We design a novel boosting algorithm and demonstrate that it circumvents a classical lower bound by Freund and Schapire ('95, '12). Whereas the lower bound shows that $\Omega({1}/{\gamma^2})$ weak hypotheses with $\gamma$-margin are sometimes necessary, our new method requires only $\tilde{O}({1}/{\gamma})$ weak hypothesis, provided that they belong to a class of bounded VC dimension. Unlike previous boosting algorithms which aggregate the weak hypotheses by majority votes, the new boosting algorithm uses more complex ("deeper") aggregation rules. We complement this result by showing that complex aggregation rules are in fact necessary to circumvent the aforementioned lower bound. (ii) Expressivity: Which tasks can be learned by boosting weak hypotheses from a bounded VC class? Can complex concepts that are "far away" from the class be learned? Towards answering the first question we identify a combinatorial-geometric parameter which captures the expressivity of base-classes in boosting. As a corollary we provide an affirmative answer to the second question for many well-studied classes, including half-spaces and decision stumps. Along the way, we establish and exploit connections with Discrepancy Theory.
翻译:泡妞是一种值得庆祝的机器学习方法, 其基础是将微弱和中度不准确的假设与强力和准确的假设相结合。 我们研究在以下假设下提升: 弱假设属于封闭能力类别。 这一假设受共同公约的启发, 弱假设是“ 容易骗人” 类的“ 规则” 。 (Schapire and Freund' 12, Shalev- Shwartz and Ben- David'14. 。 ) 形式上, 我们假设弱假设类的等级具有约束 VC 层面的维力。 我们研究两大问题:(i) 弱假设属于约束性假设, 为了产生准确的假设? 我们设计了一个新的推力算法, 并证明Vreund和Scaptireme('95,'12) 的经典下限 。 下限显示, 美元({levi) 和Ben- drimax 的默认是“ 必需 美元- main 事实 ” 。 我们的推算法性规则, 只能用来解释 。 (credeal) magial) 度 直立法 。