@machinelearningtogo: ➡️Machine Learning from Scratch, part 22: the idea behind XGBoost Same 500 made-up bank customers as parts 18 and 20 (income and missed payments; no real people): it learns from the first 400 and is tested on the last 100. How it works: The first guess is the same for everyone: 47% pay back (188 of the 400). Then 200 rounds: measure every customer's error (truth minus guess), grow a small tree with at most two questions on exactly those errors, and add 1% of its correction (learning rate 0.01). With squared error, those errors point exactly downhill on the loss, so every tree is one step of gradient descent (Friedman 2001). The first tree asks the same questions as the tree in part 18. It learns that customers earning more than 3,020 a month with at most one missed payment were guessed 47 points too low (144 of 153 paid back). Result on the 100 new customers: 84 right after 200 trees (82 already after 7). Part 18's small tree got 82, part 20's forest 84, and someone who knew the true chances would also get 84 here. In this tiny world with two features there is little left to gain. Design fixed beforehand on 39 other made-up banks: depth 2, 200 trees, learning rate 0.01 (best average there: 84.6, small tree 84.2). Bigger steps or deeper trees memorized the noise within a few dozen rounds. What XGBoost adds (Chen and Guestrin, KDD 2016): A penalty on the number of leaves and on large leaf values, second order steps that also use how the slope bends, and engineering for speed (over ten times faster than popular tools back then). With squared error and no penalty, its leaf formula gives exactly our leaf values. 17 of the 29 winning solutions on Kaggle's blog in 2015 used it. On medium sized tables, tree models like XGBoost often outperform deep learning. Every number in the video comes from the real run of the program shown. #machinelearning #xgboost #gradientboosting #python #coding

MachineLearningToGo

Open In TikTok:

Region: DE

Friday 02 October 2026 22:09:18 GMT