XGBoost Boosted Trees
Gradient-boosted trees from scratch
The model is a sum of shallow trees, added one at a time. Each new tree fits the gradient of the loss left by the trees before it.
XGBoost builds its model one small decision tree at a time, and each new tree is trained only on the mistakes the model is still making. This demo implements that algorithm from scratch, then lets you watch it happen: the dashed lines on the plot show where the next tree will cut, aimed at the points the model gets most wrong.
Start with the five-step walkthrough: a single stump, trees correcting each other, the learning rate, overfitting, and how regularization reins it in. Then hover any node or region of the plot to see the gradient sums, leaf weight and split gain behind it.
Built with plain JavaScript and SVG, with no ML library. The data is 300 noisy two-moons points from a fixed seed, split into 210 for training and 90 held out for testing, so the results are identical on every load. All 101 trees retrain in well under 100 ms whenever a setting changes.