Gradient-boosted trees, one tree at a time

XGBoost's algorithm, built from scratch. Each new tree is fit to the errors the model is still making.

The model after 0 trees

  • class 0
  • class 1
  • test point
  • next tree's cuts
  • dot size = current error

Next tree to be added: #1

Left branch = condition true. Leaf color = which class it pushes toward.

Loss as trees are added

  • training
  • test (held out)