Ask one yes/no question at a time. Each split divides the data space — watch the tree grow, one branch at a time, until every leaf is pure.
A decision tree is a flowchart of yes/no questions about features. At every internal node it asks one question, routes data left or right, and repeats until the data in each leaf belongs to one class. The algorithm picks the split that reduces impurity most — measured by entropy or Gini index. Trees are fast, interpretable, and the building block of random forests.
Cyan dots are Class A, pink are Class B. Each step adds a split plane and a new level of the tree. Watch the data space get carved into purer regions.
Four concepts that every tree algorithm — CART, ID3, C4.5 — builds on.
A split picks one feature and one threshold: "is feature X greater than value T?" All samples go left or right. The best split maximally separates the classes.
Entropy measures how mixed a node is. A node with all one class has entropy zero (pure). A 50/50 mix has entropy one. We choose splits that reduce entropy most.
An alternative to entropy that is faster to compute. Gini measures the probability of misclassifying a randomly chosen element. Used by scikit-learn's DecisionTreeClassifier by default.
A fully grown tree memorises the training data (overfits). Pruning removes branches that add little predictive power. Limiting max_depth is the simplest form.