The concept of age is difficult for children to understand as it requires coordinating knowledge across several domains of abstract concepts (e.g., time, number, biology). We tested 122 three- to five-year-old children on their identification of which of two figures is older, as well as on their knowledge of which of two numbers is greater and their ability to temporally order past memories. Consistent with prior research, we found that young children are influenced by size in making age judgments, demonstrating a bias to respond that someone who is bigger is older. However, we show that by age 4, children can incorporate numerical age cues to make accurate age judgments. Among other possible interpretations, these findings suggest that children may initially conflate age with size before identifying chronological time as the relevant domain for age, exhibiting a conceptual change for which acquiring numerical knowledge may play a key role.