I think that you are using the custom taxonomy with a incorrect concept. You should use a custom field if you want sort by its value.
Thinking in numbers can be dificult to understand what a taxonomy is. But think in a way to group things where a group is not higher or lower than other. Every term in a taxonomy has the same value, except in hierarchical taxonomies. For example, think in animal taxonomy. All Genus are in the same level, all under the same parent Family, and all Families are in the same level under the same parent Order. How can you sort by Genus or search in between Genus?. You only can search animals if they are IN/OR/NOT IN/AND in one or more Genus. A taxonomy is definetely not for for values that are going to use to sort by (a searh IN BETWEEN is a type o sort by).
You could build a hierarchical taxonomy for each integer number that represent an age, but even in this case, you have to group by a taxonomy term and them recursively search for children/parent items, what is not really a sort by/search between value. I think is not practical if compared with the idea ofe age as meta data (ie custom field).