The only downside is that you break the link between the numbers you are using to count with and the number of things you are counting. The first object corresponds to zero fingers, the second to one finger and all the way up to the eleventh object being represented with ten fingers.
