That’s the trick: allowing entropy to continue to rise in both directions of time, even though it started out large to begin with. There isn’t any state we could possibly have chosen that would have prevented this kind of evolution from happening. An arrow of time is inevitable.