Even if the asymmetry were 1 part in a billion there would be enough matter left over to account for everything we see in the universe today. In fact, an asymmetry of 1 part in a billion or so is precisely what was called for, because today there are roughly 1 billion photons in the cosmic microwave background for every proton in the universe.
