When the star becomes small, the matter particles get very near each other. But the Pauli exclusion principle says that two matter particles cannot have both the same position and the same velocity. The matter particles must therefore have very different velocities. This makes them move away from each other, and so tends to make the star expand.
