There is some remarkable but complex theory, known by the delightful term ‘the Law of the Iterated Logarithm’, that shows that if we carry out such repeated testing, even if the null hypothesis is true, then we are certain to eventually reject that null at any significance level we choose.
