Our natural tendency is to abruptly yank ourselves back to the breath. However, that will actually slow you down in the long run, so let go gently and easily instead. Any annoyance or self-judgment you may feel is just something you have to let go of. By affirming your successes, you will make quicker progress.