It’s possible to spend too much time refactoring. You don’t need to refactor code that’s unrelated to your present work. Similarly, balance your need to deliver stories with the need to have good code. As long as the code is better than it was when you started, you’re doing enough. In particular, if you think the code could be better, but you’re not sure how to improve it, it’s OK to leave it for someone else to improve later.

