Harry Potter and the Goblet of Fire (Harry Potter, #4)

by J.K. Rowling

“Okay, I can’t see the World Cup. Can I go now, then? Only I’ve got a letter to Sirius I want to finish. You know — my godfather.”

What's with the "Only"? I understand it in contrast to something but the context here is weird.