The trouble with time travel

I recently re-watched "Harry Potter and the Prisoner of Azkaban." Near the end of the movie, Hermione reveals that she has a time turner, which she and Harry use to go back in time and fix past situations. That made me wonder why we don't see the time turner again in the series. In the next book, Harry Potter and the Goblet of Fire, they could've used it on numerous occasions and saved a lot of grief, especially when it came to Harry, Cedric, or Moody. (I'm intentionally keeping it vague in c...