Your manager might have less visibility into your work and therefore might not be able to advocate for you or help you grow. An engineer working closely with a single team but reporting to a director may feel disconnected from the rest of the team or might pull the director’s attention into local disagreements that should have been solved at the team level.
