Saruman!’ he cried, and his voice grew in power and authority. ‘Behold, I am not Gandalf the Grey, whom you betrayed. I am Gandalf the White, who has returned from death. You have no colour now, and I cast you from the order and from the Council.’
Through Gandalf maintaining his allegiance to Middle Earth, and Saruman forsaking his, Gandalf has usurped Saruman's "colour" as Gandalf the White and Saruman is left with none.
