“In addition to the psychological costs, there is another reason to fear ostracism. Groups will use it without compunction to assert their will and achieve their ends.”