The greatness of St. Joseph is that he was willing to become a homeless wanderer out of love for God and Mary. He did not want to defame Mary by a divorce, even a quiet one. After Jesus Christ, St. Joseph is the humblest of all men, and he was willing to step out of the picture and disappear.
