With the town relegated to mere tenant status, the building fell wholly under the control of a dwindling population of faithful churchgoers. In the years after the Free Town Project was announced, the Grafton Center Congregational Church, which had owned the property for decades, called it quits and moved elsewhere (to the Danbury town line, under a new name).