They once called him a ‘goose-stepping extremist.’ They’re now sitting out his comeback bid.