Manager Ange Postecoglou is under pressure after another disappointing loss to London rivals Chelsea this weekend as his side blew a 2-0 lead to lose 4-3.
The club are now 11th in the Premier League as fans debate whether the manager should leave or the owner. Speaking on Sky Sports, Carragher was adamant that Levy should depart after over two decades at the the club.
"They've changed so many managers," Carragher said. "We talk about Daniel Levy a lot. What Daniel Levy's done here in terms of the training ground and and this for me, this is the best stadium in the world.
"And that's been Daniel Levy's strength in terms of a really stable football club. The stadium he's delivered. He's delivered the training ground.
"It's probably some time for somebody else to come in because to not win a trophy in that period of time, with the manager they've had.
“They've never really got out of the way in the transfer market, they've spent decent money without, you know, blowing other teams out the water. The wage bill is always one of the smallest. And you're never going to get the best players.
"So it might be a time for Daniel Levy - who I've been a supporter of because you look at the actual work that he's doing - but now that work's done, in terms of a stadium and a training ground, someone else needs to be in charge of this football club.”
- Get breaking football news faster with the new Tribal Football app! Download now on iOS and Android to stay ahead of all the latest transfers and football updates. Download now: App Store, Google Play