Potter had a challenging half-year tenure at the Stamford Bridge club under the current ownership.
Previously, Potter had led Brighton to an impressive ninth-place finish in the Premier League.
On Sky Sports, he said: “The mistake I made was thinking that new ownership means there's a chance of a change of culture.
“But actually culture runs a lot deeper than that. Before the World Cup we only lost three matches - Arsenal, Newcastle and Brighton but already the world was closing in, it was close to crisis.
“Perspective is sometimes not so easy to get, Chelsea were used to winning and it was an easy thing to think "we're not winning, it must be the coach who's not worked at that level, maybe he's the problem" and I can understand why they think that because the results weren't what they expected.”