Martin was dismissed in December and replaced by Ivan Juric. However, the change has had little effect with Saints propping up the table.
Speaking about his departure for the first time on Sky Sports, Martin said: "I'm frustrated I'm not there anymore but now I know the journey must start somewhere else.
"I'm enjoying a break, it's been for a lot for 20 years. It's my first break from football in 20 years.
"I think it's really difficult to balance being a manager, you have to go in with a long-term view and strategy but short-term, game-to-game you have to try and fix problems.
"It's always tough when you get promoted to the Premier League, it's tough for any team but I felt we had a lot of young players and we were trying to grow something.
"Am I surprised by it in football? No, not at all. Of course, I'm disappointed but it's been so much learning. I've learned so much so I feel we'll definitely be better for it.
"I see so many managers and speak to so many coaches who came in to watch us work. After they lose their job, they feel regretful that they changed too much of what they wanted at the very beginning.
"I don't feel we did that. I have disappointments over maybe making too many changes or adapting too much.
"The biggest challenge for me was convincing everyone they belonged there and without evidence of winning games, that becomes really difficult."