Fraser is out of favor with his current employers, Newcastle United, where he has fallen out with manager Eddie Howe.
Having spent last term on loan at the Saints, Fraser may be in line for a return to St. Mary’s Stadium.
Speaking to the Daily Echo, Martin said: "I think we're all hopeful that at some point that will happen.
"But he's contracted to Newcastle. I think he would love to come back and we would love to keep him. Everyone wants him.
"Me, the fans, the owners, the players and the staff - Wee Man himself, but he is under contract at another club and a good contract.
"It's their prerogative to do what they want with him at the moment and unfortunately for us and for Wee Man, it's not what we want," Martin added.
"There'll come a time where if a deal can't be done with Newcastle, we're going to have to move on to other people and I think that's a natural thing.
"So much of football is about timing and there might come a time where it's no longer viable for us or an option. We have to be looking at other people as well."