The Irons blew the points as Brighton struck twice inside the final ten minutes at Falmer stadium.
West Ham's away fans taunted Potter in support of striker Niclas Fullkrug, who had been criticised by his manager for calling out teammates in midweek.
“The German's right we’re f***ing s***e”, was heard throughout the game from the away end. Potter dropped Fullkrug to the bench for the trip to the south coast.
Asked about the chants, Potter said: “They’re entitled to sing whatever they want.”
He also denied he left out Fullkrug after his outburst following the Southampton loss.
"I can swear"
Potter added: “No one wants to hear positives. It doesn't matter. I’m sick of talking about positives.
“We have to stick together, because it's a tough last four weeks. We have to be together.
“I stand there myself, give everything to this, what I'm doing of course, I'm gonna be upset.
“I have to come here and speak calmly to you guys, but that's not how I'm feeling inside. If you want me to swear, I can swear.