Albion twice went ahead, first through Neal Maupay only to see Ben Johnson level on the hour mark. Lewis Dunk gave the Seagulls their second lead of the evening before Tomas Soucek equalized eight minutes from time.
On the result
Although the Seagulls earned a point for their efforts, Potter was disappointed not to have taken all three points back to the South Coast.
"It feels a bit like a win that has got away from us. We're disappointed not to take three points, it was there for us, but it wasn't to be. We're frustrated we haven't won.
"Performance-wise we were the team that was closer to getting the three points. That doesn't matter, though. You have got to earn it. We have to keep working to get the three points."
On the club's performance
Potter felt his side were deserved of their lead going into halftime and he looks ahead to what the Seagulls could have done better.
"We deserved to be ahead at halftime, we had been the better team and we had chances. There's a bit of response from West Ham in the second half and we could have been a little bit smarter.
"But in the end we got the goal through Lewis and we could have managed the situation. We didn't do too much wrong apart from the second goal, that's what we're disappointed with."
On the West Ham equaliser
The gaffer described what he thought happened on the Soucek goal and what Albion could have done to prevent it.
"We didn't defend the corner well for their second goal and it's those details at this level that you can be punished for. That's on us and we have to do better there.
"Maybe we could have kept the ball a little bit better in the second half but I felt we were relatively comfortable in the game. We didn't do much wrong, we could have managed the game better.
"Soucek is the main threat and the obvious threat and he gets a clear run through us."
On the amount of draws Brighton have accumulated
Last season, the Seagulls drew 14 of their 38 matches and they have drawn seven of 15 this season. Potter knows the difference between a win and a loss at this level.
"I speak about the margins and the margins are quite clear. If you can do that it sounds simple and if you can do that it significantly impacts your points tally and you can rise up the table.
"But it is all well and good saying it, we have to make it happen. We have to keep working, keep fighting, keep staying together. Try to create more chances, restrict the opposition to fewer chances and keep working."