David McGoldrick thought he had taken the game to extra time after he cancelled out Nicolas Pepe's first-half penalty but minutes later Ceballos scored following an Arsenal counter-attack.
Sheffield United have failed to win a game since the football restarted.
Sheffield United were the better team, says Berge
It was the best performance of project restart for the Blades, who looked threatening throughout, with John Lundstram's header being ruled out in the first ten minutes for offside.
Pepe dispatched his penalty confidently midway through the first half after Chris Basham fouled Alexandre Lacazette.
In the second half, the Blades had the ball in the back of the net again through Egan but it was also ruled out for offside. McGoldrick equalised for the Blades in the last five minutes but Ceballos scored the winner with only minutes left in injury time.
"We were way better from the first minute," Berge told the club's media.
"We dominated the game, we were very aggressive, won the ball and played the Sheffield United football we want too.
"It's brutal that they came away with a late winner."
"We came back to ourselves today"
Berge believes Sheffield United can take positives from the defeat moving forward.
"We were lacking in the first few games," he told the club's media.
"The aggressiveness and the feeling that we are a good team and we can play at this level, we have showed that all year, so why not now?
"We came back to ourselves today and have found the key to play tough and keep the composure on the ball but lacked that finish.
"Then they counter on us when we had the momentum."
Sheffield United host Tottenham on Thursday in what will be a big game in the race for a European spot.