The win takes the Manhattan, New York-born 22-year-old into a career-best fourth-round of a Grand Slam where she will take on Germany's Laura Siegemund for a place in the quarterfinals.
Having shocked second seed Karolína Plíšková 6-4, 6-2 in the second round in a straight-sets demolition, it appeared that the 2017 French Open champion may suddenly be exhibiting the kind of form that took her as high as number five in the world.
The Latvian did make the perfect start as she sealed a break of serve to go 2-0 ahead against a nervous-looking Badosa who found herself in new Grand Slam territory.
However, the Spaniard refused to wilt as she immediately broke back and did so again in the fifth game as she moved 3-2 up.
Badosa was into her stride as she won four games in a row to establish a 4-2 lead.
Ostapenko, however, showed flashes of brilliance and did well to save 2 set points but ultimately it would be her younger opponent who would take the opening set 6-4.
The second set began evenly poised as Ostapenko and Badosa both established holds of serve.
But the Spaniard seized the critical break in the Latvian's second service game as a mixture of poor shot selection and a lack of intensity gifted Badosa a 2-1 advantage.
This was then further reinforced with a hold of serve and a second break of the set as Badosa moved 4-1 up and inched closer to a place in the fourth round.
Badosa then wrapped up the biggest win of her career as she broke for a third time in the second set to see off the world number 43 in fairly comfortable fashion.