The issue with the lander's propulsion system followed a successful launch of a new Vulcan rocket debuted by a joint venture of Boeing and Lockheed Martin.
Space robotics firm Astrobotic's Peregrine lunar lander launched at 2:18 am local time from Cape Canaveral, Florida, on the first flight of Vulcan, a powerful rocket that had been under development for a decade by the Boeing-Lockheed venture United Launch Alliance (ULA).
But the lander failed to enter its correct sun-facing orientation in space hours after it made successful contact with ground teams and activated its propulsion system, Astrobotic said in a statement.
"The team believes that the likely cause of the unstable sun-pointing is a propulsion anomaly that, if proven true, threatens the ability of the spacecraft to soft land on the moon," the company said.
If Astrobotic can recover from the mishap and carry on with its mission, Peregrine would mark the first US soft landing on the moon since the final Apollo landing in 1972, and the first-ever lunar landing by a private company - a feat that has proved elusive in recent years.
"This is the moment we've been waiting for for 16 years," Astrobotic CEO John Thornton said after the lander's launch. Applause roared in the launch control room when Peregrine was released from its booster stage, setting the golf cart-sized craft on a 46-day journey to the moon.
The mission is the latest in recent years among countries and private companies sprinting to the moon, a renewed stage of international competition in which scientists hope the moon's water-bearing minerals can be exploited to sustain long-term astronaut missions.
The launch of Vulcan, a 60 metre tall rocket with engines made by Jeff Bezos' Blue Origin, was a crucial first for ULA, which developed Vulcan to replace its workhorse Atlas V rocket and rival the reusable Falcon 9 from Elon Musk's SpaceX in the satellite launch market.
"This has been years of hard work. So far this has been an absolutely beautiful mission," ULA CEO Tory Bruno said in the company's launch control room of the Vulcan launch.
The stakes were high for Vulcan. Boeing and Lockheed, which own ULA in a 50-50 split, have been seeking a sale of the business for roughly a year. The launch was the first of two certification flights required by the US Space Force before Vulcan can fly lucrative missions for the Pentagon, a key customer.
Peregrine was set to land on the moon on February 23 with 20 payloads aboard, most of which will seek to gather data about the lunar surface ahead of planned future human missions. It marks the first trek to the moon's surface as part of NASA's Artemis moon program.
That multibillion-dollar program, involving various countries and relying heavily on private companies such as SpaceX, envisions astronaut missions to the moon later this decade. Small landers such as Peregrine will get there first.
A second private US company under the same NASA program expects to launch a lander of its own in February. Carrying similar NASA payloads and launching to space aboard a SpaceX Falcon 9 rocket, Houston-based Intuitive Machines' said its spacecraft could make a moon landing on February 22, a day before Peregrine.
India last year became the fourth country to achieve a soft lunar landing after Russia failed in an attempt the same month. The US, China and the former Soviet Union are the only other countries that have carried out successful soft lunar landings.
Private companies with hopes of spurring a lunar marketplace have had harder times, with Japan's ispace and an Israeli company crash-landing on their first attempts.
20°C