The uncertainty sent the yen currency to a three-month low on Monday as investors and analysts prepared for days, or possibly weeks, of political wrangling to form a government and potentially a change of leader.
That comes as the country faces economic headwinds, a tense security situation fuelled by an assertive China and nuclear-armed North Korea, and less than a week before US voters head to the polls in another unpredictable election.
Ishiba's Liberal Democratic Party (LDP) and its junior coalition partner Komeito took 215 seats in the lower house of parliament, down from 279 seats, as voters punished the incumbents over a funding scandal and a cost-of-living crunch. Two cabinet ministers and Komeito's leader, Keiichi Ishii, lost their seats.
The biggest winner of the night, the main opposition Constitutional Democratic Party of Japan (CDPJ), led by Yoshihiko Noda.
The party had 148 seats, up from 98 previously, but also still well short of the 233 majority.
Yoshihiko Noda's opposition Constitutional Democratic Party has secured 148 seats. (AP PHOTO)
As mandated by the constitution, the parties now have 30 days to figure out a grouping that can govern, and there remains uncertainty over how long Ishiba - who became premier less than a month ago - can survive after the drubbing. Smaller parties also made gains and their role in negotiations could prove key.
"Whether or not Prime Minister Shigeru Ishiba resigns as LDP leader today, it seems unlikely that he will survive to lead a new government as prime minister ... though it is possible he could stay on as caretaker," said Tobias Harris, founder of Japan Foresight, a political risk advisory firm.
Ishiba is scheduled to hold a press conference at 2pm on Monday (4pm AEDT).
Ahead of the election, the LDP had been planning to convene parliament on November 7 to confirm the prime minister, according to Japanese media including Jiji and the Yomiuri newspaper.
CDPJ leader Yoshihiko Noda has said he would work with other parties to try and oust the incumbents, though analysts see this as a more remote possibility.
The LDP has ruled Japan for almost all of its post-war history and the result marked its worst election since it briefly lost power in 2009 to a precursor of the CDPJ.
In a statement, the head of Japan's most powerful business lobby said the country needed political stability to steer an economy that faced urgent tasks such as boosting the supply of nuclear energy and maintaining the momentum for wage hikes.
"We strongly hope for policy-oriented politics through the establishment of a stable government centred on the LDP-Komeito (coalition)," Keidanren Chairman Masakazu Tokura said.
