The 225-issue Nikkei Stock Average finished down 158.39 points, or 0.52 percent, from Tuesday at 30,511.71.
The broader Topix index of all First Section issues on the Tokyo Stock Exchange ended 22.48 points, or 1.06 percent, lower at 2,096.39.
Both indexes finished Tuesday at their highest levels in 31 years.
Almost every industry category went down on Wednesday, except for marine transportation issues. Major declining issues were led by real estate and insurance issues.
Investors became worried about the market getting overheated after the Nikkei benchmark rose over 7 percent following Japanese Prime Minister Yoshihide Suga's announcement of not running in the leadership race of the ruling Liberal Democratic Party (LDP).
Observers said Suga's announcement brought hopes of a new prime minister introducing fresh stimulus measures.
The record high in stocks the previous day was also attributed to progress in Japan's COVID-19 vaccination programs.
"The market tracked declines on Wall Street, while investors locked in gains after the market rose too rapidly in the short-term following Suga's announcement less than two weeks ago," said senior technical analyst at Mizuho Securities Co. Yutaka Miura.
However, as some market attendees turned to buy undervalued stocks, the losses were limited, and a brief rise in the Shanghai Stock Exchange additionally supported the market sentiment.
Expectations are still high towards a stimulus package by a new prime minister after the presidential election of the LDP on Sept. 29, which will effectively determine Suga's successor.
On the First Section, decliners outnumbered advancers 1,783 to 329, while 76 finished unchanged.
Exporters were struck following the yen strengthening overnight against the U.S. dollar. Among automakers, Mazda Motor shrank 2.2 percent, and Nissan Motor dropped 1.1 percent, while electronics manufacturer Panasonic fell 1.5 percent.
Some of Apple Inc.'s parts suppliers observed a downtrend in the tech giant's shares after it unveiled the new iPhone 13.
Murata Manufacturing went down 2.7 percent, and Alps Alpine decreased 2.3 percent.
Park24 tumbled 7.9 percent after the parking lot operator reported Tuesday a net loss for the period from November to July.
Trading volume on the main section declined to 1,205.72 million shares from Tuesday's 1,339.01 million shares.