According to the post-election poll results published by ZDF, the Christian Democratic Union and the Christian Social Union (CDU/CSU) led with 28.6% of the votes, while Mr. Scholz's Social Democratic Party (SPD) only reached 16.5% - a record low since World War 2.
This is a bitter result for the SPD, and also a defeat in the election, Prime Minister Scholz acknowledged, and sent his congratulations to his opponent Friedrich Merz.
The biggest surprise in this election was the rise of the far-right party The alternative choice for Germany AfD (Alternative für Deutschland), when this party won 20.4% of the votes - the best achievement in the party's history. Afd co-leader Alice Weidel called the result historic and said the AfD was ready to join coalition negotiations to create a real political change in Germany.
The AfD's rise puts Germany in a controversial position, as the party pursues tough immigration policies and policies to cut support to Ukraine - issues that have caused strong division in German society.
Chancellor Scholz's ruling coalition, which has been shaken for a long time, officially collapsed after the free Democratic Party (FDP) left last November. The remaining partners also suffered heavy losses: The Green Party only reached 12.3%, while the FDP struggled to surpass 4.7% to enter the National Assembly.
Analysts say a big alliance between CDU/CSU and SPD could be the most likely option for a new government, although Merz has previously denied a cooperation with Scholz.

Under Merz's leadership, the CDU/CSU has shifted to the right wing, especially in immigration policy - marking a major change compared to the mid-term of former Chancellor Angela Merkel.
This election has become even more dramatic as American politics and technology are not left out. US Vice President J.D. Vance met with Ms. Alice Weidel right before the election to discuss Ukraine and Germany's internal policies.
Meanwhile, billionaire Elon Musk publicly criticized Chancellor Scholz and expressed support for the AfD. He even appeared at a partys election event in January.
With this result, Germany is entering a period of political turmoil, where Mr. Scholz's political future is uncertain, while the right wing - from conservative to far-right - is on the rise.