It's however possible that the conservatives will run the Austrian playbook. Talks with the social democrats fail, "we have a responsibility to Germany to form a government", then make a coalition with the AfD (Nazis).
Back to your original question: the BSW will likely contest the elections, as they have missed the 5% threshold by only ~14k votes and there are evident irregularities. For example, many Germans living abroad, for example those living in the US, had almost no chance to cast their vote.
Very unlikely. A coalition with the AfD is rejected by 85% of CDU voters. Merz has many times unequivocally rejected a coalition with the AfD.
there are evident irregularities
The only irregularity is voters abroad having received their mail in ballots too late. That was caused by the short timeframe to prepare the early elections. If that qualifies as a problem will be decided by the courts.
Election denial is less scary if they only have 20% support. If they tried it I imagine the vote to ban them would come back and maybe get enough support this time.