Akiko Adachi had given Japan the lead four minutes into the second half only for Josefine Jalmestal to equalise with only 30 seconds on the clock to send the match into overtime.
The two teams could still not be separated after six minutes of overtime and a round of three extra throws for each side before the second throw of sudden death decided the outcome of the semi-final.
Masae Komiya fired a shot diagonally across court and past the fingertips of Sweden’s Sofia Naesstrom, before the Swede saw her return throw come off the post to hand Japan a 4-3 victory.
Komiya said: ‘I have made so many mistakes today, so I thought, “I've just got to do this, to make a goal to win. That's my responsibility”.’
Their opponents in the final will be world champions China, who overcame an early scare when Katja Heikkinen scored with Finland’s first shot of the match.
But the Beijing 2008 silver medallists recovered well and goals from Wang Shasha, Chen Fengqing (2) and Lin Shan sealed a 4-1 win for the Chinese.
On China's mental attitude, coach Wang Jinqin said: ‘We should win, we must win, it's normal. We never think we'll lose.’
In the men’s semi-finals, Finland completed their fine turnaround in the competition as a goal in each half from Jarno Mattila helped them record a 2-0 victory over previously-unbeaten Turkey.
European champions Finland lost their first three group games, including a 4-0 loss to the Turks, but an improvement in their defensive play has helped them qualify for the gold-medal match.
Awaiting Finland in the final will be Brazil, who saw off world champions Lithuania for the second time at London 2012.
The South Americans battled back from trailing 1-0 to Genrik Pavliukianec’s first-half goal to win 2-1 - thanks to a penalty from Alexsander Almeida Maciel Celente and a goal for Romario Diego Marques.