After breaking the 72-arrow world record in last week's ranking round, Republic of Korea's Im was installed as a red-hot favourite.
Instead, he fell well below his own high standards and saw European champion Van der Ven shoot superbly for a 7-1 success.
Im strayed with sevens in both the second and third sets and had a high score of 27 from four sets.
Van der Ven missed the gold rings just once in 12 attempts and will now face Kuo Cheng-Wei in the quarter-finals.
There was better news for Im's Republic of Korea's team-mates Oh Jin-hyek and Kim Bubmin as Archery's dominant nation look to secure a third gold from four, having also won both women's events.
Oh put in one of the morning's standout performances, claiming a straight sets victory over Poland's Rafal Dobrowolski.
Kim joined him in the quarter-finals with a 7-1 win over 15-year-old Moldovan Dan Olaru.
Olaru had accounted for the highly-rated Jake Kaminksi and former Olympic bronze medallist Simon Terry on Wednesday but fell short against the second seed.
Kim shot a consistent round of arrows, grouping well in the nine ring and adding four 10s as his teenage opponent managed four consecutive sets of 26.
Kim takes on China's Dai Xiaoxiang for a semi-final berth after he edged a tense contest with Australian Taylor Worth.
Reigning Olympic champion Viktor Ruban, whose standing has fallen since Beijing, also remains in contention to retain his title.
The Ukrainian defeated France's Gael Prevost 6-4 and will shoot against Oh in the afternoon session.
Related content
- Athletes
- Sports
- Countries
