Drilling out the rivets is the only way.
It's a simple endeavor on pop rivets but more of a big deal on hardened steel assembly rivets.
I'd use a drill press or a super steady hand and of course a bit that is the diameter of the rivet shaft, or slightly less, so you don't damage or enlarge the existing holes.