Any time two surfaces grind against each other, they’re both being abraded, but it’s a question of relative degree. For example, your strings grind away at your frets (bending) and plastic pick (I suspect). The cumulative damage from the strings on non-stainless frets is annoying, eventually requiring a fret job.
You are right that metal picks will likely damage the strings, but if the practitioners suggest that the strings will die from other reasons before the pick kills them, then why not use a metal pick (assuming that you like the sound)? In fact, Dunlop has a stainless steel pick, that will probably trash everything (including your guitar finish), as well as what seem to be anodized aluminum picks (a sapphire surface, quite hard). But in practice, I think the what kills most strings is skin cells and dirt getting into the windings and making them sound dull; this is why I like Elixir strings (now that I went back to 9 gauge instead of 8), they have a teflon tube that keeps the string windings clean.
NOTE: I HAVE NEVER PLAYED A METAL PICK AND HAVE NO PRACTICAL EXPERIENCE WITH THEM.
https://www.elixirstrings.com