Well a lot in my opinion has to do with guitar originally being primarily a chordal instrument, before gain allowed us to play like violinists (or heard above the naturally louder brass and such instruments that essentially led most music in ensembles prior to the advent of amplification).
In fact the search for amplification which really begins with resonator guitars and eventually to amplifiers and electric was at the time in the 20s, 30s and 40s a need for the guitarist to be heard above the louder instruments in big bands common of the era where the guitarist was competing for sonic real estate with a full horn section and other “loud” instruments. The first electric guitar being a lap steel isn’t a coincidence, the lead instrument needed to be heard in the Hawaiian music craze.
A lot of the limitations of our instrument has to do with the tuning. I’m surprised more people haven’t experimented with this aspect. Aside from Robert Fripp I can’t think of many, Stanley Jordan maybe.
The mostly 4ths and major 3rd tuning of standard E 440 facilitates a lot of chordal flexibility (shiftable barre chords for instance) but like you say for nps it creates some sticking points for scalar playing. Where as the 5ths tunings of the viol instruments or for instance mandolin creates more options for playing scales through and makes it easier imo. I was surprised when I picked up mandolin how easy scale and runs are but how difficult and awkward chords can be. It’s all tradeoffs unfortunately.
Troy touches on this in the MAB clip in the original pickslanting vids when he discusses his surprise that the scalar runs on strings heard in film/tv scores are nearly absent in lead guitar, it’s just not something that easily accessibly in standard tuning.
It’s the same reason some stuff played on accordions (especially the diatonic button accordions common in eastern Europe) are murder to play on standard keyboard instruments.
It harkens to one of my favorite music jokes, ask a singer, wind, or brass player to play a chord by themselves without an ensemble.