Improv still requires some kind of guideline or constrains to work within.
It’s why the second Viennese’s school went from free atonality to Schoenberg’s 12 tone method in short order it was nearly impossible to create completely atonal music with out some kind of constraint. Humans always seek some kind of guiding principal or “center” to navigate back to and that ended up with atonal pieces trending toward some kind of key center.
Ironically in about 2 decades this atonality that was going to free the composer descended into some of the most rule bound music to the point a computer could compose these pieces as everything was serialized.
I’m of the opinion that sounds that are “foreign” to a listener come off as exciting and what is especially going to draw them in are going to be the most exotic elements of the music to that particular listener. This is why jazz constantly evolved styles, bebop, modal, post bop, fusion, smooth, etc. Anytime a style becomes too familiar aka people figure out the pattern and it becomes predictable (the inverse of blues which is almost built on the predictability of its harmonic and chordal progressions).
Someone who doesn’t listen to extreme metal is going to be wide eyed the first time they hear the drumming or the harsh vocals.
Someone who doesn’t listen to rap is going to have a hard time keeping up with the flurry of lyrics and the subtle intricacies of rhythm at play and how fast things are being said (I think this is a primary complaint of older audiences when it comes to rap) Conversely someone who primarily listens to rap may have a hard time with music that is harmonically or melodically complex. I think this is why you see a lot of diametrical opposition between the prog crowd and the rap crowd.
RIP DOOM
Someone who doesn’t listen to jazz isn’t going to be used to some of the extended harmonies that are used. (some of these used to be not so uncommon in pop/rock music in the post vocal jazz age but as we have drifted further from that era have become less common)
Someone who doesn’t listen to shoegaze/ambient/minimalist music isn’t going to be used to listening to music primarily for timbral and textural aspects.
A personal anecdote would be the first time I heard the solo in King Crimson’s “21st Century Schizoid Man” is was jarring and frightening because it’s atonal it’s not what a listener accustomed to the pentatonic, major/minor, modal sound of rock guitar leads is expecting. It smacks you in the side of the head.