I’m not sure I understand the premise of the question, I guess.
How does notation reveal that a composition is in a pentatonic scale? Nothing about the key signature or the way the pitches are written on the staff says “pentatonic,” it’s just that you, as a fairly savvy musician, should be able to recognize that if you see a G major/E minor key signature, the music is resolving to the tonic of E, and the pitches being played are E, G, A, B, and D, and nothing else, that it’s an E minor pentatonic line.
And that takes it to another degree, really - how does a key signature reveal that it’s in E minor, and not G major, in the first place? If you see one sharp, it could as easily be G major as E minor, and yet while we’re talking about “Rambling Man,” you’re very confident in your assessment that the tune is in G major, and not E minor (at least before we get into the inclusion of the b7 in the chorus). I think the answer for that is that you’re also turning to other context clues, and noting the fact that the song continously resolves to a G chord means it’s in G, not E minor. But, nowhere on the staff does it say “the tonic is G,” you’re just using context clues to determine that.
I’d see it as kind of the same question. I actually don’t know whjat standard practice is here, but I’d assume that to minimize the number of accidentals you would write a D Dorian composition in the key signature of C, but based on the fact the music continuously resolves to a D minor and treats the D minor as the tonic, know that it’s a D Dorian composition and not a C major one. Or an A minor/A aeolean one, for that matter, and since A Aeolean is just as much of a mode as D Dorian is, I think the principle of just looking at HOW the music is functioning and knowing what you’re working wiith from that is on pretty solid footing.
As for whether or not this makes modes “only useful to theorists,” I don’t know if I agree - I suppose you could look at a piece with no sharps or flats that resolves to D the whole time as “a C major composition, but for some reason it keeps resolving the the ii” but I think it’s probably more functionally useful to look at it based first on the tonic, D, and then on how the pitches employed all relate to each other, in this case Dorian.
Idunno. I don’t think it’s needlessly complicated, so much as just really flexible.