I have/had the same problem. What I discovered is that when I start a phrase with an upstroke, I don’t feel as comfortable as when I start with a down stroke. So…if we were doing the G major scale, I start on the second note of the phrase which would be the 7th fret on the high e string (B) then goto the 5th with an upstroke, sweep up and accent the down strok on every string. When I did this, things felt much more comfortable.
My hypothesis is that I clearly use the down stroke to initiate my phrases and I have neglected the upstroke. I am currently working on this by isolating the parts that are difficult, playing slow but also pushing myself hard at a faster target speed, practicing the picking without fretting hand AND fretting hand without picking. I also use a metronome.
I can’t guarantee this will help you, but it really pushed me to another level. If my hypothesis is correct, it would stand to reason that the reciprocal would also be true (predominantly starting with upstrokes), but as we have seen with CTC, this is not always the case.