You probably noticed it first on the driver's side because it gets used more than the passenger. The handle pivot is not bushed so it loosens up and the upper edge begins to scrape against the top of the handle. Even brand new there is very little room. I shave the top of the handle before painting to avoid this. When you paint a used handle the paint thickness may be enough to cause the problem.
I have a good driver's side handle listed in the classies if anyone is interested.