There are many many cases where troops that were not in the Infantry branch were handed rifles and used as such. Sometimes their units were used like Infantry for their entire time in combat. However, unless the unit actually became a part of the Infantry branch or one individually changed their service classification to an Infantry one, they would not qualify for a CIB.
On the unit level, it was uncommon for them to be switched, and there are still many who served in Artillery, Armor, Signal Corps, and Engineer units that feel that they deserving of the same recognition because they held a rifle and got shot at just as much.
I suspect that many of the fellows in the diary you mention that "left for the Infantry" were being transferred out of the AAF branch as well. Most I bet were replacements or folks that were not for some reason or another getting the job done. By 1944, the AAF was getting more low level guys volunteering than they could use....but the Infantry wasn't.
In the end, we are still talking about if a certain badge belongs on a uniform. It is very easy for someone to pin something on that does not belong. They did not just hand CIB's out to anybody. Those guys earned then, and had paperwork in their files to prove it.
That is not to say that guys did not transfer or change branches, so there is some room for some oddities to exist. However, I don't think it was that common. And most of these oddities were corrected along the way....like the Air Force after the war making guys take off their CIBs and replace them with Bronze Star ribbons...to make sure that their uniforms stayed "uniform"....which is really the point of it all anyway.