Rifle Skill is a measure of how good a soldier is with his basic weapon skills, mostly hitting their target. The total rifle skill is the physical chance that your soldier will hit a German. Rifle purely affects weapon skill, not CQC skill. Some units naturally have more rifle skill than others (e.g. Spec Ops and Snipers), while in others, Rifle skill is naturally lower. (like Gunners).
50 Rifle = 50% chance that your soldier will hit their target. Cap of accuracy ends at 90%, though your soldiers can earn more rifle skill than 90. Units like Gunners have accuracy cap slightly nerfed, as to not make them super-overpowered.
How to improve Rifle
Your units improve Rifle skill in few ways: the most simple is ranking up on field. Another option is giving them Ranger Training. Or pray for crates with rifle+ bonuses.
Vehicles have gun skill instead, but it works same way. E.G more rifle = more hits. The only difference is that vehicles cannot be boosted with Ranger Training, and gain gun skill very slowly through normal leveling on the battlefield.
High rifle skill is deadly when combined with exp (Exp = speed, while Rifle is hit chance).
As you go up the waves, rifle for the Germans increases gradually. While the Germans aren't that accurate before ~wave 200, later make sure your men are in cover, otherwise you're going to take a lot of casualties. Snipers and similar are perpetually lethal, however, and become devastating as you achieve higher waves.