People who idle are either mafia by virtue of actually being mafia, or implicitly mafia by virtue of not participating and therefore acting as deadweight (since idle players are the same as a 'no' vote). If somebody is idling, then it is in town's interest to exterminate them as soon as possible, because it is a guaranteed way to get rid of someone who is helping mafia (and, again, they're either helping mafia intentionally or by virtue of not playing). If town doesn't do this, then as we saw here eventually the deadweight creates a situation whereby mafia wins by doing nothing other than withholding their vote. I suspect mafia realized this and that accounts for our late-comer idlers.
I predicated every nomination of an idler on the basis of, "If they come back, I'll pick someone else" precisely because the point is to encourage participation or get rid of somebody not participating. That this didn't work even a little bit suggests our targets either were actually mafia, and therefore we were right to nominate them, or opted out of the game such that not even the threat of getting lynched would get them to participate... and therefore we were right to nominate them.
I had my suspicions about other players, but those were essentially irrelevant. If I pick somebody playing, my chance of lunching somebody in the mafia is less than 100%. If I pick somebody idling, my chance of lunching somebody who is either explicitly or implicitly mafia is 100%. It's that simple.