Each time I wanted a roll for all the players (or pretended I wanted one) I asked someone to draw a card and then compared it with my bonuses master card:
What I found is that every time I asked for a fake roll, there was zero overhead, so I asked for one far more often. About two-thirds of the rolls were fake. Hence we had 20 cards, and during a session we went through about 12 of them, but only 4 of them were actually used - the other 8 went back in the pack at the end of the session. At the start of the second session, I rubbed out the used cards, and we re-rolled them. This mix of "old and new" helped avoid Leland's concern:
"if you only have a few cards a player might know that (s)he's got a lot of high (or low) rolls coming up today."
He's right that you need a reasonable number of cards for the system to work well (else you know what's coming up) but you only have a real upfront cost the first session you use it. Leland also had an interesting suggestion:
"you could probably speed up that initial cost by just having players fill out entire cards instead of having to pass them all around."
My players took it in turns to fill out the whole pile and then pass it on, not passing it on one at a time, but this would still be quicker. The trade off between speed and knowing that players "physically rolled the dice that affect their character's fate..." is one that you'd find out in more prolonged play.
What was an issue was I was trying out a "best two out of three" mechanic at the same time, which meant three rolls per PC per card. This meant a lot of rolls, so I'm abandoning that particular idea for this situation!
The second issue was that it brought up the question of what we were rolling for. Since I had removed any overhead, the question of when is it "one roll for the party" versus "one roll for each PC" became very prominent. That's an issue I'm still mulling over.