First of all, I collect the seeds when they are really 'fluffy'. I keep the seeds in a paper bag until they are dried completely ( 1-3 weeks, depending on the region where you live). After that I shake the paper bag and the 'fluff' wil be separated from the seeds. If the 'fluff' is not separated after that, the seed is not ripe, so not viable.
In nature the 'fluff' is needed to spread the seeds, but the 'fluff' wil be separated from the ripe seeds at a certain time. But if you sow seeds in vermiculite you need to have seeds without any chaff.