Temperature is extremely important for plants more or less governing much of their biology. If it is too high then it will affect seed setting and it will do so at temperatures that may be different for different cultivars.
However, daylilies also have compatibility issues. Some of the species are self-incompatible. That means if you use the plant's own pollen on it pods will not form at all or only rarely form and mature. When plants have self-incompatibility they must have ways to identify self pollen from non-self pollen. Unfortunately, that means one needs to collect many different (unrelated as possible) individuals of that species to get as many different "identifiers" as possible. I don't think many specimens of each species were used originally to form the daylily breeding populations so there can be compatibility issues between some pod and pollen parents.
Compatibility may be to blame if a cross sets a pod occasionally and then aborts it.
Some daylilies are pod sterile (or have very low fertility). Others are pollen sterile and yet others may be both pod and pollen sterile.
Then there is the problem of the natural lower fertility of tetraploids. They have problems in meiosis and will produce unbalanced chromosome numbers in some of their gametes and the corresponding lower fertility.
