How old is your HVAC system?
The ductwork? Do you have pets?
Are you good about changing your HVAC filters?
In my limited experience, and observations when removing ducts, most ductwork does not get that dirty. The grills, if they’re on the floor, tend to collect dirt, pet hair, and dust, however.
Vacuum out your floor registers and ducts regularly and change your filters.