New pool?
If so likely material that was in lines during build plus pebble material from initial finish work, plus the material that worked lose first few months
Just swapped out my filters and housing yesterday. Little white sediment isn't bad that is the diatomaceous earth the filters use, dark like that is usually dirt. Watch you pressure guage at the top of the housing, it will let you know when filters need to be replaced or you have too much crap in the filters. Our filter is between 20-30 psi due to us having pop up jets. Most filter housing have a bleed off at 45-50 psi depending on the system
Mine looked like that the first couple times I cleaned my cartridge filter. Most of that was from the paver sand that got into the pool. What do you have around your pool? Ours is 12” travertine tile with pavers surround.