Here's how it breaks down.
Go into your front yard. Grab a shovel. fill a 5 gallon bucket. Look at the small hole in amazement. Pickup bucket and act like you're going to put the dirt back in BUT WAIT!
That is when they come in and say "hey, we can't control what already exists, but since you pulled it out, we're going to need to test it for contaminants, heavy metals, minerals that exceed regulated benchmarks, consistency, and we'll get back to ya."
Then they come in 6 weeks later and say, "well, we have some bad news. It looks like the magnesium content is .015 above allowable benchmark in Multnomah county, and to make matters worse, we ran a compaction test and confirmed that the material is unsafe to be used as fill, it would need to be integrated with 10% 3/8" gravel and 15% 20 mesh sand. The tests are gonna be $2,500, and you'll have to bring in some approved material, at your cost of course, we'll have to test that, and please file a permit with the county. Please be aware a survey may need to be conducted as well. And you'll want to hire an environmental firm to represent you on the filed permit, assuming it's been approved, because they'll need to take stormwater outfall samples to ensure the mineral composites do not concentrate when purging to the local waterways. If they do, that'll be another permit."
That is what happens when you take something from the earth and try to put it back. That is why people in the mining industry despise federal regulations. Not because of safety, but because of that bullshit.