Try hiking on a gravel path versus a dirt trail. Gravel moves under your feet CONSTANTLY. I once fell because the gravel gave way, it was painful. Packed dirt is just SO much easier to walk on.
Oh man, you're totally right about that. I learned the hard way on a hiking trail last year that gravel just shifts under your feet constantly. What ended up working for me was getting shoes with a stiffer sole, not super thick, just less bendy. It stopped the smaller rocks from poking through so much and gave me way better support. I also had to slow down and pay more attention to where I was planting each step instead of just plowing ahead.