Hey Kenny. Before trying anything drastic, I'd try some 3 or aught steel wool and more walnut oil, and some very clean rags yo finish. The walnut oil might have gotten a bit gummy, esp. if there was a bit of excess on the surface, which could attract grit. If that doesn't resolve it, I might try mineral oil and steel wool, but I'd try that on an unobtrusive area first. It may be that you need to resand, but I'd not jump to that conclusion, esp if you sanded initially.
Keep us posted. John
Keep us posted. John