It sounds like the stone walls have absorbed the odor of the fertilizer. You could try ventilating it for several days, leave the door open, put a fan in the shed and let it run. Maybe you could scrape off the top layer of the soil on the ground. Is it possible to hose down the walls?

chemical type fertilizer smell?
or manure/organic fertilizer type smell?

It's the nitrogen/ammonia that smells. First I would power wash the stone and scrape a layer of the dirt off and dispose of it outside somewhere where the smell won't be a problem, replace with some crushed gravel or suitable floor material.

After it dries, sprinkle the floor with a powder you can get at the farm supply store for horse stalls to neutralize urine odor - one brand is "Sweet PDZ". It absorbs and locks up ammonia odors from horse urine, and will work for your fertilizer smell too.

Here is a link that might be useful: Sweet PDZ