I'd obviously avoid any abrasive processes, but a gentle scrubbing with a RV brush would probably get it fairly clean. Use a house-wash attachment to your hose to apply the cleanser, scrub, rinse. Look for a cleanser that announces its particular effect on mildew. If the siding is still stained after physically scrubbing, maybe look into an application of bleach.
Oh, yeah, protect the plants from the cleanser and definitely from any bleach.
Casey

I'd try what Casey has suggested in an inconspicuous spot. If it works do the whole surface.

But start at the bottom, not the top. Sounds counter-intuitive because you'll be washing the upper ick down over the lower, already- cleaned walls. But the lower walls be wet and the stuff from above will slide right off. If you start at the top, you may get the stuff running down and making stains. Finish up each vertical section with a thorough rinse from the top down.

Work vertically in sections about 4 to 6 feet wide. That is pretty much as far as you can reach from side to side once you're up on a ladder.

There are all kinds of scrub brushes: flat, rectangular floor ones, ones with splayed-out bristles (good for cleaning the under-edge of a siding lap); narrow ones which work particularly well in the right angle between boards, etc. Even tooth and nail brushes may come in handy. Be prepared to try an assortment. I have washed a few houses by hand, and I find the OXO brushes pretty comfortable in the hand, but start with what you already have.

When rinsing or using the hose be particularly mindful of not spraying up at too narrow an angle which might drive water under the laps. Try to get the water on more at a right angle, where it will just cascade down.

If you're going to work on a ladder you might try getting one of those U-shaped metal stand-offs. They make it easy to scrub against a house. You can take the cleaning solution up in a bucket and just hang it over the ladder rung with a paint bucket hook.

You're going to get wet doing this, wear old clothes, and do it on a warm day.

HTH,

L.

An 'acid brush' on a painter's pole should do fine.