"Firstly, not sure if my current mortgage holder would permit it."

They may not even notice since no change in title will occur until the contract is fulfilled.

"Secondly, if he defaulted, couldn't that make things messy sorting things out and freeing up the land for resale?"

There would not be anything in the chain of title to sort out.
He failed to fulfill the contract, the title has not been altered in any way.
You still own the property the whole time.

There is no foreclosure process since the buyer has no ownership interest.

You do need to establish who will pay property taxes and other costs associated with the land.
Since you still own the land you would be ultimately responsible for the carrying costs.

Most of the risk is on the purchaser.
They can default under the contract, or the seller could fail to deliver clear title or otherwise encumber the property.

A good RE lawyer can set everything up for you.

Read this below from a Colorado State legal perspective.
An attorney should be handling everything!

Here is a link that might be useful: Pitfalls of Land Installment Contracts

"Firstly, not sure if my current mortgage holder would permit it."
I would be checking with the mortgage holder first thing.

Example...a house setting on a double lot, or with an extra lot is of more value than on a single lot. If that lot is part of the mortgage, I wouldn't think you could just sell it off without an OK from the mortgage company. Had you maybe planned to apply the proceeds to the mortgage?

"have the escrow agent return the escrowed deed"

Get a local RE attorney to handle the job.

Many of the comments must be applicable in Colorado, but are FAR frm uniform.

In Virgina deeds are not "escrowed" and a deed signed but not delivered could easily be challenged as conditionally executed and invalid.

This does leave the buyer vulnerable to the seller transferring the property to someone else, leaving the buyer to try and recover their money when the deed cannot be delivered.

Thanks for the info so far from everyone. It sounds like most of the risk is on the buyer's end, but there could be some issues on the seller's end as well. Are there other ways to meet the buyer's need for a delayed sale while minimizing risk on our end?

I did talk to my mortgage holder about a potential sale and they said they'd do a partial lien release on the lot if we sold it. We have about 50% equity in the combined property, so that leaves them plenty of collateral. Then I asked if they'd do a partial lien release without a sale and they said no, unless I forked over some cash (works out to about 1/3 the market value of the land). I haven't asked them whether they'd permit a contract for deed, but I'd really rather not go down that road.

"Are there other ways to meet the buyer's need for a delayed sale while minimizing risk on our end? "

Rent with an option to buy.