I thought the real estate agent kept the check in escrow and returned it to you if the sale went through.

Sorry, I should have been clearer. I wrote the check to the real estate company, so it was deposited to an escrow account (I assume). Previous to this, we made a back-up offer on another house. The seller's agent on that house held the check, but did not deposit it to an escrow account. My questions is can the seller's agent deposit the check and hold the money in escrow even though my offer is only a back-up offer?

A COPY of the check is all the buyer needs to present with an offer, until a contract is executed. Once the contract is executed, the agent has 3 days to get it into the escrow account.
You should have kept the check until contract. If you did hand over the check, the agent has no choice but to put it into escrow. The first agent should not have kept hold of a clients check for such a long time, w/o depositing it into an escrow account.