I would try liquid Bissell carpet cleaner with warm water and a nail brush, or liquid Tide w/o bleach.

This is one of the products we use in the Rug Cleaning Biz

Scroll down to the Goo Remover under the Select a Product.

Here is a link that might be useful: Goof-Off

I had a similar thing happen when the painter duct taped down some strips of paper. The glue residue attracted dirt, and soon everywhere there had been tape, there was a dirty mark.

I called the people in the link below (see bottom of page), and the woman was very helpful. They gave me specific directions on how to remove the residue, and it worked.

Here is a link that might be useful: stain removal guide