You could try a heat gun to loosen the glue, but on a textured wall, the glue will fill in the textured areas to the point where if you can get it smooth enough, skim coat the entire area and re-texture to match.

I had a similar situation in a rental property I owned when I tore paneling off a wall. I just tore all the glue off, without regard as to if the paper on the drywall came off or not. I had hundreds of gouge marks, tears,etc. Take a spray bottle full of water and wet those areas, then fill them in with drywall mud. Wetting the areas keeps the drywall from sucking up the moisture from the patch, which can make adhering a problem.

I ended up painting that wall in a semi-gloss, which can REALLY show up any defects. Even looking down the wall from an angle, you couldn't tell that anything was done to the wall!! So that repair went very well for me and with some attention to detail, you can do the same with your wall.