Silicone is obstinate stuff. Plastic scraper and a fine stainless wire brush followed by some acetone and paper towels. I recommend the no-sag Dicor instead of silicone. I see some spider cracks in there too so you might have to glob over those. If you use eterna-bond or peel-n-seal you can spray paint to match color so it's not so obvious. Another good repair for surface stuff like that if you don't have large gaps is aluminum foil tape, and you can paint over that too.