The insulation is necessary for your comfort level if you plan to stay in your minivan fulltime. The whole rear of my van is covered in rtech foam insulation, every square inch. From trial and error I found out a little insulation works but alot of insulation works even better, and adding some more after that won't hurt.
On my van I didnt remove any panels, I just put the rtech foam insulation on top of the panels. You can hotglue or use gorilla duct tape if you plan to remove later. If you don't like the cold, you will need several layers. And also cover all areas where air might come in, in windy conditions cold air will force its way in your van through the smallest holes. I actually had to duct tape the door seams that I dont use from the outside to keep the cold air from coming in.
In some of the warmer areas like southern california, the daytime temps in winter will be in the 70s but at night they can get in the 40s. Thats the reason for all my insulation, I can deal with the hot weather but cold gets very uncomfortable quickly.
Temperatures in my van without heater in the winter. Without insulation the inside and outside temps would be the same. Anything below 50f requires a sleeping bag. I havent had to use a heater or sleeping bag in the past 3 winters.