Wood is not heavy at all if you use it properly.
I built a full camper in 2001 with panels made from 2.7mm thick luan plywood skins, with XPS foam and vertical pieces of fir every ~17" in the core. At that time I could get 3/4" foam, and the 1x2 boards are 3/4" thick so I didn't need to do much cutting. All glued together with construction glue (PL Premium is a good choice). Then an external coat of hand laid fiberglass, and then paint. I was very pleased with how it turned out. Light, cheap, nice wood interior, insulated, and tough. Happy to give more details if you are interested.
One thing that just occurred to me is that newer Tacomas have a composite bed, and I'm not sure if you could put much weight on the bed-rails. Also all of them have torsionally flexible frames (the old PU frames were fully boxed and stiff) which makes camper mounting more of an issue. The problem is if you hard mount the camper to the frame or bed rails, then when the frame flexes the camper needs to flex as well. The panels I described above are stiff, and I'm not sure if they could take that. If the back is just a big hatch/door it would probably be fine, as an open-ended box flexes more easily.
Oh ya, as JanaBanana said, if you have the shortbed, it's going to be tight trying to sleep back there.