Sounds like GE II for Bathrooms. Some things are unpleasant and only show on the MSDS for the substance (labels are rarely informative and I cannot remember what GE II bathroom says on the MSDS; you could find it online), but in general people feel that the potential presence of mold-inhibitors in that silicone is a cause for concern. May have no effect, may have an effect. I've seen issues in reef aquaria attributed to the presence of mold inhibitors in silicone, but correlation does not signify causation. Anyone can kill a whole tank and blame it on anything. "Aquarium safe" rarely means anything other than they've marked up the price.
You could try planting everything after the silicone is well cured (a week or until all trace of curing smells is gone) and very well rinsed, and then just let the plants sit and go for a month or so. It would be an experiment, can't really say how big a problem it might be, or if the "mold inhibitors" are anything but colorful marketing in that particular silicone. If you're worried about it enough to not want to do that, you've answered your own question and can do a tear down.