Polymer-based nanocomposites (PNCs) are formed by dispersing nanoparticles (NPs) within a polymer matrix which creates polymer interphase regions that drive property enhancement. However data-driven PNC design is challenging due to limited data. To address the challenge we present ViscoNet a surrogate model for finite element analysis (FEA) simulations of PNC viscoelastic (VE) response. ViscoNet leverages pre-training and finetuning to accelerate predicting VE response of a new PNC system. By predicting the entire VE response ViscoNet surpasses previous scalar-based surrogate models for FEA simulation offering better fidelity and efficiency. We explore ViscoNet's effectiveness through generalization tasks both within thermoplastics and from thermoplastics to thermosets reporting a mean absolute percentage error (MAPE) of < 5% for rubbery modulus and < 1% for glassy modulus in all cases and 1.22% on tan δ peak height prediction. With only 500 FEA simulations for finetuning ViscoNet can generate over 20k VE responses within 2 minutes with 1 CPU compared to 97 days with 4 CPUs via FEA simulations.