Fujita decomposition on families of abelian varieties

Let F: V→ B be a smooth non-isotrivial 1 - dimensional family of complex polarized abelian varieties and Vb= F- 1(b) be the general fiber. Let F1⊂ R1F∗C be the associated Hodge bundle filtration, Fb1=H1.0(Vb). Under the assumption that the Fujita decomposition for F1 is non trivial, that is there is a non trivial flat sub-bundle 0 ≠ U⊂ F1, we show that Vb has non-trivial endomorphism: End(Vb) ≠ Z.