Serotonin-induced DNA synthesis in bovine aortic smooth muscle cells was totally abolished by pretreatment of cultures with 5 ng/ml pertussis toxin. The half maximally effective concentration of toxin was approximately 10 pg/ml. Pertussis toxin did not affect platelet-derived growth factor (PDGF)-stimulated DNA synthesis and actually enhanced the mitogenic effect of the phorbol ester, phorbol 12-myristate 13-acetate. Pertussis toxin did not inhibit serotonin-stimulated inositol phosphate accumulation or increases in intracellular calcium or cAMP concentrations under conditions sufficient to completely inhibit serotonin-induced (3H)thymidine incorporation. These results demonstrate that a novel, pertussis-sensitive pathway is required for serotonin-, but not platelet-derived growth factor-induced DNA synthesis in vascular smooth muscle cells. The pertussis-sensitive step does not involve cAMP, phosphoinositide hydrolysis, mobilization of intracellular calcium, or phorbol ester-sensitive protein kinase C activity.