In mammalian striatal tissue and cat platelets, [3H]ketanserin labels besides serotonin-S2 receptors nonserotonergic saturable binding sites. The sites have been distinguished and characterized in [3H]ketanserin binding assays by selective inhibition with tetrabenazine (Ki = 4 nM), a monoamine depleting agent. In rats, the nonserotonergic ketanserin sites were enriched in the striatum (KD = 12.4 +/- 0.3 nM, maximal number of binding sites = 53.2 +/- 11.8 fmol/mg of tissue at pH 7.7, 37 degrees C) and nucleus accumbens. The sites were decreased by 65 to 78% after 6-hydroxydopamine lesions, suggesting an association with dopaminergic nerve terminals. In in vitro superfusion experiments using [3H]dopamine, [3H]norepinephrine and [3H]serotonin loaded rat brain tissue and [3H]serotonin loaded human platelets, 5 min superfusion with 10(-6) M ketanserin, tetrabenazine and reserpine caused instantaneously a marked increase in tritium efflux. The effect was attenuated by the monoamine oxidase inhibitor, pargyline, in brain slices but not in platelets. High-performance liquid chromatography analysis of endogenous catecholamines, serotonin and metabolites in superfusates from striatal slices revealed that stimulation with these drugs provoked mainly release of 3,4-dihydroxybenzeneacetic acid, homovanillic acid, and 5-hydroxyindoleacetic acid. Potencies of a series of ketanserin derivatives, benzoquinolizine derivatives and a variety of drugs affecting neurotransmission were assessed in the in vitro release test using [3H]dopamine loaded striatal slices, and in [3H]ketanserin binding assays to nonserotonergic sites in the striatum and to serotonin-S2 receptors in brain tissue. Activities of drugs in the release test correlated strongly with their binding affinities for nonserotonergic ketanserin sites (rs = 0.83, n = 30, P less than .001). High potency in the latter two tests was confined to few close structural congeners of ketanserin and tetrabenazine. Distinct structural activity relationships for interaction with nonserotonergic ketanserin sites and serotonin-S2 receptors were found. It was concluded that nonserotonergic ketanserin sites mediate release of oxidated metabolites of biogenic amines from nerve endings and of serotonin from platelets. Hence release of biogenic amine metabolites or of cytoplasmic amines is probably not a mere diffusion process but involves specific membranous molecules. Unlike tetrabenazine, ketanserin caused no obvious depletion of central catecholamine and indoleamine stores. Implications of these findings for the mechanism of action of the drugs are discussed.