The canine coronary artery possesses a smooth muscle relaxant serotonin (5-HT) receptor distinct from previously characterized 5-HT receptors. On the basis of the ability of LY53857 to block weakly coronary smooth muscle relaxation to 5-HT, we examined several structurally related ergolines in endothelium denuded rings of canine coronary artery precontracted with PGF2 alpha (10 microM). 5-HT (10 nM-100 microM)-induced relaxation was antagonized competitively by the ergoline esters LY53857 (-log KB = 6.5) and sergolexole (-log KB = 6.4) and by the ergoline amide amersergide, (-log KB = 6.7). In contrast to the relatively low affinity of these ergolines, LY215840, another ergoline amide, antagonized 5-HT-induced relaxation in a competitive manner with the highest affinity (-log KB = 8.3). This effect was independent of the 5-HT2 receptor affinity of these ergolines, because LY215840, LY53857, sergolexole and amesergide all possessed similar 5-HT2 receptor affinity. Further, all four ergolines possessed affinity for the human 5-HT7 receptor, and LY215840 had the highest 5-HT7 receptor affinity (Ki = 14.7 nM). Finally, in vascular smooth muscle under basal tone, LY215840 (1 microM) blocked the relaxant response to high concentrations of 5-HT and 5-MeOT without altering their contractile potency. LY215840 (1 microM) did not alter contraction to sumatriptan, an agent that lacks relaxant activity. In contrast, LY215840 (1 microM) markedly potentiated contraction to 5-carboxamidotryptamine, the most potent coronary relaxant agonist and the agonist with the highest 5-HT7 receptor affinity. The ability of LY215840 to block the relaxant 5-HT receptor in canine coronary artery may reflect its 5-HT7 receptor antagonist activity and make it a useful tool to probe the relationship between the 5-HT7 receptor and the coronary vasoactive properties of 5-HT.