The pharmacology, calcium dependence and G protein mediation of the membrane hyperpolarization of rat dorsolateral septal nucleus (DLSN) neurons in response to nicotinic agonists was examined to classify the nicotinic receptor mediating the response. Intracellular recording from DSLN neurons in a brain slice preparation was used to determine whether chlorisondamine, trimethaphan, cytisine or strychnine inhibited the membrane hyperpolarization in response to application of the nicotinic agonist 1,1-dimethyl-4-phenylpiperazinium (DMPP). Chlorisondamine was found to block the response only at a high concentration (500 microM) although strychnine (100 microM) was without effect. Cytisine was neither an effective agonist nor an antagonist (500 microM). Surprisingly, trimethaphan appeared to act as an agonist, rather than an antagonist, with a potency and efficacy similar to that reported for nicotine at this receptor. The response was dependent on intracellular calcium stores because it persisted in the absence of extracellular calcium but was blocked by intracellular injection of 1,2-bis(2-aminophenoxy)ethane-N,N,N',N'-tetraacetic acid (BAPTA). Injection of GTP gamma S into the neurons blocked the nicotinic response. Apamin, iberiotoxin and charybdotoxin reduced but did not block the response at concentrations that selectively block calcium-dependent potassium channels. These results indicate that the nicotinic response in DLSN neurons may be mediated by a metabotropic nicotinic receptor coupled to a calcium-dependent potassium channel through the activation of a G-protein and release of intracellular calcium stores. The unusual pharmacology of the nicotinic receptor on DLSN neurons indicates that it may be a novel receptor which has yet to be cloned.