Melatonin signal transduction was examined in median eminence/pars tuberalis (ME/PT) explants from Djungarian hamsters. High affinity melatonin receptors in hamster ME/PT were first quantified by in vitro autoradiography using the potent melatonin agonist 125I-labeled melatonin ([125I]MEL). Scatchard analysis of [125I]MEL binding in ME/PT revealed high affinity receptors [dissociation constant (Kd) = 2.75 X 10(-11) M]. [125I]MEL binding was markedly reduced by guanine nucleotides; treatment with the nonhydrolyzable GTP analog guanosine 5'-O-(3-thiotriphosphate) caused a 10-fold decrease in receptor affinity. Melatonin (10 nM) significantly inhibited forskolin-stimulated cAMP accumulation in ME/PT, but not in pituitary or pineal glands. In ME/PT explants, melatonin and 6-chloromelatonin inhibited forskolin-stimulated cAMP accumulation in a dose-dependent manner with similar potency (significant inhibition for each at concentrations greater than or equal to 100 pM). Serotonin significantly inhibited forskolin-stimulated cAMP levels only at doses greater than or equal to 100 microM. Inhibition of [125I]MEL binding in ME/PT by these three indolamines paralleled that determined for inhibition of forskolin-stimulated cAMP accumulation. Pertussis toxin treatment (1 microgram/ml) blocked the ability of melatonin (10 nM) to inhibit forskolin-stimulated cAMP accumulation and significantly reduced [125I]MEL binding. Pertussis toxin ADP-ribosylated the alpha-subunits of at least two guanine nucleotide-binding proteins in ME/PT explants with molecular weights of approximately 40 K. Melatonin did not increase phosphodiesterase activity in ME/PT explants. The results strongly suggest that a signal transduction pathway for melatonin in mammals involves inhibition of adenylyl cyclase by a pertussis toxin-sensitive guanine nucleotide-binding protein.