Serotonin (5-HT)-induced stimulation or progesterone (P4) production by bovine luteal cells was characterized with respect to the receptor subtype mediating this response, the steroidogenic response to 5-HT metabolites, the role of adenylate cyclase, and the 5-HT concentration of bovine luteal tissue. Addition of 5-HT (10(-5) M) stimulated the production of P4 (P less than 0.05) and this stimulation was inhibited by the 5-HT antagonist mianserin at a concentration of 10(-5) M (P less than 0.05), but not at a mianserin concentration of 10(-7) M. Additionally, the response to 5-HT could not be inhibited by ketanserin (10(-5) M), a 5-HT2 receptor antagonist. Incubation of luteal cells with a specific 5-HT1 agonist, (+/-)-8-hydroxydipropylaminotetralin HBr (DPAT) (10(-4) M), stimulated the production of P4 (P less than 0.05) and this response could not be blocked by mianserin at 10(-7) M or by ketanserin, but was inhibited by mianserin at 10(-5) (P less than 0.05). The addition of the 5-HT metabolite 5-methoxytryptamine (5-MTA) stimulated P4 production (P less than 0.05) and this response could be inhibited by mianserin (10(-5) M, P less than 0.05). Neither, N-acetyl-5-HT nor 5-methoxytryptophan significantly affected P4 production. The addition of the phosphodiesterase inhibitor 3-isobutyl-methylxanthine (IBMX, 0.1 mM) potentiated the effects of 5-HT and DPAT (P less than 0.05), but this effect was additive rather than synergistic. In contrast, the addition of luteinizing hormone (10 ng/ml) plus IBMX resulted in a significant synergistic response (P less than 0.05).(ABSTRACT TRUNCATED AT 250 WORDS)