Spectroscopic, calorimetric, DNA cleavage, electrophoretic, and computer modeling techniques have been employed to characterize the DNA binding and topoisomerase poisoning properties of three protoberberine analogs, 8-desmethylcoralyne (DMC), 5,6-dihydro-8-desmethylcoralyne (DHDMC), and palmatine, which differ in the chemical structures of their B- and/or D-rings. DNA topoisomerase-mediated cleavage assays revealed that these compounds were unable to poison mammalian type II topoisomerase. By contrast, the three protoberberine analogs poisoned human topoisomerase I according to the following hierarchy: DHDMC > DMC > palmatine. DNA binding by all three protoberberine analogs induced negative flow linear dichroism signals as well as unwinding of the host duplex. These two observations are consistent with an intercalative mode of protoberberine binding to duplex DNA. However, a comparison of the DNA binding properties for DMC and DHDMC, which differ only by the state of saturation at the 5,6 positions of the B-ring, revealed that the protoberberine analogs do not "behave" like classic DNA intercalators. Specifically, saturation of the 5-6 double bond in the B-ring of DMC, thereby converting it to the DHDMC molecule, was associated with enhanced DNA unwinding as well as a reversal of DNA binding preference from a DNA duplex with an inaccessible or occluded minor groove {poly[d(G-C)]2} to DNA duplexes with accessible or unobstructed minor grooves {poly[d(A-T)]2 and poly[d(I-C)]2}. In addition, a comparison of the DNA binding properties for DHDMC and palmatine revealed that transferring the 11-methoxy moiety on the D-ring of DHDMC to the 9 position, thereby converting it to palmatine, was associated with a reduction in binding affinity for both duplexes with unobstructed minor grooves as well as for duplexes with occluded minor grooves. These DNA binding properties are consistent with a "mixed-mode" DNA binding model for protoberberines in which a portion of the ligand molecule intercalates into the double helix, while the nonintercalated portion of the ligand molecule protrudes into the minor groove of the host duplex, where it is thereby available for interactions with atoms lining the floor and/or walls of the minor groove. Furthermore, saturation at the 5,6 positions of the B-ring, which causes the A-ring to be tilted relative to the plane formed by rings C and D, appears to stabilize the interaction between the host duplex and the minor groove-directed portion of the protoberberine ligand. Computer modeling studies on the DHDMC-poly[d(A-T)]2 complex suggest that this interaction may involve van der Waals contacts between the ligand A-ring and backbone sugar atoms lining the minor groove of the host duplex. The hierarchy of topoisomerase I poisoning noted above suggests that this minor groove-directed interaction may play an important role in topoisomerase I poisoning by protoberberine analogs. In the aggregate, our results presented here, coupled with the recent demonstration of topoisomerase I poisoning by minor groove-binding terbenzimidazoles [Sun, Q., Gatto, B., Yu, C., Liu, A. , Liu, L. F., & LaVoie, E. J. (1995) J. Med. Chem. 38, 3638-3644], suggest that minor groove-directed ligand-DNA interactions may be of general importance in the poisoning of topoisomerase I.