Ask Mr. Smarty Plants

From this  Poisonous Plants of North Carolina, we learned that Buxus sempervirens,  Boxwood,  has a steroidal alkaloid in its leaves. Boxwood, as you said, is not native to North America, but to Africa, Temperate Asia and Europe, so it falls out of our area of expertise at the Lady Bird Johnson Wildflower Center. 

If we understand you, what you are looking for is a barrier to keep deer and rabbits from eating the plants in your pollinator garden, a laudable effort. You are probably correct that both deer and rabbits would avoid eating the leaves of the boxwood shrubs, although it probably would not kill them, just make them sick, so they would remember not to eat them the next time. The problem there is that deer can really jump high, so if there are goodies on the other side of a boxwood hedge, they are not going to have a problem with that. And rabbits are wonderful for getting under a hedge, perhaps even digging an entrance for all the other bunnies. 

Animals that we thought were wild are a real problem for gardeners everywhere, and we get frequent questions, especially about deer, in Texas.  We have a list of deer-resistant plants, and by using the search process for Virginia, produced this list. Unfortunately, about all the help that would be to you is that you could hopefully select some native pollinator plants from that list. Probably not what you were looking for. 

Since, as we said, we have had so many previous questions and answers about keeping animals out of the garden, we are going to provide you with links to some of those answers that we hope will give you some help in protecting your garden.  Most of these simply concern plants you might be able to use in your garden, not that will keep out the deer and rabbits. First, here is an excerpt from an answer dated November 7, 2008:

"To start with, there are no truly deer or rabbit proof plants. And sometimes rabbits, like people, will be choosy and make different choices in one yard than another right down the street. If they are hungry enough, they will eat any of these plants, and even when they're not terribly hungry, will nibble the fresh new growth on plants they would ordinarily avoid. Most browsing animals prefer not to eat plants that are aromatic or prickly but, again, in times of drought or bad weather, they'll eat what they can get.

Another thing we would like to point out about these lists is that the listed plants are not necessarily native to North America. The Lady Bird Johnson Wildflower Center is all about choosing plants native to North America and to the area in which they are being grown, because they will already be adapted and will require less water, fertilizer and maintenance to do well. Many of the annuals, including the pansies and marigolds you mentioned, are not natives and we wouldn't recommend them anyway.

University of Arizona Cooperative Extension Deer and Rabbit Resistant Plants

Better Homes and Gardens Top Rabbit-Resistant Plants (this is a slide show of photographs, with information about each plant)

Lewis Gardens Deer and Rabbit Resistant Plants"

Other answers you might want to take a look at: