I spent some time over the last week removing the caryopteris. I didn't replant them in another spot, I just cut them out. They propagate very easily and I have a couple plants in other locations I can use for propagation. I still have some clearing to do before I attempt mulching. Wild strawberries and blackberries invaded the undergrowth and need removed. Also there are some irises that need a much better home since they have way too much shade here now.
My biggest challenge will be keeping the deer away. As a temporary solution I purchased some netting to cover the plants after I transplant them over. I'll make up a deer repellent spray soon to ward off the deer. The problem with that is I have to reapply after rains. It's just another one of those gardening challenges that we'll have to overcome!