Are there special rules for conducting a 2^3 factorial DoE where the yield is subgroup data?

Do you mean that for each experimental run, you are collecting multiple measurements?

If this is the case, this "subgroup" would consist of experimental repeats. Repeats are multiple measurements taken during a single setup of an experimental run. (If the experimental conditions are changed then at a later time are recreated as a new setup, this is called a replicate).

As a general practice, replicates are better than repeats because you are more likely to be able to duplicate experimental results from replicated experiments.

With repeats, the typical method is to calculate the mean and standard deviation of each subgroup. Perform the analysis on the mean to determine the usual information on main effects and interactions. Calculate the log of each standard deviation, then perform a similar analysis on the log(StDev). Look for factors that will significantly reduce the log(StDev) in order to minimize variation.

This is even more effective when used with an outer array that deliberately induces variation into the repeats.

I was thinking more on the order of something like shooting a target 10 times per run. I.e.,

2^3 with 10 shots per run.

Those are repeats.

If you took one shot per run, then changed the setup to another random run then back to the original run for another shot and did the same another 8 times you would have 10 replicates.

The distinction is extremely important because replicates contain more variation than repeats. If factors are significant across repeats, but not across replicates you will not be able to consistently reproduce the results of your experiment.

I don't think I can do that. My thought was to shoot a 10 round group, calculate the mean radius, record values for shot 1-10 and mean radius. The problem is I need to capture the effects on the rifle system over 10 shots.

Are you literally talking about shooting a rifle?

Is your objective to reduce the spread of the 10-shot pattern? If that is the case, then you do want to run repeats, then analyze using a measure of the spread such as the minimum circumscribed diameter of the pattern.

If you can run replicates of the 10-shot patterns, I would recommend doing so.

Yes, for accuracy.