How to determine concordance between multiple test methods when no reference method exists?

[Mike W]

Hi everyone, I would appreciate any assistance on this. Here's my dilemma. I feel like my brain is melting on this one, but it might just be easier than Im making it out to be. . .

I have 4 different quantitative tests (each one from a different manufacturer labeled Tests A-D below) that all measure the same analyte in the same unit measurement. (IU/mL)

I run 10 unique patient samples on all 4 of them - but I do not know the " true" amount of analyte in the samples/I dont have a reference test with values to compare to, I just know that they are "Positive" for the analyte being tested based on previous knowledge.

1. How do I properly show concordance/agreement based on the data from the 10 patient samples between these 4 tests? What type of plot/equation bests suits what I am trying to convey? is it Passing Bablok? Bland-Altman?
2. 2. Is concordance/agreement even the correct term I should be using here since there is no reference method/ I dont know have any reference data or reference test to compare to?

Below is an example of the data I plan to do concordance analysis with.

	Reference method Concentration Amount	Test A	Test B	Test C	Test D
sample 1	unknown	23	24	27	33
sample 2	unknown	211	257	300	200
sample 10	unknown	145	110	166	176

thank you so much and appreciate any input

 

[Miner]

@Bev D is good with these scenarios.

 

[Bev D]

Do you need to have approval from a serious regulatory body to use one or more of these tests? By regulatory body I mean FDA or USDA? If so you will need to consult a degrees biostatistician for the submission. But you can do this first to test the viability. If you do not need regulatory approval then the following approach is probably all you need. This is the Youden approach (google him - he was the authority on these types of tests in the last century…)

You will need to test each specimen TWICE on each of the 4 tests. Reproducibility and method comparison can’t be reliably assessed until intra-test repeatability is known. This can be a substantial contributor to variation so any graphical or statistical assessment will be flawed without inclusion of the repeatability variation.

Then you plot your results on what is called a Youden plot. This is a square plot (x and y axes are the same length with same increments. This essentially a special scatter/correlation plot with teh first measurement represented along the x axis and the second measurement represented along the y axis. A 45 degree 1:1 line is drawn through the plot. If there is no measurement error of any kind all points will fall on this line. Variation about the line is measurement repeatability and variation from the line shows bias. Variation along the line of the different sets is reproducibility.

Do NOT perform a. Regression calculation.

I explain ho to do this and present examples in my presentation on MSA in the resources section here. Page 49 has a graphics that will be very similar to what you describe. I also have a free spreadsheet (MSA tools) in the resources section that will plot your data…

If you post your plot we can then help you assess the results.

THEN you can perform a Bland Altman…

 

[Mike W]

Wow thank you so much for your reply Bev. I am opening your presentation on MSA as I type this.

 

[Semoi]

Just a quick remark: The formula used in the original Bland Altman paper is incorrect. For "small" sample sizes this difference matters. Therefore, I recommend to fix it.

 

[Miner]

Just a quick remark: The formula used in the original Bland Altman paper is incorrect. For "small" sample sizes this difference matters. Therefore, I recommend to fix it.

It would be helpful if you provide the correct formula.