View Report Details

Optimised uncertainty at minimum cost to achieve fitness-for-purpose in food analysis

Project Code: E01034

31/03/2002

University of Sussex
Ramsey, M

The ‘Optimised Uncertainty’ (OU) method has been applied for the first time to the sampling and analysis of foods. It matches the expenditure on measurements against the potential cost of the consequences that may arise from the misclassification of the food. The optimisation is achieved using estimates of the uncertainty in measurement that arises from the processes of both the primary sampling and the chemical analysis. Six applications of the OU method have been studied to evaluate the utility of the method and the potential usefulness of the approach. Three peer-reviewed scientific papers and one draft paper, attached as appendices, contain the full details of the OU method and the applications.

Uncertainty arising from the process of physical sample preparation (including both random and systematic errors) has been evaluated by a novel approach. The case study investigated was on multi-pesticide residues in retail strawberry samples. A semi-balanced hierarchical design, coupled with robust analysis of variance, was utilised to provide estimates of random error from sample preparation (sprep) during normal practice. Adaptation of the original loss function (Thompson and Fearn, 1996), central to the OU methodology, will allow for the optimisation of the sample preparation process.

This report briefly brings together all of the work, draws general conclusions, and identifies requirements for further work.

Aims and Objectives of the investigation

The scientific objectives of this research project were to develop and apply the idea of Optimised Uncertainty (OU) to the primary sampling and chemical analysis of food materials. The general approach used to estimate the uncertainty arising from both of these sources of uncertainty was that of Ramsey (1998). The concept of using a financial loss function to judge the fitness-for-purpose of measurements was first proposed by Thompson and Fearn (1996). The research aimed to develop methodologies for applying these two techniques to food analysis in general. A full description of the methodology and its first application, to the measurement of aflatoxins in pistachio nuts, is given in Ramsey et al. (2001, Appendix A).

In order to assess the feasibility and usefulness of the OU method for food analysis in general, four further food/analyte combinations were selected with a range of contrasting properties in terms of the costs of sampling and analysis, the expected uncertainty arising from sampling and analysis, and potential costs arising as a consequence of misclassification. Specific properties are described for three of these four case studies in detail in Lyn et al. (2002, Appendix B). The fourth study was that of an analyte-commodity combination characterised by high sampling costs. All of these five studies considered single food types each with a single analyte, but a sixth investigation considered optimisation of the multi-analyte situation. The monitoring of trace elements in infant food was used to exemplify this situation where one food commodity is monitored for a range of analytes (Lyn et al. 2003, Appendix C). The objective was to find the most effective way to optimise the uncertainty in this type of multi-analyte situation.

The measurement process usually involves some form of physical sample treatment prior to chemical preparation and analysis. Physical sample preparation is usually overlooked during uncertainty estimation. The objective was to estimate uncertainty of measurement caused by physical sample preparation, considering both random and systematic errors (Lyn et al. 2003, in draft in Appendix D). In view of this investigation, the final objective required the modification of the OU method for the inclusion of uncertainty and cost of physical sample preparation.

Experimental Procedures

The details of procedures are given in full in each of the four scientific papers in Appendices A-D.

The broad approach in the application of the OU method, in all cases, was:

1. To specify an experimental design for primary sampling and chemical analysis for the selected food/analyte combination, that allows the uncertainty to be estimated by the method of Ramsey (1998).

2. Sampling of foods either directly by, or monitored by, project staff.

3. Chemical analysis by subcontracted labs, to the agreed experimental design, with normal in-house quality control, and all measurements reported in unrounded and untruncated form (e.g. no censorship below nominal detection limits).

4. Calculation of measurement uncertainty and its components, for selected analytes from each case study, using robust Analysis of Variance.

5. Construction of a loss-function based on estimates of consequence costs and the calculation of optimal uncertainty and minimal expectation of loss.

6. Calculation of optimal division of expenditure between sampling and chemical analysis.

7. Comparison of the optima found with the original un-optimised values, and consideration of what steps are needed to achieve the required changes.

The general approach to the estimation of uncertainty from physical sample preparation was:

1. To specify an experimental design which allows for the estimation of uncertainty from physical sample preparation (including systematic error estimation from this source) in addition to primary sampling and chemical analysis.

2. Sampling and chemical analysis by contracted laboratory (CSL, York) to the agreed experimental design and monitored by project staff.

3. Calculation of measurement uncertainty, including individual estimates of random error from primary sampling and both random and systematic errors from sample preparation and chemical analysis using robust analysis of variance.

The general approach for the adaptation of OU method for optimisation of sample preparation processes:

1. Construction of a loss function that allows for the optimisation of measurement uncertainty (including physical sample preparation) and optimal apportionment of expenditure between primary sampling, sample preparation and chemical analysis.

Results and Discussion

The findings of each experiment have been reported fully in the four scientific papers (Ramsey et al. 2001, Lyn et al. 2002, Lyn et al. 2003 and Lyn et al. draft, Appendices
A-D). A comprehensive description and analysis of the OU case studies to date can be found in Lyn, 2003. An overview of the conclusions can be summarised as follows.

1. The first application of the OU method to aflatoxin in pistachio nuts demonstrated the feasibility of this new approach.

2. The detailed recommendations from this first application, suggested that the measurement uncertainty could usefully be reduced by 31% (Ramsey et al. 2001). This could be achieved, for example, by increasing the expenditure on sampling by a factor of four (from £6 to £26). This is predicted to lower the overall expectation of loss by over £400 per batch.

3. Recommendations from the four further applications (Lyn et al. 2002 and Lyn, 2003) showed the wider applicability of the OU method, across a wide range of situations. Generally reductions in uncertainty (of around 20-40%) were judged as economically justified, and increases in expenditure on sampling were identified as being the most cost effective way to achieve this. This was not only because the sampling was generally the limiting source of uncertainty, but also because the sampling in three of the cases was much cheaper and therefore a much more cost-effective source of improvement. The single case of high sampling costs indicated that the most cost-effective source of improvement was the chemical
analysis. The analytical source of uncertainty was not limiting for two cases, and a decrease of expenditure was recommended. In effect, this indicated that a transfer of funds from analysis to sampling was the best way of reducing uncertainty. In the two other cases improvement in both sampling and analysis were indicated as being beneficial. There were two cases where the OU method indicated that an increase (rather than decrease) in the overall measurement uncertainty was financially advantageous. These decreases would each result in decreases in the expectation of financial loss of c.30% (>£20 per batch). This indication arose for one case, partially because the financial consequences of misclassification were relatively low (i.e. false non-compliance for the measurement of meat concentration in sausages) and for the other due to the high cost of sampling (i.e. false non-compliance for pesticide (propargite) in wholesale apples). This implies that the current measurement practises are too precise for the intended purpose, and could therefore be made less precise to allow more uncertainty. This is exemplified by the decrease in expenditure indicated for sampling and analysis in the aforementioned cases.

4. The two options for calculating the optimal value of uncertainty, visual inspection and Newton-Raphson, were both shown to be effective but to have different advantages. Visual inspection is mathematically simpler and also provides a graphical representation that shows the width of the area of optimal uncertainty, as well as the minimum value. This can be useful in judging how close to the optimal value a practical solution needs to be. The Newton-Raphson method is more mathematically precise, but may be harder to be interpreted by users.

5. Achieving the exact value of the minimum cost calculated by the OU method might not be achievable in real terms, and might often not be necessary. The general recommendations of the OU method, such as a need to half the sampling uncertainty, are most useful rather than using it to achieve an exact level of uncertainty.

6. The OU methodology was successfully adapted for use with a type of foodanalyte combination that has a target range of composition, rather than a simple threshold value (e.g. fat concentration in spreadable fats). This demonstrated the more general applicability of the approach.

7. In the case of optimising the sampling and analysis for a food product with multiple analytes, the conclusion was to apply the OU method to the most ‘sensitive’ analyte. Identification of this most sensitive analyte depends on a combination of the analyte with an expected concentration values closest to its threshold value, and also by consideration of the expectation of loss for that element.

8. The optimal amount of money to be spent on sampling varied considerably (from £4 to £190) depending on the type of food-analyte combination and the misclassification scenario, as did that for chemical analysis (from £4 to £170). This shows how the OU method can be used to judge what expenditure is most effective for any particular food/analyte combinations.

9. The OU method makes it possible to compare the optimal amount of overall measurement expenditure between different food/analyte combinations. Comparing across the five food types studied here, the one that was liable to give rise to the highest expectation of financial loss was the ‘fat in spreadable fats’ (Lyn et al. 2002, Appendix B). At over £1000 per batch un-optimised, and still nearly £700 when optimised, this is much higher than the optimised values for the other foods (all <£400). These were however, only initial findings using a new method, and further improvements to the OU method (see next section) could produce more robust recommendations.

10. The costs of uncertainty estimation for the analyte-commodity combinations used this assessment of the OU methodology range from £307 (added water in milk) to £3560 (propargite in wholesale apples). The additional costs can be justified by the information provided as a result. Without information concerning uncertainty the potential for erroneous decisions can go unnoticed with the detriment of the fitness for purpose of the measurements made. For the case studies of ‘added milk in water’ and ‘propargite in apples’, the potential savings in expected loss, following optimisation (under false compliance scenario), were £70 and £470 respectively (Appendix E). Although the cost of uncertainty estimation outweighs these predicted savings for a single batch, the uncertainty estimates could be used as input parameters in successive surveys and, in this respect, the extra expense is justified over less than eight batches.

11. The Expectation of Loss function (Thompson and Fearn, 1996 and Ramsey et al. 2001) has been successfully adapted for the additional optimisation of uncertainty arising from the process of physical sample preparation. Full details of the mathematical formulae required for the optimisation are given in Appendix F.

12. Uncertainty from sample preparation can be estimated using a top-down approach to uncertainty estimation. The system of fully duplicated sampling and sample preparation and semi-duplicated chemical analysis, coupled with robust Analysis of variance provided estimates of sample preparation uncertainty (sprep)(and also ssamp and sanal). A full description of the methodology can be found in Appendix D.

13. Systematic error arising through sample preparation has been estimated from spike recovery trial data. This source of error has been shown, in some cases, to dominant the systematic error with up to 50% loss of the analyte from sample preparation alone. The additional random error incurred through the correction of such systematic error can dominate the random component of measurement uncertainty (e.g. for the measurement of tolylfluanid in strawberries, Appendix D).

14. Four possible options for calculating and expressing the uncertainty, some including systematic error, have been highlighted and exemplified by the case study of the pesticide tolylfluanid in retail strawberry samples.

15. It is recommended that estimates of the contribution to the overall uncertainty arising from sample preparation be made and expressed, in addition to other error estimates from the processes of sampling and analysis. Where possible, estimates of both random and systematic errors should be computed. If the systematic errors are ignored, the analyte concentration estimates will not represent the true value and any decisions made as a result of the data may be flawed.

Requirements for further work

A clear need for further development, refinement and application of the OU method was identified.

1. The loss function applied in this example was just one of many possible functions, and other alternative functions may be advantageous for some applications.

2. The measured concentration of the constituent of the food was considered at particular fixed levels in these studies, but the method could be refined to look at the cost implications across a range of concentration (discussed in detail in Ramsey et al. 2001).

3. The case studies used here to demonstrate the OU method were of necessity and limited in scope. The FSA has a wide range of methods with very different variables (such as consequence costs) that could usefully be examined.

4. Values for several input variables are required for application of the OU method. An investigation to establish of typical values for various analyte/food combination needs to be undertaken to make a source of default values that could be used in preliminary application of the OU method, prior to making actual measurements of uncertainty.

5. Assess the general applicability for sample preparation uncertainty estimation in food analysis by application of the methodology to a wider range of analytematrices combinations.

6. Application of Expectation of Loss function, adapted for sample preparation optimisation, to a greater range of analyte-commodity combinations. Optimisation of the sample preparation process would in many cases require multi-analyte optimisation (e.g. Lyn et al. 2003). Assessment of this approach is also required.

7. The OU method has been shown to work for a range of analyte-commodity combinations, but no test has yet been made of whether to potential savings indicated can be realised in practise. Further work is required therefore to take the actions suggested by the OU method (e.g. changing expenditure and uncertainty of sampling and/or analysis) and comparing the consequences with those predicted by the OU method. (This objective is being addressed in a new contract E01055).

8. To consider the optimal expenditure that is justified for the estimation of uncertainty. A high quality estimate of uncertainty will be expensive and may not be necessary in order to provide sufficient information to reduce the overall expenditure. Research is therefore required into establishing the most costeffective approach to uncertainty estimation.

Main File

Some of the files on this site may be in a format that your computer can't read. However, you can download Readers and Viewers for the following document types below: