diff --git a/.gitignore b/.gitignore index f15b7e7..235dcbc 100644 --- a/.gitignore +++ b/.gitignore @@ -8,4 +8,5 @@ # Local testing environment files Gemfile.lock -docs/_site/ \ No newline at end of file +docs/_site/ +.Rproj.user diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 6621b7f..0e4a3bb 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -23,13 +23,11 @@ Remember to run `git pull` regularly to keep up to date with the main branch. The 'main' branch is protected, meaning you will have to create a new branch to make any changes. -To do this, on your local machine, first update your 'main' branch using `git pull`. Then run `git checkout -b branch-name` to create and switch to a new branch. You can switch between branches (for example, to work on different fixes) by using `git checkout branch-name`. - -Branch names should be short but descriptive, with each word separated by a hyphen. Author initials should be added to the prefix. For example: `"bc-update-data-vis"`. +To do this, on your local machine, first update your 'main' branch using `git pull`. Then run `git checkout -b branch-name` to create and switch to a new branch. You can switch between branches (for example, to work on different fixes) by using `git checkout branch-name`. Branch names should be short but descriptive, with each word separated by a hyphen. Use `git add` and `git commit -m "a clear descriptive commit message"` as usual to stage and commit your changes. -Once you have finished making your changes, push your branch to GitHub (using `git push` - it may give you instructions for setting an upstream branch). Then log in to GitHub and raise a pull request to have your changes reviewed and merged into 'main'. +Once you have finished making your changes, push your branch to GitHub (using `git push` - it may give you instructions for setting an upstream branch like using git push --set upstream origib branch-name). Then you can click the link it gives you to open GitHub and raise a pull request to have your changes reviewed and merged into 'main'. ## Building websites with Jekyll and GitHub diff --git a/docs/_layouts/guidance-page.html b/docs/_layouts/guidance-page.html index 029f38d..ad32a58 100644 --- a/docs/_layouts/guidance-page.html +++ b/docs/_layouts/guidance-page.html @@ -24,7 +24,8 @@ - + + @@ -57,4 +58,4 @@

{{ page.title }}

- \ No newline at end of file + diff --git a/docs/assets/img/missing data/missing-data-ex5.svg b/docs/assets/img/missing data/missing-data-ex5.svg new file mode 100644 index 0000000..b5e67aa --- /dev/null +++ b/docs/assets/img/missing data/missing-data-ex5.svg @@ -0,0 +1,70 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +0 +1 +2 +3 +4 +5 + + + + + + + +2018 +2019 +2020 +2021 +2022 +2023 +Year +Value + + diff --git a/docs/assets/img/missing data/missing-data-ex6.svg b/docs/assets/img/missing data/missing-data-ex6.svg new file mode 100644 index 0000000..e08de5a --- /dev/null +++ b/docs/assets/img/missing data/missing-data-ex6.svg @@ -0,0 +1,73 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +No data for 2020 +(COVID-19 pandemic) + + +0 +1 +2 +3 +4 +5 + + + + + + + +2018 +2019 +2020 +2021 +2022 +2023 +Year +Value + + diff --git a/docs/assets/img/missing data/missing-data_bad-chart-example.png b/docs/assets/img/missing data/missing-data_bad-chart-example.png deleted file mode 100644 index d23a14e..0000000 Binary files a/docs/assets/img/missing data/missing-data_bad-chart-example.png and /dev/null differ diff --git a/docs/assets/img/missing data/missing-data_good-chart-example.png b/docs/assets/img/missing data/missing-data_good-chart-example.png deleted file mode 100644 index d1fbacd..0000000 Binary files a/docs/assets/img/missing data/missing-data_good-chart-example.png and /dev/null differ diff --git a/docs/assets/img/permalink.png b/docs/assets/img/permalink.png new file mode 100644 index 0000000..47b2e8f Binary files /dev/null and b/docs/assets/img/permalink.png differ diff --git a/docs/_includes/expandable-sections.js b/docs/assets/scripts/expandable-sections.js similarity index 100% rename from docs/_includes/expandable-sections.js rename to docs/assets/scripts/expandable-sections.js diff --git a/docs/assets/scripts/section-links.js b/docs/assets/scripts/section-links.js new file mode 100644 index 0000000..5d9051e --- /dev/null +++ b/docs/assets/scripts/section-links.js @@ -0,0 +1,32 @@ +// Function to copy link to clipboard +function copylink(text) { + navigator.clipboard.writeText(text).then(() => { + alert(`Link copied to clipboard:\n${text}`); + }).catch(err => { + console.error('Failed to copy link', err); + }); +} + +// Add copy link icons to all heading elements +document.querySelectorAll('h1, h2, h3, h4, h5').forEach(heading => { + // Ensure the heading has an ID for the anchor link + if (!heading.id) { + heading.id = heading.textContent.trim().toLowerCase().replace(/\s+/g, '-').replace(/[^\w-]/g, ''); + } + + // Create the icon element + const iconWrapper = document.createElement('span'); + iconWrapper.className = 'section-link'; + iconWrapper.title = 'Section link'; + iconWrapper.innerHTML = ' '; + + // Append the icon to the heading + heading.appendChild(iconWrapper); + + // Add event listener for copying the link + iconWrapper.addEventListener('click', (e) => { + e.stopPropagation(); // Prevent the click event from bubbling up + const url = `${window.location.origin}${window.location.pathname}#${heading.id}`; + copylink(url); + }); +}); diff --git a/docs/assets/styles.css b/docs/assets/styles.css index bd132af..187016d 100644 --- a/docs/assets/styles.css +++ b/docs/assets/styles.css @@ -143,6 +143,30 @@ ul, ol{ } +/*Section links*/ +.section-link{ + background-image: url('img/permalink.png'); + width: 18px; + height: 18px; + opacity: 0; + margin-left: 5px; + display: inline-block; + background-size: cover; +} + +.section-link:hover{ + cursor: pointer; +} + +h1 span, h2 span, h3 span, h4 span, h5 span { + transition: opacity 0.3s ease-in-out; +} + +h1:hover span, h2:hover span, h3:hover span, h4:hover span, h5:hover span{ + opacity: 0.5; +} + + /*Fancy elements*/ blockquote{ margin: 1em 0 2em 0; @@ -342,4 +366,4 @@ td:first-child { @media only screen and (max-width: 985px) { #copyright-block{display:block;} #copyright-notice{margin-left:0; padding-left:0;} -} \ No newline at end of file +} diff --git a/docs/missing-data.md b/docs/missing-data.md index eb97f73..e521a4f 100644 --- a/docs/missing-data.md +++ b/docs/missing-data.md @@ -1,185 +1,379 @@ --- title: "Handling missing data" layout: guidance-page +editor_options: + markdown: + wrap: 72 --- - > ## Main messages -> - Try to understand the causes of missing data and minimise the chance of it happening. -> - Assess input data for missing values to understand which variables and records are affected, how much is missing and whether there might be bias in the missingness. -> - Decide how to deal with missing data: update, discard or replace it. -> - Tell your users about the prevalence and treatment of missing data. +> +> - Try to understand the causes of missing data and minimise the +> chance of it happening. +> - Assess input data for missing values to understand which variables +> and records are affected, how much is missing and whether there +> might be bias in the missingness. +> - Decide how to deal with missing data: update, discard or replace +> it. +> - Tell your users about the prevalence and treatment of missing +> data. {% include sources-text.md %} ## Assessing missingness before analysis begins -Data is missing when no value is provided for a given variable. This is common for many datasets and missing values can reduce statistical power and introduce bias if not handled appropriately. +Data is missing when no value is provided for a given variable. This is +common for many datasets and missing values can reduce statistical power +and introduce bias if not handled appropriately. -You should try to understand the extent, causes and type of missingness as soon as you receive your input data. This knowledge will help inform how you handle the data, analyse it and communicate your results. The earlier an issue can be recognised, the quicker a plan can be developed for how to address it. +You should try to understand the extent, causes and type of missingness +as soon as you receive your input data. This knowledge will help inform +how you handle the data, analyse it and communicate your results. The +earlier an issue can be recognised, the quicker a plan can be developed +for how to address it. ### Check for presence and extent of missingness -You should inspect for missingness each time you receive a dataset, whether it's part of a routine or one-off data collection. -This could be as simple as counting or visualising the missing values in your dataset to understand their volume and location. Most software records missing data as a blank value or as ‘NA’ (not available), but check with your data providers in case they have used other special symbols. Pay particular attention to unexpected patterns, which may expose new underlying data quality issues. +You should inspect for missingness each time you receive a dataset, +whether it's part of a routine or one-off data collection. -Input validation tests (‘assertions’) are also encouraged early in your analytical pipelines so that you can 'sense check' your data. These tests should be automated when you need to re-run the same checks frequently, such as for routine data collections, for greater efficiency (according to principles of [Reproducible Analytical Pipelines (RAP)](rap)). +This could be as simple as counting or visualising the missing values in +your dataset to understand their volume and location. Most software +records missing data as a blank value or as ‘NA’ (not available), but +check with your data providers in case they have used other special +symbols. Pay particular attention to unexpected patterns, which may +expose new underlying data quality issues. -You should be aware of whose job it is to check for missing data. The responsible analyst has ownership of missing data and that the assurer must ensure that checks have taken place. You can refer to our [quality assurance guidance](quality-assurance) for further information. +Input validation tests (‘assertions’) are also encouraged early in your +analytical pipelines so that you can 'sense check' your data. These +tests should be automated when you need to re-run the same checks +frequently, such as for routine data collections, for greater efficiency +(according to principles of [Reproducible Analytical Pipelines +(RAP)](rap)). +You should be aware of whose job it is to check for missing data. The +responsible analyst has ownership of missing data and that the assurer +must ensure that checks have taken place. You can refer to our [quality +assurance guidance](quality-assurance) for further information. ### Understand causes of missingness -The presence of missing data may indicate problems with the way the data has been collected or processed. Understanding the causes can help you to fix the underlying problem and allow you to be more transparent with users of your data. -Some examples include: +The presence of missing data may indicate problems with the way the data +has been collected or processed. Understanding the causes can help you +to fix the underlying problem and allow you to be more transparent with +users of your data. -- measurement error, which could be caused by using defective tools for data collection; for example, a faulty machine may fail to take a reading from a patient’s sample -- handling errors, which can be caused by administrative lapses during data entry; for example, someone might fail to transcribe a handwritten value into a recording system -- data processing errors, which can be introduced when handling or analysing data; for example, naively coercing a text string to numeric type in R will result in a ‘NA’ value (so '4%' would become 'NA' rather than 4) -- survey-related errors, including non-response and sampling biases; these can occur when participants fail to respond to a question and when a survey sample is unrepresentative of the target population -- data providers could not or did not provide data ahead of deadlines +Some examples include: +- measurement error, which could be caused by using defective tools + for data collection; for example, a faulty machine may fail to take + a reading from a patient’s sample +- handling errors, which can be caused by administrative lapses during + data entry; for example, someone might fail to transcribe a + handwritten value into a recording system +- data processing errors, which can be introduced when handling or + analysing data; for example, naively coercing a text string to + numeric type in R will result in a ‘NA’ value (so '4%' would become + 'NA' rather than 4) +- survey-related errors, including non-response and sampling biases; + these can occur when participants fail to respond to a question and + when a survey sample is unrepresentative of the target population +- data providers could not or did not provide data ahead of deadlines ### Understand the type of missingness -It is important to understand how missingness may be distributed through your dataset and whether it can be explained. This impacts the strategies you might take to replace missing values or analyse your data. - -Typically, [three main types of missing data are recognised](https://stefvanbuuren.name/fimd/sec-MCAR.html). Data for a given variable can be: - -{% capture card_content_1 %} -When missingness is independent of any variable. An example of this would be if a batch of blood samples from a fully randomised set of patients is dropped and broken. This is true randomness. -{% endcapture %} - -{% capture card_content_2 %} -When there’s a pattern to the missingness that can be explained by another variable. For example, you might have missing blood-sample data that can be explained by men being less likely than women to submit a sample. Values are still missing at random within each group, but one group has more missing values than the other. A better term is perhaps ‘missing conditionally at random’. -{% endcapture %} - -{% capture card_content_3 %} -When there’s a pattern of missingness that’s unrelated to any variable we know about. This might occur if blood samples became contaminated and unusable due to the failure of a certain manufacturer’s vials, a variable we didn’t collect information about. -{% endcapture %} - - -{% include cards-container-start.html %} - {% include card.html content=card_content_1 title="Missing completely at random (MCAR)" %} - {% include card.html content=card_content_2 title="Missing at random (MAR)" %} - {% include card.html content=card_content_3 title="Missing not at random (MNAR)" %} -{% include cards-container-end.html %} - - -It can be difficult in practice to identify the type of missing data. In particular, the difference between MCAR and MNAR hinges on a relationship that may or may not exist with variables you haven’t collected. MAR may be a good default position if you’ve designed your collection to cover variables that explain the missingness, but they may not be obvious and it may not be practical to do this. +It is important to understand how missingness may be distributed through +your dataset and whether it can be explained. This impacts the +strategies you might take to replace missing values or analyse your +data. + +Typically, [three main types of missing data are +recognised](https://stefvanbuuren.name/fimd/sec-MCAR.html). Data for a +given variable can be: + +{% capture card_content_1 %} When missingness is independent of any +variable. An example of this would be if a batch of blood samples from a +fully randomised set of patients is dropped and broken. This is true +randomness. {% endcapture %} + +{% capture card_content_2 %} When there’s a pattern to the missingness +that can be explained by another variable. For example, you might have +missing blood-sample data that can be explained by men being less likely +than women to submit a sample. Values are still missing at random within +each group, but one group has more missing values than the other. A +better term is perhaps ‘missing conditionally at random’. {% endcapture +%} + +{% capture card_content_3 %} When there’s a pattern of missingness +that’s unrelated to any variable we know about. This might occur if +blood samples became contaminated and unusable due to the failure of a +certain manufacturer’s vials, a variable we didn’t collect information +about. {% endcapture %} + +{% include cards-container-start.html %} {% include card.html +content=card_content_1 title="Missing completely at random (MCAR)" %} {% +include card.html content=card_content_2 title="Missing at random (MAR)" +%} {% include card.html content=card_content_3 title="Missing not at +random (MNAR)" %} {% include cards-container-end.html %} + +It can be difficult in practice to identify the type of missing data. In +particular, the difference between MCAR and MNAR hinges on a +relationship that may or may not exist with variables you haven’t +collected. MAR may be a good default position if you’ve designed your +collection to cover variables that explain the missingness, but they may +not be obvious and it may not be practical to do this. ## Handle missing data -According to the [Code of Practice for Statistics](https://code.statisticsauthority.gov.uk/the-code/), producers of statistical publications are required to 'identify and address' any 'limitations in data sources'. As such, you must decide how to handle missing data in order to maintain reliability and transparency of your outputs. This is a good principle to follow even if you are not producing a statistical publication. -This section describes three broad approaches for dealing with missing data: +According to the [Code of Practice for +Statistics](https://code.statisticsauthority.gov.uk/the-code/), +producers of statistical publications are required to 'identify and +address' any 'limitations in data sources'. As such, you must decide how +to handle missing data in order to maintain reliability and transparency +of your outputs. This is a good principle to follow even if you are not +producing a statistical publication. -1. Update the missing data. -1. Discard the missing data. -1. Generate replacement values. +This section describes three broad approaches for dealing with missing +data: +1. Update the missing data. +2. Discard the missing data. +3. Generate replacement values. ### Update the missing data -In the first instance, you should try to improve completion rates. Failing this, you may also be able to recover missing data by returning to the source of collection, by using data collected from another source, or by deductive reasoning on the basis of other variables. + +In the first instance, you should try to improve completion rates. +Failing this, you may also be able to recover missing data by returning +to the source of collection, by using data collected from another +source, or by deductive reasoning on the basis of other variables. #### Return to the source -It may be possible to go back to your data suppliers and recover data that was lost through administrative error, for example, or to request a re-collection of data for the cases where it’s missing. This may be difficult in practice, however, because of the effort required. (The [Code of Practice for Statistics](https://code.statisticsauthority.gov.uk/the-code/) also states that you must ‘reduce the burden on those providing their information, and on those involved in collecting, recording and supplying data’.) + +It may be possible to go back to your data suppliers and recover data +that was lost through administrative error, for example, or to request a +re-collection of data for the cases where it’s missing. This may be +difficult in practice, however, because of the effort required. (The +[Code of Practice for +Statistics](https://code.statisticsauthority.gov.uk/the-code/) also +states that you must ‘reduce the burden on those providing their +information, and on those involved in collecting, recording and +supplying data’.) #### Infer from other variables -You may be able to use other variables to deduce the true value of a missing data point. For example, if a user writes ‘Kent’ in a ‘county’ field, but doesn’t complete the ‘region’ field, it can be assumed that the region is ‘South East England’. Arguably the data isn’t really ‘missing’ because it’s derived from other fields. As a result, you should consider whether it’s worth collecting this kind of derived information in the first place. -Regardless, you should consider carefully whether your inference is appropriate. For example, you are likely to make an error if you infer gender or ethnicity from a person’s name. +You may be able to use other variables to deduce the true value of a +missing data point. For example, if a user writes ‘Kent’ in a ‘county’ +field, but doesn’t complete the ‘region’ field, it can be assumed that +the region is ‘South East England’. Arguably the data isn’t really +‘missing’ because it’s derived from other fields. As a result, you +should consider whether it’s worth collecting this kind of derived +information in the first place. + +Regardless, you should consider carefully whether your inference is +appropriate. For example, you are likely to make an error if you infer +gender or ethnicity from a person’s name. #### Match to another dataset -You might be able to fill gaps in your collection by matching to another dataset on some unique identifier, like patient NHS numbers. The dataset being joined might not be complete either, but the combination of sources could help to reduce missingness. If the dataset being joined has complete information for the variable of interest, you should consider whether it's worth collecting that data or whether you should prefer to join it from secondary sources. -Beware: the matched data could have its own biases. [The Goldacre Review](https://www.gov.uk/government/publications/better-broader-safer-using-health-data-for-research-and-analysis) cites an Electronic Health Record (EHR) example for which completeness of ethnicity data is improved by matching Hospital Episode Statistics (HES), though this only improves coverage for the population subset that is most ill. +You might be able to fill gaps in your collection by matching to another +dataset on some unique identifier, like patient NHS numbers. The dataset +being joined might not be complete either, but the combination of +sources could help to reduce missingness. If the dataset being joined +has complete information for the variable of interest, you should +consider whether it's worth collecting that data or whether you should +prefer to join it from secondary sources. + +Beware: the matched data could have its own biases. [The Goldacre +Review](https://www.gov.uk/government/publications/better-broader-safer-using-health-data-for-research-and-analysis) +cites an Electronic Health Record (EHR) example for which completeness +of ethnicity data is improved by matching Hospital Episode Statistics +(HES), though this only improves coverage for the population subset that +is most ill. ### Discard the missing data -You could analyse your dataset by ignoring missingness in some way. This may only be useful when your dataset is mostly complete or if you have reason to believe that your missing data is MCAR. + +You could analyse your dataset by ignoring missingness in some way. This +may only be useful when your dataset is mostly complete or if you have +reason to believe that your missing data is MCAR. For example, you could use only the: -- complete cases, which are the records that have a response value in every variable in the dataset (‘listwise deletion’) -- available cases, which are the non-missing values that remain within a given variable (a process known as ‘pairwise deletion’) +- complete cases, which are the records that have a response value in + every variable in the dataset (‘listwise deletion’) +- available cases, which are the non-missing values that remain within + a given variable (a process known as ‘pairwise deletion’) -Both approaches are straightforward and require no additional calculation. Of these, complete-case analysis may be preferred where there is little data missing across the whole dataset, since the available-case approach could result in different samples being used for each variable in your analysis. Both methods ignore whether there’s any bias in the missingness and could result in a lot of useful data being thrown away, particularly if you use only the complete cases. +Both approaches are straightforward and require no additional +calculation. Of these, complete-case analysis may be preferred where +there is little data missing across the whole dataset, since the +available-case approach could result in different samples being used for +each variable in your analysis. Both methods ignore whether there’s any +bias in the missingness and could result in a lot of useful data being +thrown away, particularly if you use only the complete cases. ### Generate replacement data -There are several methods for generating replacements for missing values. The method used will depend on the type and cause of your missingness, as well as how much of it is missing. This section is intended as a non-exhaustive and non-technical introduction to some possible approaches. + +There are several methods for generating replacements for missing +values. The method used will depend on the type and cause of your +missingness, as well as how much of it is missing. This section is +intended as a non-exhaustive and non-technical introduction to some +possible approaches. #### Single imputation -[Single imputation methods](https://www.researchgate.net/profile/Geert-Van-Der-Heijden/publication/6814409_Review_A_gentle_introduction_to_imputation_of_missing_values/links/6048b70a299bf1e0786bfbfe/Review-A-gentle-introduction-to-imputation-of-missing-values.pdf) are simple approaches that use non-missing values to generate replacements for missing values. -Average imputation is when you generate a replacement value from a simple average of the non-missing values within a variable. This is typically the mean for continuous variables and the mode (most common) value for categorical variables. Note that mean imputation will maintain the overall mean of a variable, but will reduce its variance. +[Single imputation +methods](https://www.researchgate.net/profile/Geert-Van-Der-Heijden/publication/6814409_Review_A_gentle_introduction_to_imputation_of_missing_values/links/6048b70a299bf1e0786bfbfe/Review-A-gentle-introduction-to-imputation-of-missing-values.pdf) +are simple approaches that use non-missing values to generate +replacements for missing values. -Regression imputation is when you predict missing values from a model involving other variables in the dataset. For example, height and weight may be correlated, so regression imputation could help you to infer missing values. This is preferential to average imputation because it takes account of the relationships between variables, but it can be sensitive to outliers or misspecification of the model. +Average imputation is when you generate a replacement value from a +simple average of the non-missing values within a variable. This is +typically the mean for continuous variables and the mode (most common) +value for categorical variables. Note that mean imputation will maintain +the overall mean of a variable, but will reduce its variance. -#### Interpolation -If you have data collected over time, you can use prior information to generate a replacement value. The simplest approach would be to reuse the last known value, known as 'Last Observation Carried Forward' (LOCF). This within-record context helps reduce bias but ignores the potential and magnitude of change. +Regression imputation is when you predict missing values from a model +involving other variables in the dataset. For example, height and weight +may be correlated, so regression imputation could help you to infer +missing values. This is preferential to average imputation because it +takes account of the relationships between variables, but it can be +sensitive to outliers or misspecification of the model. -Interpolation is typically used to predict a value for an unsampled point, but we can use it to replace missing values as well. For example, you could fit a model to the data over time and predict the missing value. The quality of this method is dependent on selecting an appropriate model for the data. It could be a simple linear model if appropriate, for example, or could build in seasonality. +#### Interpolation -#### Multiple imputation -[Multiple imputation](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2714692/) acknowledges that there’s uncertainty in generating one replacement value, like in single imputation. To overcome this, you can generate more than one prediction. +If you have data collected over time, you can use prior information to +generate a replacement value. The simplest approach would be to reuse +the last known value, known as 'Last Observation Carried Forward' +(LOCF). This within-record context helps reduce bias but ignores the +potential and magnitude of change. -The general process for multiple imputation involves two steps: +Interpolation is typically used to predict a value for an unsampled +point, but we can use it to replace missing values as well. For example, +you could fit a model to the data over time and predict the missing +value. The quality of this method is dependent on selecting an +appropriate model for the data. It could be a simple linear model if +appropriate, for example, or could build in seasonality. -1. Create many different ‘complete’ copies of the dataset where a statistical model has been used to generate plausible imputations. -2. Fit your model to each dataset copy and pool the results. +#### Multiple imputation -One specific technique is called [Multiple Imputation by Chained Equations](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3074241/) (MICE). It follows these steps: +[Multiple +imputation](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2714692/) +acknowledges that there’s uncertainty in generating one replacement +value, like in single imputation. To overcome this, you can generate +more than one prediction. -1. Perform simple imputation (for example, mean imputation) for all variables. These will be treated as ‘placeholders’ for now. -2. For one variable, remove the placeholders and regress against the other variables. Use the predictions to replace the missing values. -3. Repeat the steps above for the remaining variables in the dataset to complete one ‘cycle’. -4. Perform multiple cycles to help converge the imputations. +The general process for multiple imputation involves two steps: -Incorporating variance into the replacement values can result in a better estimate of ‘true’ values and therefore a less-biased outcome compared to single imputation. The method is more complicated, however, given the need for selecting and specifying an appropriate imputation model. It may also be more difficult to communicate this method to end-users. +1. Create many different ‘complete’ copies of the dataset where a + statistical model has been used to generate plausible imputations. +2. Fit your model to each dataset copy and pool the results. + +One specific technique is called [Multiple Imputation by Chained +Equations](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3074241/) +(MICE). It follows these steps: + +1. Perform simple imputation (for example, mean imputation) for all + variables. These will be treated as ‘placeholders’ for now. +2. For one variable, remove the placeholders and regress against the + other variables. Use the predictions to replace the missing values. +3. Repeat the steps above for the remaining variables in the dataset to + complete one ‘cycle’. +4. Perform multiple cycles to help converge the imputations. + +Incorporating variance into the replacement values can result in a +better estimate of ‘true’ values and therefore a less-biased outcome +compared to single imputation. The method is more complicated, however, +given the need for selecting and specifying an appropriate imputation +model. It may also be more difficult to communicate this method to +end-users. ## Communicate about missing data -Transparency is required when reporting on the prevalence and handling of missingness, statistical methods for imputation, and the limitations of the approaches taken. However, the level of detail should be proportionate to the complexity of methods used. -When you publish your report you should describe the characteristics of missing data and how it was handled. There are usually three places where you can do this: +Transparency is required when reporting on the prevalence and handling +of missingness, statistical methods for imputation, and the limitations +of the approaches taken. However, the level of detail should be +proportionate to the complexity of methods used. -1. A detailed breakdown in a dedicated Quality and Methodology Information (QMI) document published alongside your release. -1. A brief explanation within the text of the published report. -1. In the documentation for data releases, such as the notes and placeholder symbols of a published spreadsheet. +When you publish your report you should describe the characteristics of +missing data and how it was handled. There are usually three places +where you can do this: +1. A detailed breakdown in a dedicated Quality and Methodology + Information (QMI) document published alongside your release. +2. A brief explanation within the text of the published report. +3. In the documentation for data releases, such as the notes and + placeholder symbols of a published spreadsheet. ### 1. In the Quality and Methodology Information (QMI) report -We recommend that you create a separate QMI document to provide extra detail that caters to the needs of different users and provides full transparency about how the outputs were produced. -Examples of specific information to include are: +We recommend that you create a separate QMI document to provide extra +detail that caters to the needs of different users and provides full +transparency about how the outputs were produced. -- the volume of missing data by variable -- biases in the presence of missing data, for example if prevalence was greater for records with certain characteristics -- possible reasons for missing data, especially if the rate is different compared to prior or similar publications -- any statistical methods used to take account of missingness, such as imputation or removal -- steps you will take to maximise completeness in future, if possible +Examples of specific information to include are: +- the volume of missing data by variable +- biases in the presence of missing data, for example if prevalence + was greater for records with certain characteristics +- possible reasons for missing data, especially if the rate is + different compared to prior or similar publications +- any statistical methods used to take account of missingness, such as + imputation or removal +- steps you will take to maximise completeness in future, if possible ### 2. In written reports #### The main body text -The main publication should include brief information about data, quality, and methods, but only to the extent necessary to understand the data. The report should refer users to the QMI document for greater depth on missing data, where appropriate. - -If the incomplete data is because of a data provider failing to submit, explain at the start of the affected section the figures impacted and outline plans for when and how updates will be made once the data does become available. - -The [Government Data Quality Framework](https://www.gov.uk/government/publications/the-government-data-quality-framework/the-government-data-quality-framework-guidance) asks producers to communicate the trade-offs between timeliness of data and its completeness. Missing data may be a consequence of needing to publish within a given time frame, which is perhaps more likely in an urgent setting like a national health threat. In these cases, it's important to highlight that missing data may become available in future and so a ‘provisional’ label could be applied to data or analysis to make this clear to users. Information about the impact of late-arriving data can also be expressed in a separate QMI report published alongside the main report. +The main publication should include brief information about data, +quality, and methods, but only to the extent necessary to understand the +data. The report should refer users to the QMI document for greater +depth on missing data, where appropriate. + +If the incomplete data is because of a data provider failing to submit, +explain at the start of the affected section the figures impacted and +outline plans for when and how updates will be made once the data does +become available. + +The [Government Data Quality +Framework](https://www.gov.uk/government/publications/the-government-data-quality-framework/the-government-data-quality-framework-guidance) +asks producers to communicate the trade-offs between timeliness of data +and its completeness. Missing data may be a consequence of needing to +publish within a given time frame, which is perhaps more likely in an +urgent setting like a national health threat. In these cases, it's +important to highlight that missing data may become available in future +and so a ‘provisional’ label could be applied to data or analysis to +make this clear to users. Information about the impact of late-arriving +data can also be expressed in a separate QMI report published alongside +the main report. #### Strategies for tables and charts -You should represent missing data in data summaries like tables and charts so that the user is aware that data is incomplete. This will also ensure that missingness is acknowledged if the table or chart are used elsewhere in isolation and out of context. -Specifically for tables, you must use placeholder symbols that highlight when a value is missing. [The Government Analysis Function guidance on symbols](https://analysisfunction.civilservice.gov.uk/policy-store/symbols-in-tables-definitions-and-help/) suggests using '[b]' for a break in time series, or '[u]' if the value is of low reliability, which could happen when summarising a variable that contains a lot of missing data. To give an example: if data wasn't collected in 2020 due to the coronavirus pandemic, a placeholder symbol '[b]' should be given in a table to highlight the break to users. +You should represent missing data in data summaries like tables and +charts so that the user is aware that data is incomplete. This will also +ensure that missingness is acknowledged if the table or chart are used +elsewhere in isolation and out of context. -A placeholder symbol allows you to provide information about why a data point is missing, which is not apparent if you leave the cell blank. A blank cell may also lead to accessibility issues for users of assistive technology. +Specifically for tables, you must use placeholder symbols that highlight +when a value is missing. [The Government Analysis Function guidance on +symbols](https://analysisfunction.civilservice.gov.uk/policy-store/symbols-in-tables-definitions-and-help/) +suggests using '[b]' for a break in time series, or '[u]' if the value +is of low reliability, which could happen when summarising a variable +that contains a lot of missing data. To give an example: if data wasn't +collected in 2020 due to the coronavirus pandemic, a placeholder symbol +'[b]' should be given in a table to highlight the break to users. -Where data is missing, you should include a note under each affected table or chart to explain how it has been impacted by the missing data and explain why. +A placeholder symbol allows you to provide information about why a data +point is missing, which is not apparent if you leave the cell blank. A +blank cell may also lead to accessibility issues for users of assistive +technology. +Where data is missing, you should include a note under each affected +table or chart to explain how it has been impacted by the missing data +and explain why. -{% capture card_content_4 %} -Example 1: table of fictional annual values, England, 2018 to 2023. +{% capture card_content_4 %} Example 1: table of fictional annual +values, England, 2018 to 2023. | Year | Value | |------|-------| @@ -189,10 +383,11 @@ Example 1: table of fictional annual values, England, 2018 to 2023. | 2021 | 4 | | 2022 | 3 | | 2023 | 5 | + {% endcapture %} -{% capture card_content_5 %} -Example 2: table of fictional annual values, England, 2018 to 2023. +{% capture card_content_5 %} Example 2: table of fictional annual +values, England, 2018 to 2023. | Year | Value | |------|-------| @@ -203,19 +398,26 @@ Example 2: table of fictional annual values, England, 2018 to 2023. | 2022 | 3 | | 2023 | 5 | -Note: The shorthand '[b]' is used to denote where there was a break in the time series due to the coronavirus (COVID-19) pandemic. -{% endcapture %} - -{% include cards-container-start.html %} - {% include card.html content=card_content_4 title="❌ To be avoided: leaving a blank space in a table where data is missing." %} - {% include card.html content=card_content_5 title="✔️ Best practice: use a placeholder symbol to indicate that data is missing and include a note beneath the table." %} -{% include cards-container-end.html %} +Note: The shorthand '[b]' is used to denote where there was a break in +the time series due to the coronavirus (COVID-19) pandemic. {% +endcapture %} +{% include cards-container-start.html %} {% include card.html +content=card_content_4 title="❌ To be avoided: leaving a blank space in +a table where data is missing." %} {% include card.html +content=card_content_5 title="✔️ Best practice: use a placeholder symbol +to indicate that data is missing and include a note beneath the table." +%} {% include cards-container-end.html %} -Where a value has been estimated or imputed, official guidance suggests adding '[e]' for estimated. As an example, let's say that we interpolated the value missing in 2020. When you present this data in a table, you should add the placeholder symbol '[e]' in a separate column, in this case for the year 2020, and include a short text note underneath the table explaining the shorthand. Avoid adding more than one value or piece of information to a single cell, as this can be difficult to interpret for users of assistive technology. +Where a value has been estimated or imputed, official guidance suggests +adding '[e]' for estimated. As an example, let's say that we +interpolated the value missing in 2020. When using a presentation table, +e.g. in a report, you can place shorthand in cells alongside a data +point. In this case for the year 2020, we have shown the shorthand '[e]' +and included a short note underneath the table explaining the shorthand. -{% capture card_content_6 %} -Example 3: table of fictional annual values, England, 2018 to 2023. +{% capture card_content_6 %} Example 3: table of fictional annual +values, England, 2018 to 2023. | Year | Value | |------|-------| @@ -225,58 +427,76 @@ Example 3: table of fictional annual values, England, 2018 to 2023. | 2021 | 4 | | 2022 | 3 | | 2023 | 5 | -{% endcapture %} -{% capture card_content_7 %} -Example 4: table of fictional annual values, England, 2018 to 2023. - -| Year | Value | Note | -|------|-------|------| -| 2018 | 2 | | -| 2019 | 3 | | -| 2020 | 4 | [e] | -| 2021 | 4 | | -| 2022 | 3 | | -| 2023 | 5 | | - -Note: The shorthand '[e]' is used to indicate that the value in 2020 has been estimated. {% endcapture %} -{% include cards-container-start.html %} - {% include card.html content=card_content_6 title="❌ To be avoided: leaving a blank space in a table where data is missing." %} - {% include card.html content=card_content_7 title="✔️ Best practice: use a placeholder symbol to indicate that data has been estimated and include a note beneath the table." %} -{% include cards-container-end.html %} - - -You should also highlight discontinuities in time series data when presenting charts, for example by leaving a gap between points in a line chart to show where data is missing between them. You could also use dashed lines or shade the chart background to highlight where data is missing, and add a short text note underneath to explain the reason. - - -{% capture card_content_8 %} -Example 5: line chart of fictional annual values, England, 2018 to 2023. - -A line chart which does not clearly show that there is a missing value - -This figure shows a line chart with a discontinuity. It joins two data points in 2019 and 2021, despite a missing data point in 2020. +{% capture card_content_7 %} Example 4: table of fictional annual +values, England, 2018 to 2023. + +| Year | Value | +|----------|-------| +| 2018 | 2 | +| 2019 | 3 | +| 2020 [e] | 4 | +| 2021 | 4 | +| 2022 | 3 | +| 2023 | 5 | + +Note: The shorthand '[e]' is used to indicate that the value in 2020 has +been estimated. {% endcapture %} + +If you were presenting this data in a spreadsheet, you should add the +placeholder symbol '[e]' in a separate note column. When working in +spreadsheets, always avoid adding more than one value or piece of +information to a single cell, as this can be difficult to interpret for +users of assistive technology. For example, if you want to sum a column +but one of the data points has a shorthand marker next to it, the data +point will be ignored. + +{% include cards-container-start.html %} {% include card.html +content=card_content_6 title="❌ To be avoided: leaving a blank space in +a table where data is missing." %} {% include card.html +content=card_content_7 title="✔️ Best practice: use a placeholder symbol +to indicate that data has been estimated and include a note beneath the +table." %} {% include cards-container-end.html %} + +You should also highlight discontinuities in time series data when +presenting charts, for example by leaving a gap between points in a line +chart to show where data is missing between them. You could also use +dashed lines or shade the chart background to highlight where data is +missing, and add a short text note underneath to explain the reason. + +{% capture card_content_8 %} Example 5: line chart of fictional annual +values, England, 2018 to 2023. + +A line chart which does not clearly show that there is a missing value + +This figure shows a line chart with a discontinuity. It joins two data +points in 2019 and 2021, despite a missing data point in 2020. {% endcapture %} -{% capture card_content_9 %} -Example 6: line chart of fictional annual values, England, 2018 to 2023. - -A line chart which clearly shows a missing value by leaving a gap in the line, marking out the gap with dashed lines and an annotation +{% capture card_content_9 %} Example 6: line chart of fictional annual +values, England, 2018 to 2023. -This figure shows a line chart with a discontinuity. It leaves a gap in the line and marks out the gap with two vertical dashed lines and annotation stating the data was not collected in 2020 due to the coronavirus (COVID-19) pandemic. -{% endcapture %} +A line chart which clearly shows a missing value by leaving a gap in the line, marking out the gap with dashed lines and an annotation -{% include cards-container-start.html %} - {% include card.html content=card_content_8 title="❌ To be avoided: ignoring missing data in a line chart by joining the next nearest available data points." %} - {% include card.html content=card_content_9 title="✔️ Best practice: leave a gap in a line chart where there is missing data and highlight it with dashed lines and a text explanation." %} -{% include cards-container-end.html %} +This figure shows a line chart with a discontinuity. It leaves a gap in +the line and marks out the gap with two vertical dashed lines and +annotation stating the data was not collected in 2020 due to the +coronavirus (COVID-19) pandemic. {% endcapture %} +{% include cards-container-start.html %} {% include card.html +content=card_content_8 title="❌ To be avoided: ignoring missing data in +a line chart by joining the next nearest available data points." %} {% +include card.html content=card_content_9 title="✔️ Best practice: leave +a gap in a line chart where there is missing data and highlight it with +dashed lines and a text explanation." %} {% include +cards-container-end.html %} {% capture expandable_content_1 %} -``` +``` # Create data frame: df <- data.frame(year = seq(2018, 2023, 1), value = c(2, 3, NA, 4, 3, 5)) @@ -307,39 +527,78 @@ df |> y = 2, size = 18 * 5/14) ``` -{% endcapture %} -{% include expandable-block-start.html %} - {% include expandable-section.html number="1" content=expandable_content_1 title="Example 6 R code" %} -{% include expandable-block-end.html %} +{% endcapture %} +{% include expandable-block-start.html %} {% include +expandable-section.html number="1" content=expandable_content_1 +title="Example 6 R code" %} {% include expandable-block-end.html %} -You can read more about best practice for data visualisation in our [data visualisation guidance pages](data-vis). +You can read more about best practice for data visualisation in our +[data visualisation guidance pages](data-vis). ### 3. In data files -It's possible that some users may use data releases without referring to the main report or QMI document. It's important that supporting data files have sufficient information to provide context for the presence and handling of missing data, where appropriate. + +It's possible that some users may use data releases without referring to +the main report or QMI document. It's important that supporting data +files have sufficient information to provide context for the presence +and handling of missing data, where appropriate. #### In spreadsheets -You should provide notes and a note sheet in your published spreadsheets, which allow for short written explanations about issues like missing data and how it was prepared or analysed. You should also use symbolic shorthand to explain why data is missing from certain cells. For example, '[b]' for a break in a time series, '[e]' for estimated data and '[z]' to indicate that a variable was ‘not applicable’ to a particular record. These symbols are preferable to using blank cells, NA or '..' from the perspective of clarity and accessibility. -You can read more in our [guidance on accessible reference tables in spreadsheets](spreadsheets). +You should provide notes and a note sheet in your published +spreadsheets, which allow for short written explanations about issues +like missing data and how it was prepared or analysed. You should also +use symbolic shorthand to explain why data is missing from certain +cells. For example, '[b]' for a break in a time series, '[e]' for +estimated data and '[z]' to indicate that a variable was ‘not +applicable’ to a particular record. These symbols are preferable to +using blank cells, NA or '..' from the perspective of clarity and +accessibility. + +You can read more in our [guidance on accessible reference tables in +spreadsheets](spreadsheets). #### In machine-readable data files -It can be difficult to provide contextual information for certain types of data files, like machine-readable comma-separated Values (CSV) files. In this case, you should provide to users a data dictionary to explain how you’ve represented missing data. Like spreadsheets, you should consider using symbolic placeholders where data is missing. -You might also consider using the comma-separated Values on the Web (CSVW) format, which allows you to supply metadata information alongside the data. With CSVW you can set ‘null’ and ‘default’ properties in the metadata that could be used to indicate missing values. You can also use the special ‘required’ property as a validation check if you were expecting a field to be complete. +It can be difficult to provide contextual information for certain types +of data files, like machine-readable comma-separated Values (CSV) files. +In this case, you should provide to users a data dictionary to explain +how you’ve represented missing data. Like spreadsheets, you should +consider using symbolic placeholders where data is missing. +You might also consider using the comma-separated Values on the Web +(CSVW) format, which allows you to supply metadata information alongside +the data. With CSVW you can set ‘null’ and ‘default’ properties in the +metadata that could be used to indicate missing values. You can also use +the special ‘required’ property as a validation check if you were +expecting a field to be complete. ## Sources -1. [Government Analysis Function: Communicating quality, uncertainty and change](https://analysisfunction.civilservice.gov.uk/policy-store/communicating-quality-uncertainty-and-change/) -1. [Government Analysis Function: Data visualisation: tables](https://analysisfunction.civilservice.gov.uk/policy-store/data-visualisation-tables/) -1. [Government Analysis Function: Using symbols and shorthand](https://analysisfunction.civilservice.gov.uk/policy-store/symbols-in-tables-definitions-and-help/) -1. [Government Analysis Function: Data visualisation: charts](https://analysisfunction.civilservice.gov.uk/policy-store/data-visualisation-charts/) -1. [Government Analysis Function: Making spreadsheets accessible: a checklist of the basics](https://analysisfunction.civilservice.gov.uk/policy-store/making-spreadsheets-accessible-a-brief-checklist-of-the-basics/) -1. [Government Data Quality Hub: The Government Data Quality Framework](https://www.gov.uk/government/publications/the-government-data-quality-framework/the-government-data-quality-framework) -1. [Government Data Quality Hub: The Government Data Quality Framework -guidance](https://www.gov.uk/government/publications/the-government-data-quality-framework/the-government-data-quality-framework-guidance) -1. [Office for National Statistics: The Duck Book - Quality assurance of code for analysis and research](https://best-practice-and-impact.github.io/qa-of-code-guidance/) -1. [The Aqua Book: guidance on producing quality analysis for government](https://www.gov.uk/government/publications/the-aqua-book-guidance-on-producing-quality-analysis-for-government) -1. [The Goldacre Review: Better, broader, safer: using health data for research and analysis](https://www.gov.uk/government/publications/better-broader-safer-using-health-data-for-research-and-analysis) -1. [UK Statistics Authority: Code of Practice for Statistics](https://code.statisticsauthority.gov.uk/the-code/) +1. [Government Analysis Function: Communicating quality, uncertainty + and + change](https://analysisfunction.civilservice.gov.uk/policy-store/communicating-quality-uncertainty-and-change/) +2. [Government Analysis Function: Data visualisation: + tables](https://analysisfunction.civilservice.gov.uk/policy-store/data-visualisation-tables/) +3. [Government Analysis Function: Using symbols and + shorthand](https://analysisfunction.civilservice.gov.uk/policy-store/symbols-in-tables-definitions-and-help/) +4. [Government Analysis Function: Data visualisation: + charts](https://analysisfunction.civilservice.gov.uk/policy-store/data-visualisation-charts/) +5. [Government Analysis Function: Making spreadsheets accessible: a + checklist of the + basics](https://analysisfunction.civilservice.gov.uk/policy-store/making-spreadsheets-accessible-a-brief-checklist-of-the-basics/) +6. [Government Data Quality Hub: The Government Data Quality + Framework](https://www.gov.uk/government/publications/the-government-data-quality-framework/the-government-data-quality-framework) +7. [Government Data Quality Hub: The Government Data Quality Framework + -guidance](https://www.gov.uk/government/publications/the-government-data-quality-framework/the-government-data-quality-framework-guidance) +8. [Office for National Statistics: The Duck Book - Quality assurance + of code for analysis and + research](https://best-practice-and-impact.github.io/qa-of-code-guidance/) +9. [The Aqua Book: guidance on producing quality analysis for + government](https://www.gov.uk/government/publications/the-aqua-book-guidance-on-producing-quality-analysis-for-government) +10. [The Goldacre Review: Better, broader, safer: using health data for + research and + analysis](https://www.gov.uk/government/publications/better-broader-safer-using-health-data-for-research-and-analysis) +11. [UK Statistics Authority: Code of Practice for + Statistics](https://code.statisticsauthority.gov.uk/the-code/) diff --git a/docs/quality-assurance.md b/docs/quality-assurance.md index b5e1082..d43864a 100644 --- a/docs/quality-assurance.md +++ b/docs/quality-assurance.md @@ -8,8 +8,8 @@ layout: guidance-page >- Quality assurance should be planned in advance, with roles, responsibilities, and the extent of assurance activities understood fully by the entire team. >- Quality assurance should be appropriate and proportionate to the scope, risks, methodology, and data source of a piece of analysis. >- Understanding user needs is important when measuring the quality of your data. Perfect data quality may not always be achievable and therefore focus should be given to ensuring the data is as fit for purpose as it can be. ->- Teams should hold discussions to review quality and plan quality assurance on a semi-regular basis, identifying gaps in current procedures (our [quality review conversation tool](https://github.com/BenCuff/github-pages-test/blob/main/docs/assets/downloads/QA%20review%20conversation%20tool.xlsx) can help facilitate this). ->- A quality assurance log should be created and maintained (see our [QA log template](https://github.com/BenCuff/github-pages-test/blob/main/docs/assets/downloads/QA%20log%20template.xlsx)). This is a crucial tool to help teams plan, record, and sign off on quality; it provides an audit trail of everything that was checked and what the outcomes were. +>- Teams should hold discussions to review quality and plan quality assurance on a semi-regular basis, identifying gaps in current procedures (our [quality review conversation tool](https://github.com/ukhsa-collaboration/statistics-production-hub/raw/main/docs/assets/downloads/QA%20review%20conversation%20tool.xlsx) can help facilitate this). +>- A quality assurance log should be created and maintained (see our [QA log template](https://github.com/ukhsa-collaboration/statistics-production-hub/raw/main/docs/assets/downloads/QA%20log%20template.xlsx)). This is a crucial tool to help teams plan, record, and sign off on quality; it provides an audit trail of everything that was checked and what the outcomes were. >- The quality assurance process and any outstanding quality concerns should be clearly communicated alongside the final results of the analysis. @@ -27,7 +27,7 @@ The quality of an analytical output may be thought of in simple terms as its “ 4. Comparability and Coherence 5. Accessibility and Clarity -All quality assessments of official statistics should be conducted in line with these dimensions and both our [QA review conversation tool](https://github.com/BenCuff/github-pages-test/blob/main/docs/assets/downloads/QA%20review%20conversation%20tool.xlsx) and [QA log template](https://github.com/BenCuff/github-pages-test/blob/main/docs/assets/downloads/QA%20log%20template.xlsx) sets out these dimensions for you to consider. +All quality assessments of official statistics should be conducted in line with these dimensions and both our [QA review conversation tool](https://github.com/ukhsa-collaboration/statistics-production-hub/raw/main/docs/assets/downloads/QA%20review%20conversation%20tool.xlsx) and [QA log template](https://github.com/ukhsa-collaboration/statistics-production-hub/raw/main/docs/assets/downloads/QA%20log%20template.xlsx) sets out these dimensions for you to consider. See inside the expandable sections below for example questions you might consider as part of each dimension. @@ -136,7 +136,7 @@ It is our role as analysts to explain how any limitations in our outputs might i -These key stages along the data journey, alongside the 5 Dimensions of Quality of the European Statistical System (ESS), form the basis of our [QA review conversation tool](https://github.com/BenCuff/github-pages-test/blob/main/docs/assets/downloads/QA%20review%20conversation%20tool.xlsx) and [QA log template](https://github.com/BenCuff/github-pages-test/blob/main/docs/assets/downloads/QA%20log%20template.xlsx). +These key stages along the data journey, alongside the 5 Dimensions of Quality of the European Statistical System (ESS), form the basis of our [QA review conversation tool](https://github.com/ukhsa-collaboration/statistics-production-hub/raw/main/docs/assets/downloads/QA%20review%20conversation%20tool.xlsx) and [QA log template](https://github.com/ukhsa-collaboration/statistics-production-hub/raw/main/docs/assets/downloads/QA%20log%20template.xlsx). ## Roles and responsibilities @@ -204,18 +204,18 @@ If the work is being used for a new purpose, the analytical assurer should obtai 2. QA log template
- -
### What is the QA review conversation tool? -Our [QA review conversation tool](https://github.com/BenCuff/github-pages-test/blob/main/docs/assets/downloads/QA%20review%20conversation%20tool.xlsx) was created to provide a useful starting point for thinking about the QA of statistical outputs in line with the five European Statistical System (ESS) Quality Dimensions. It has been designed to facilitate team discussions about quality at either the start of a new project or at regular intervals for cyclical releases to help teams plan QA, identifying gaps in current procedures. We have compiled discussion questions that teams can use to reflect on the quality of their analysis, identify areas for improvement, and consider how to communicate quality to users. The outcomes of these discussions should inform QA plans and QA logs which teams should be using during QA activities. +Our [QA review conversation tool](https://github.com/ukhsa-collaboration/statistics-production-hub/raw/main/docs/assets/downloads/QA%20review%20conversation%20tool.xlsx) was created to provide a useful starting point for thinking about the QA of statistical outputs in line with the five European Statistical System (ESS) Quality Dimensions. It has been designed to facilitate team discussions about quality at either the start of a new project or at regular intervals for cyclical releases to help teams plan QA, identifying gaps in current procedures. We have compiled discussion questions that teams can use to reflect on the quality of their analysis, identify areas for improvement, and consider how to communicate quality to users. The outcomes of these discussions should inform QA plans and QA logs which teams should be using during QA activities. ### What is the QA log template? -To help teams record and sign off on quality, we have also created a [QA log template](https://github.com/BenCuff/github-pages-test/blob/main/docs/assets/downloads/QA%20log%20template.xlsx). This approach is standard practice across government statistics and should ensure that a QA mindset is present when conducting analytical projects. A QA log should be used every time you are producing a publication. +To help teams record and sign off on quality, we have also created a [QA log template](https://github.com/ukhsa-collaboration/statistics-production-hub/raw/main/docs/assets/downloads/QA%20log%20template.xlsx). This approach is standard practice across government statistics and should ensure that a QA mindset is present when conducting analytical projects. A QA log should be used every time you are producing a publication. A QA log will: @@ -248,9 +248,9 @@ The table below gives some considerations when completing this section of the re | Guiding questions | Specific considerations | Why do I need to know the answer to this? | What help is available? | |-------------------|-------------------------|-------------------------------------------|-------------------------| -| What is the need for this analysis or statistical release? | Does the output contribute something new that cannot be found in existing research or literature?

Has there been a clear user need for this analysis shown through user consultation? | Understanding why the analysis is needed and what it will be used for is critical for understanding whether what you have done is fit for purpose.

If you are responsible for part of an analytical process, understanding the end use will help you to make sure that your part of the output does what is needed to meet user needs. | [Guidance: The Aqua Book (HM Treasury)](https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/416478/aqua_book_final_web.pdf)


[Guidance: Analysis Functional Standard (Government Analysis Function)](https://www.gov.uk/government/publications/government-analysis-functional-standard--2) | +| What is the need for this analysis or statistical release? | Does the output contribute something new that cannot be found in existing research or literature?

Has there been a clear user need for this analysis shown through user consultation? | Understanding why the analysis is needed and what it will be used for is critical for understanding whether what you have done is fit for purpose.

If you are responsible for part of an analytical process, understanding the end use will help you to make sure that your part of the output does what is needed to meet user needs. | [Guidance: The Aqua Book (HM Treasury)](https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/416478/aqua_book_final_web.pdf)

[Guidance: Analysis Functional Standard (Government Analysis Function)](https://www.gov.uk/government/publications/government-analysis-functional-standard--2) | | Who uses your analysis or statistical release? | Does the work meet user needs appropriately?

Have you addressed feedback obtained from recent user consultations?

What are the expectations of my users?

Are the length and content of the publication appropriate for its users?

Are terms defined for your less-technical users? | Understanding who uses your analytical output will help you to make sure that it meets their needs.

It also helps you to tailor your outputs to make sure all your users are fully supported in using the outputs effectively. | [Guidance: User engagement top tips (Government Analysis Function)](https://analysisfunction.civilservice.gov.uk/policy-store/user-engagement-top-tips/) | - | What analytical question you are addressing? | Are questions suitably explicit and do they clearly reflect the evidence gaps the analysis is intended to fill?

Will the output align with overarching departmental or programme goals? | Having a clear understanding of the problem your team is trying to solve ensures that the analysis you design is fit for purpose.

If you do not know how your work is contributing to answering an analytical need, you may be unaware of important requirements or limitations for your part of the work. | [Guidance: The Aqua Book (HM Treasury) ](https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/416478/aqua_book_final_web.pdf)


[Guidance: Analysis Functional Standard (Government Analysis Function)](https://www.gov.uk/government/publications/government-analysis-functional-standard--2)


[UKHSA Data Strategy (UK Health Security Agency)](https://intranet.ukhsa.gov.uk/sites/data-analytics-and-surveillance/SitePageModern/49668/data-strategy) | + | What analytical question you are addressing? | Are questions suitably explicit and do they clearly reflect the evidence gaps the analysis is intended to fill?

Will the output align with overarching departmental or programme goals? | Having a clear understanding of the problem your team is trying to solve ensures that the analysis you design is fit for purpose.

If you do not know how your work is contributing to answering an analytical need, you may be unaware of important requirements or limitations for your part of the work. | [Guidance: The Aqua Book (HM Treasury) ](https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/416478/aqua_book_final_web.pdf)

[Guidance: Analysis Functional Standard (Government Analysis Function)](https://www.gov.uk/government/publications/government-analysis-functional-standard--2)

[UKHSA Data Strategy (UK Health Security Agency)](https://intranet.ukhsa.gov.uk/sites/data-analytics-and-surveillance/SitePageModern/49668/data-strategy) | | Is everyone aware of their role and responsibility in the process? | Is a Senior Responsible Officer (SRO) required as the analysis is business critical?

What is the role of the commissioner in your project? Are there any key changes to the publications that need flagging?

Does the assurer understand the analytical methods used in the project? Are there any skills gaps? | For effective quality assurance, you need to be clear on roles and responsibilities throughout the chain of production. It is not enough to say that an individual will carry out some QA.

Assigning the roles set out in the AQUA book provide a useful framework for considering the different ways QA should be built into the life cycle of a project.

For more information, see Roles and responsibilities section of this guidance. | [Guidance: The Aqua Book (HM Treasury)](https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/416478/aqua_book_final_web.pdf) {% endcapture %} @@ -266,7 +266,7 @@ The table below gives some considerations when completing this section of the re |Guiding questions | Specific considerations | Why do I need to know the answer to this? | What help is available? | |------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | Is all of your documentation about the project easy to find? | Is there a single location (for example, on GitHub or SharePoint) where all project documentation is either stored or linked?

Have you got a link to your project code repository within the documentation? | Your analysis must be well documented so that somebody new can understand it and pick it up. Poor documentation means that other people will not understand why the process is configured as it is, how the process works or how to run the process safely, potentially leading to errors. | [Guidance: The Aqua Book (HM Treasury)](https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/416478/aqua_book_final_web.pdf)

[Guidance: QA of Code for Analysis and Research (Government Analysis Function)](https://best-practice-and-impact.github.io/qa-of-code-guidance/intro.html) | -| Have you got a consistent project structure set up? | Does the project structure align with other analytical projects in your organisation?

Is there a clear system to track version control within your project? | All projects should have appropriate version control including a clear system for version labelling and a version control log.

It is recommended to use a project template such as the [UKHSA Project Template](https://confluence.collab.test-and-trace.nhs.uk/display/QA/UKHSA+Project+Template). A template provides a consistent project structure and some built-in automated QA. | [UKHSA Project Template (GitHub link)](https://confluence.collab.test-and-trace.nhs.uk/display/QA/UKHSA+Project+Template)

[Guidance: GitHub docs (GitHub)](https://docs.github.com/en/get-started/start-your-journey/hello-world) | +| Have you got a consistent project structure set up? | Does the project structure align with other analytical projects in your organisation?

Is there a clear system to track version control within your project? | All projects should have appropriate version control including a clear system for version labelling and a version control log.

It is recommended to use a project template such as [govcookiecutter](https://github.com/best-practice-and-impact/govcookiecutter). A template provides a consistent project structure and some built-in automated QA. | [govcookiecutter (GitHub link)](https://github.com/best-practice-and-impact/govcookiecutter)

[Guidance: GitHub docs (GitHub)](https://docs.github.com/en/get-started/start-your-journey/hello-world) | | Is there a recorded plan for the analysis? | Has an appropriate plan been agreed with adequate consideration of time, resource quantity and skills required?

Will the proposed time frames allow for adequate quality assurance?

Where are the highest risk points for errors in the process? What measures do you or could you take to mitigate risk at these points? | Analysis will often involve a trade off between time, resource and quality.

Discussions about the desired and achievable levels of QA should take place at the very start of a project. The reality is that most analysis will be carried out under time and resource pressure and we will not be able to carry out all the QA activity we would ideally like to. In these circumstances, QA activities will need to be prioritised based on the risk of not carrying them out. | [Guidance: The Aqua Book (HM Treasury)](https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/416478/aqua_book_final_web.pdf)

[The Government Data Quality Framework (Government Data Quality Hub)](https://www.gov.uk/government/publications/the-government-data-quality-framework/the-government-data-quality-framework)

[Guidance: Tips for urgent quality assurance of data (Government Analysis Function)](https://analysisfunction.civilservice.gov.uk/policy-store/tips-for-urgent-quality-assurance-of-data) | | Can you summarise and explain the end-to-end process of your analysis? | How does your work feed in to the bigger analytical picture? | Having an overview of the analysis (especially if you only work on part of it) ensures that you and your team understand how your work feeds into the wider product.

It can help you to identify potential quality risks or issues, both upstream and downstream of your own work as well as how your activity supports and underpins downstream processing. | [Generic Statistical Business Process Model (GSBPM; United Nations Economic Commision for Europe)](https://unece.org/statistics/modernstats/gsbpm) @@ -284,7 +284,7 @@ The table below gives some considerations when completing this section of the re | Guiding questions | Specific considerations | Why do I need to know the answer to this? | What help is available? | |-----------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| Is the input data quality well-understood and sufficient for project needs? | Are the source, collection methods, size, accuracy, and precision of the data known? Have you checked for missing and duplicated data? Have you addressed these? Have the data been checked for any extreme or implausible values? Do any data quality concerns pose a risk for the final outcomes of the analysis? | The quality of input data directly affects the validity of any results or conclusions drawn from it. Understanding the data quality helps you assess the reliability and trustworthiness of findings. You should assess your datasets for missing values to understand which variables and records are affected, how much is missing and whether there might be bias in the missingness. Always tell your users about the prevalence and treatment of missing data. | [Guidance: Missing data (internal link)](missing-data) [Guidance: Quality statistics in government (Government Analysis Function)](https://analysisfunction.civilservice.gov.uk/policy-store/quality-statistics-in-government/) | +| Is the input data quality well-understood and sufficient for project needs? | Are the source, collection methods, size, accuracy, and precision of the data known? Have you checked for missing and duplicated data? Have you addressed these? Have the data been checked for any extreme or implausible values? Do any data quality concerns pose a risk for the final outcomes of the analysis? | The quality of input data directly affects the validity of any results or conclusions drawn from it. Understanding the data quality helps you assess the reliability and trustworthiness of findings. You should assess your datasets for missing values to understand which variables and records are affected, how much is missing and whether there might be bias in the missingness. Always tell your users about the prevalence and treatment of missing data. | [Guidance: Missing data (internal link)](missing-data)

[Guidance: Quality statistics in government (Government Analysis Function)](https://analysisfunction.civilservice.gov.uk/policy-store/quality-statistics-in-government/) | | How do you know the method you are using is appropriate? | Are the methods agreed to be the most suited for the project needs compared to other potential alternatives? Does the methodology deliver the required degree of certainty and precision? When changes have been made to methodologies, are they justified, well-documented, and clearly communicated? | You should be able to explain why the method(s) you use are suitable for this analysis and be able to support your choice with evidence. This might include reference to academic peer-reviewed publications or other projects that are similar. If you cannot explain why you chose the methods you use and why they are right for your analysis and the data you are using, you cannot be sure that your approach is sound. | | | Is the code or software functioning as expected? | Have an appropriate range of automated checks been built into the analytical pipeline? Has the code or other implementation of the analysis been checked by an independent analyst? Have intermediate outputs from every stage of analysis been checked? | You need to be sure that your analysis produces the outputs that you think it should and that the processes you run work as expected. If you cannot demonstrate that code and processes you have set up are functioning correctly, you cannot confirm the quality of the results. | [Guidance: RAP Framework - Bronze, Silver and Gold standards (internal link)](rap-framework) | | Do the outputs appear realistic? | Are there any anomalies or unexpected trends in the data? Have they been sufficiently investigated? Does the output appear realistic for the real-world behaviour that is being analysed? Are outputs consistent with results from historical data or other data from other regions? Have the methodology and outputs been peer reviewed to ensure they appear reasonable? | Stakeholders rely on analytical outputs to make informed decisions. You need to ensure that these decisions are based on accurate and plausible data, and so if the data looks different to previous trends this should be investigated and communicated clearly to users. | | @@ -302,10 +302,10 @@ The table below gives some considerations when completing this section of the re | Guiding questions | Specific considerations | Why do I need to know the answer to this? | What help is available? | |---------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| Has the code been properly documented? | Has version control software such as Git and GitHub been used to track and manage changes? Is code documentation embedded in the project rather than saved elsewhere? Have functions been documented and wrapped up in a package where useful to do so? | Your code must be well documented so that somebody new can understand it and pick it up. Poor documentation means that other people will not understand why the process is configured as it is, how the process works or how to run the process safely, potentially leading to errors. Good version control ensures that you have a full understanding of when, why and how changes were made to your analysis process. If it is hard to track changes, this makes it hard to retrace steps if there is a problem and means you do not fully understand the process. | [Guidance: GitHub docs (GitHub)](https://docs.github.com/en/get-started/start-your-journey/hello-world) [Guidance: Quality Assurance of Code for Analysis and Research (Government Analysis Function)](https://best-practice-and-impact.github.io/qa-of-code-guidance/intro.html) | -| Is the process from initial data extraction to final report fully reproducible? | Without any input from the analytical project team, could a third party reproduce our results? Have manual steps in data extraction, analysis, and report creation been automated where possible? If mathematical formulas have been used to describe methodology, have these been presented with all terms and steps defined? Where manual steps were involved, have these been thoroughly documented? Has a standard operating procedure (SOP) been produced if necessary? Have the documentation and quality assurance logs been peer reviewed to ensure that the process is reproducible? If we repeat our processes with different software, will we get the same results? | If the process from initial extraction to the final report contains manual steps, then these steps need to be documented so that the team can understand the process. Manual steps can lead to a higher risk of human error, so they need to be clearly understood. | [Guidance: Quality Assurance of Code for Analysis and Research (Government Analysis Function)](https://best-practice-and-impact.github.io/qa-of-code-guidance/intro.html) [Guidance: The Aqua Book (HM Treasury)](https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/416478/aqua_book_final_web.pdf) [Guidance: Reproducible Analytical Pipelines (RAP): an introduction (internal link)](rap) | -| Have best practices for coding been followed? | Has an agreed-upon style guide been followed in all of the code written for analysis? Does code adhere to best practice standards? Are there plans in place to ensure there is sufficient capability within the team to use and modify the pipeline as needed in the future? | If best practices have not been followed, there may be a higher risk of error. By understanding the team's agree-upon styles and standards, you can ensure consistency when modifying the pipeline. | [Guidance: Quality Assurance of Code for Analysis and Research (Government Analysis Function)](https://best-practice-and-impact.github.io/qa-of-code-guidance/intro.html) [Guidance: The Aqua Book (HM Treasury)](https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/416478/aqua_book_final_web.pdf) [Guidance: Quality Assurance of Code for Analysis and Research (Government Analysis Function)](https://best-practice-and-impact.github.io/qa-of-code-guidance/intro.html) | -| Do you consistently use peer review to check scripts and code, documentation, implementation of methods, processes and outputs? | Have the methodology and outputs been peer reviewed to ensure they appear reasonable? Is your code or analysis ever peer reviewed by someone outside your team? | Peer review is a standard part of analysis best practice. It is helpful because it helps to identify where steps are unclear, documents are hard to understand or there might be problems with calculations or implementation of methods. Routine peer review helps to improve the quality of processes and to reduce risk by identifying potential problems. | [Guidance: Quality Assurance of Code for Analysis and Research (Government Analysis Function)](https://best-practice-and-impact.github.io/qa-of-code-guidance/intro.html) [Guidance: Quality statistics in government (Government Analysis Function)](https://analysisfunction.civilservice.gov.uk/policy-store/quality-statistics-in-government/#quality-assurance) | +| Has the code been properly documented? | Has version control software such as Git and GitHub been used to track and manage changes? Is code documentation embedded in the project rather than saved elsewhere? Have functions been documented and wrapped up in a package where useful to do so? | Your code must be well documented so that somebody new can understand it and pick it up. Poor documentation means that other people will not understand why the process is configured as it is, how the process works or how to run the process safely, potentially leading to errors. Good version control ensures that you have a full understanding of when, why and how changes were made to your analysis process. If it is hard to track changes, this makes it hard to retrace steps if there is a problem and means you do not fully understand the process. | [Guidance: GitHub docs (GitHub)](https://docs.github.com/en/get-started/start-your-journey/hello-world)

[Guidance: Quality Assurance of Code for Analysis and Research (Government Analysis Function)](https://best-practice-and-impact.github.io/qa-of-code-guidance/intro.html) | +| Is the process from initial data extraction to final report fully reproducible? | Without any input from the analytical project team, could a third party reproduce our results? Have manual steps in data extraction, analysis, and report creation been automated where possible? If mathematical formulas have been used to describe methodology, have these been presented with all terms and steps defined? Where manual steps were involved, have these been thoroughly documented? Has a standard operating procedure (SOP) been produced if necessary? Have the documentation and quality assurance logs been peer reviewed to ensure that the process is reproducible? If we repeat our processes with different software, will we get the same results? | If the process from initial extraction to the final report contains manual steps, then these steps need to be documented so that the team can understand the process. Manual steps can lead to a higher risk of human error, so they need to be clearly understood. | [Guidance: Quality Assurance of Code for Analysis and Research (Government Analysis Function)](https://best-practice-and-impact.github.io/qa-of-code-guidance/intro.html)

[Guidance: The Aqua Book (HM Treasury)](https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/416478/aqua_book_final_web.pdf)

[Guidance: Reproducible Analytical Pipelines (RAP): an introduction (internal link)](rap) | +| Have best practices for coding been followed? | Has an agreed-upon style guide been followed in all of the code written for analysis? Does code adhere to best practice standards? Are there plans in place to ensure there is sufficient capability within the team to use and modify the pipeline as needed in the future? | If best practices have not been followed, there may be a higher risk of error. By understanding the team's agree-upon styles and standards, you can ensure consistency when modifying the pipeline. | [Guidance: Quality Assurance of Code for Analysis and Research (Government Analysis Function)](https://best-practice-and-impact.github.io/qa-of-code-guidance/intro.html)

[Guidance: The Aqua Book (HM Treasury)](https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/416478/aqua_book_final_web.pdf)

[Guidance: Quality Assurance of Code for Analysis and Research (Government Analysis Function)](https://best-practice-and-impact.github.io/qa-of-code-guidance/intro.html) | +| Do you consistently use peer review to check scripts and code, documentation, implementation of methods, processes and outputs? | Have the methodology and outputs been peer reviewed to ensure they appear reasonable? Is your code or analysis ever peer reviewed by someone outside your team? | Peer review is a standard part of analysis best practice. It is helpful because it helps to identify where steps are unclear, documents are hard to understand or there might be problems with calculations or implementation of methods. Routine peer review helps to improve the quality of processes and to reduce risk by identifying potential problems. | [Guidance: Quality Assurance of Code for Analysis and Research (Government Analysis Function)](https://best-practice-and-impact.github.io/qa-of-code-guidance/intro.html)

[Guidance: Quality statistics in government (Government Analysis Function)](https://analysisfunction.civilservice.gov.uk/policy-store/quality-statistics-in-government/#quality-assurance) | {% endcapture %} @@ -397,4 +397,4 @@ Below is a summary of five key documents as they relate to QA. This is not a com 1. [Government Analysis Function: Communicating quality, uncertainty and change](https://analysisfunction.civilservice.gov.uk/policy-store/communicating-quality-uncertainty-and-change/) 1. [Office for Statistics Regulation: Regulatory guidance - thinking about quality when producing statistics](https://osr.statisticsauthority.gov.uk/publication/regulatory-guidance-thinking-about-quality-when-producing-statistics/pages/2/) 1. [The Aqua Book: guidance on producing quality analysis for government](https://www.gov.uk/government/publications/the-aqua-book-guidance-on-producing-quality-analysis-for-government) -1. [UK Statistics Authority: Code of Practice for Statistics](https://code.statisticsauthority.gov.uk/the-code/) \ No newline at end of file +1. [UK Statistics Authority: Code of Practice for Statistics](https://code.statisticsauthority.gov.uk/the-code/)