The vision of the 2006 Census Quality Management Strategy (QMS) is to have 'Data that is fit for use'.
The mission of the 2006 Census QMS is 'to build quality throughout the census process'.
The 2006 Census QMS provides a framework and direction on quality management throughout all the stages of the 2006 Census. Quality management, in this sense, is to be incorporated into the overall design of the statistical system used to produce the census results, and everyone involved has a part to play in ensuring quality throughout the census cycle.
Fitness for use' is the definition of quality that the 2006 Census QMS is based on. This incorporates all of the characteristics of a product or service that affect its ability to satisfy both stated and implied needs, or put another way, fitness for use is defined by the needs of users.
Having quality goals and key strategies to achieve those quality goals are the key aspects of the 2006 Census QMS.
The 2006 Census quality goals are:
- Produce accurate counts of populations and dwellings
- Produce outputs throughout census that are accurate, relevant, timely, consistent and coherent
- Provide information on methodologies and data quality
- Provide accessible outputs and information
- Conduct the census on time and within budget.
The strategies to achieve the 2006 Census quality goals are:
Maximise the coverage of population and dwellings
Set and use quality standards throughout the census
Communicate effectively within the census process, with respondents, and with users
Identify, understand and manage key risks to quality throughout the census process
Monitor and evaluate quality.
For the 1996 Census, Statistics New Zealand set out to provide good quality data within budget, and to have the output database ready for use a year after census day. However, these objectives were not achieved, in part because all of the necessary processes and procedures were not aligned to achieving an agreed level of quality within the fixed budget and timeframe.
For the 2001 Census, it was clear that a different approach was needed, and this became known as the 2001 Census Quality Management Strategy (QMS). The strategy sought to provide direction for the 2001 Census in terms of quality and ensuring that agreed output quality levels were met. The 2001 Census QMS marked a significant change in philosophy, with a move to an overarching strategy which would be followed through the developmental and operational phases (ie development, enumeration, processing, evaluation and output) of the census.
The 2001 Census quality philosophy reflected the strategic direction of the day. The overall strategic direction was set by the Government Statistician, and had a clear output focus. At a similar time, wider corporate thinking led to the development of a more strategic approach to quality across the organisation, in the form of the draft Statistics New Zealand Quality Assurance Framework. This provided a framework for managing and improving quality in surveys.
The clear setting out of quality goals both in terms of processes and outputs produced by these processes was central to the strategic approach adopted. This approach required a clear identification of the key output quality priorities, which could then be used as the basis for coordinated decision-making across the different census phases. The goals were wide ranging and it was critical to identify a set of strategies that could be consistently applied across the board. A key feature of the strategies was that they were able to be implemented across the census cycle, while retaining the output focus. Five key strategies were identified in the 2001 Census QMS to achieve the output quality goals and, by so doing, produce data of sufficient quality. For a list of the 2001 Census QMS goals and strategies please refer to Appendix A.
The review of the 2001 Census QMS found that the 2001 Census QMS had provided a starting point for managing quality throughout the census process. However, the review also found that the 2001 Census QMS had not been fully implemented as originally envisaged, but it did provide a good guide to managing quality which had not been attempted with previous censuses. The 2001 Census successfully met user expectations of making data available within a year of census day, which was partly due to the quality management approach. Risks were well-managed and budgets were tightly controlled. The review found that quality was better understood, quality measures could be discussed with users, and metadata relating to quality was provided for the first time.
The 2006 Census QMS builds on the 2001 Census QMS and has taken into account the recommendations made in the review of the 2001 Census QMS which included feedback from internal and external census data users. Some examples of the adopted recommendations are:
- incorporating the 2006 Census QMS into all parts of the census from data collection through to output
- making the QMS easier to use and ensuring all parts of the strategy are followed
- becoming more output focused (ie focusing on key outputs/cross-tabulations, rather than just individual topics and variables)
- continuing the principle of prioritising variables
- improving the education of internal and external users about the use of the QMS and the quality of census outputs, including the continued provision of metadata to ensure users are aware of the issues and limitations of the data.
During the development of the 2006 Census QMS, international and corporate quality strategies were assessed to ensure that alternative approaches were considered and adopted where appropriate. The 2006 Census QMS quality goals are linked to quality management frameworks from overseas organisations, namely Statistics Canada and the Australian Bureau of Statistics (ABS). The 2006 Census Strategic Plan was likewise taken into account during the development of the 2006 Census QMS. The quality review process in the development of the 2006 Census QMS took into consideration feedback from census staff, subject matter areas, Statistical Methods, an external reviewer (John Cornish), and Census Project Board members.
Definition of quality
The 2006 Census QMS uses 'fitness for use' as the basis for defining quality. This definition covers all of the characteristics of a product or service that affect its ability to satisfy stated and implied needs. In other words, 'fitness for use' is defined by the needs of the user. Throughout the census process attention to quality achievement (achieving 'fitness for use') is required in accordance with the 2006 Census QMS.
The aim of the census process is to provide census data to users. Census data needs to be fit for use. The dimensions of quality addressed in the 2006 Census QMS, and considered within the constraints of time and budget, include accuracy, relevance, timeliness, consistency, coherence, information (eg metadata) and accessibility. These dimensions of quality are interrelated and need to be balanced in accordance with the data items and an understanding of the needs of census data users. This idea is based on the approach outlined in a 2003 ABS Joint Statistical Meetings (JSM) conference paper "2006 Census in Australia: Quality perspective and challenges".
The 2006 Census quality goals are based on the aforementioned dimensions of quality and the key purposes of population censuses. The quality goals are:
G1. Produce accurate counts of populations and dwellings
G2. Produce outputs (from initial planning to delivery of data and information)
- throughout the census that are:
G3. Provide information on methodologies and data quality
G4. Provide accessible outputs and information
G5. Conduct the census on time and within budget.
Five key strategies have been identified to achieve the 2006 Census quality goals and, in the process, produce data that is fit for use. These strategies also bring together and reiterate good work practices that may already exist in past or current census projects.
Strategy 1: Maximise the coverage of population and dwellings
One of the main purposes of the population census is to obtain the most accurate counts possible, that is, to accurately measure total population and dwelling counts and high level sub-populations. To help achieve this purpose, the census needs to have processes and procedures in place to maximise the coverage of population and dwellings. There is a great need to be responsive to the needs of respondents. The key areas of census identified in the 2006 Census QMS that deal with maximising coverage, including the obtaining of accurate and complete responses, are:
- Questionnaire design
- Data collection
- Multimodal questionnaire design and data collection
- Data analysis.
Strategy 2: Set and use quality standards throughout the census process
The development of quality standards is a key part of being responsive to the needs of census data users and ensuring that the data is fit for use. These quality standards should reflect what is required to meet key uses of census data within the constraints of time, budget and the capability of the survey. Setting up quality standards and ensuring that they are used to direct the development of procedures and processes throughout the census will help ensure the achievement of acceptable data quality (ie 'fitness for use'). Where appropriate, consultation will be undertaken with users to ensure their needs are considered within the decision-making process.
Census variables have been prioritised and placed into different levels based on the uses of census data and the constraints of the census process. This will help to ensure that the quality of the data reflects the uses to which the data is put.
Level 1 - Foremost variables
Foremost variables are core census variables that are output and are the key reason for conducting a census. In broad terms, the foremost variables are final counts of population and dwellings, age, sex, ethnicity and location. Some of these (eg usual residence by age by sex) are the key outputs used for maintaining the accuracy of population estimates. These variables are given the highest priority in terms of quality (ie accuracy, relevance, timeliness, consistency and coherence), time and resources across all phases of the census.
Level 2 - Defining variables
Defining variables are variables which define key subject populations for which the census provides measures that are important for policy development, evaluation or monitoring. Defining variables are used frequently in cross tabulations with foremost variables. They represent key sub-populations and measures that are of high public interest, for example, families and households, iwi, birthplace, and labour force status. These variables are closely linked to the main purpose of population censuses, and in the New Zealand context may only be available in detail (eg at the sub-national level) from the population census.
Level 3 - Supplementary variables
Supplementary variables are variables that do not directly fit in with the primary purpose of a population census, but are important to certain groups. Occupation, language, and religious affiliation are all examples of supplementary variables. These variables have third priority in terms of effort and resources. However, there are minimum quality standards (eg within fitness for use specifications) that have to be met in order to make the output data suitable for use.
Other data not included in the three quality levels
These variables are traditionally not output and are not captured, but are instead used to help with coding or as back-up information for field management and prosecutions. Name and address, telephone number, and name of employer are all examples of this type of variable. There are minimum quality standards to be met for enabling these variables to be fit for use.
For a full list of the variables in each quality level please refer to Appendix B.
Strategy 3: Communicate effectively within the census process, with respondents and with users
Effective communication is required to help ensure that the other quality strategies work and to achieve the census quality goals. The communication strategy brings together and highlights good existing work practices. In order that there are no surprises, effective communication is required with internal staff (eg within the census process, with subject matter experts/stakeholders, internal data users and service areas) and also external contacts (eg respondents, contractors, and census data users). Key areas for achieving overall effective communication have been identified. These include education and training, documentation, use of classifications and standards, specification templates, and providing information on methodologies, processes and data quality.
Strategy 4: Identify, understand and manage key risks to quality throughout the census process
In order to achieve the successful implementation of quality throughout the census process and management systems, it is essential that the key risks to quality are identified and risk management tools and processes are set up. The management of risk involves a number of steps as outlined below. The first and most important is to try and avoid risk wherever possible. However, where risks are identified, there is a process for managing them. The risk management steps are:
- Minimise risk
- Identify risk
- Assess and prioritise risk
- Monitor risk
- Manage risk.
The Census Business Unit at Statistics New Zealand has implemented a three-tier approach for managing risk:
- Project level – managed by a project manager
- External perspective - for example, collective suppliers - Senior Statistics New Zealand staff and suppliers meet with an external facilitator to identify and manage risk on a regular basis
- Programme level - Census and Business Transformation Strategy (BTS) management meet each month to identify and manage risks at the programme level. As part of this the Programme Level Risk Register is reviewed monthly for the purpose of identifying where risks can be reduced.
To date, the following parts of the risk management process have been created:
- Regular meetings to identify, assess and ensure mitigation actions are in place.
- All risk registers held in a central depository accessible to the Census Business Unit and Statistics New Zealand. PlanWise is being used to assist in this process.
- Regular reports produced - Suppliers Perspective Risk Register, 2006 Census Programme Risk Register, and project progress reports. Risks considered high or significant, as well as the movement of risks and updates on mitigation actions, are reported at each census monthly managers meeting for comment and feedback.
- Reviews following all major tests.
Strategy 5: Monitor and evaluate quality
Quality across all aspects of the census process needs to be monitored and evaluated. Likewise, timely feedback needs to be provided to relevant people to ensure informed decisions are made, actions to address issues are carried out, and that there is transparency throughout the census process. Monitoring and evaluation is also important for the managing of quality for future censuses.
Monitoring of quality should be undertaken throughout the census process to assess the success of different processes and the potential impact they will have on data quality. In order to do this, there needs to be monitoring systems and measures in place at all phases. Monitoring quality will also help with obtaining accuracy and relevance, and should be promoted across all of the census process. This will ensure that the agreed quality standards are continually being met, that the processes set up to minimise errors are working effectively, and that unanticipated errors do not adversely affect quality at any stage.
Information from the measurement and monitoring of data quality will be used to evaluate the 2006 Census and provide input into the development of the next census, as well as improve current processes by providing timely feedback. Another key use of evaluation is to inform users. It is important that all phases and subject matter areas work together during evaluation and ensure that everyone has the same expectations. There needs to be an ongoing evaluation of data, systems and processes throughout the census process (eg there is a project to evaluate data and certain systems through the processing and post-processing phases). Evaluation also includes project reviews/debriefs.
The overarching rationale for having a quality management strategy for the 2006 Census is to ensure that the final output data produced is fit for use. This requirement is reflected in the 2006 Census QMS's vision statement. According to that vision, success for the 2006 Census QMS will be measurable by the extent to which the data is fit for use and metadata is provided to help inform data users about the quality of the data and underlying strategies, methodologies and processes. Examples of tools for evaluating and measuring this include: comparing the data produced with the requirements in the quality standards, and conducting a survey of census data users on the quality of the data and whether or not they considered it fit for their use.
Once the effects of the 2006 Census QMS have been established, attention can then be turned inside the census to the processes to establish causes. The mission statement of the 2006 Census QMS is to build quality throughout the census process. According to that statement, success for the QMS will be measurable by the extent to which all the processes involved in the census are quality processes that contribute to producing data that is fit for use. It therefore seems reasonable to examine each process to find out whether the data received by it was fit for use at that stage of the process, and whether or not it delivered data that was fit for use at the next stage. It is also important to establish how easy, and to what extent, the QMS was to use, by asking, for example:
- Did people understand its requirements?
- Did it provide an overall framework and direction on quality management?
- How easy was it to implement?
- How much of it was implemented?
- Did it cover all that it needed to cover, or contain anything unnecessary?
Examples of tools for evaluating and measuring these aspects include: census-wide review/debrief reports, which address the use of the 2006 Census QMS; focus groups/surveys, which ask staff about the QMS (eg whether or not it was easy to use); and the final 2006 Census QMS implementation monitoring report, which checks to see what QMS action points were or were not implemented at the end of the 2006 Census. Whether the QMS goals were achieved and whether or not the QMS strategies were implemented and helped to achieve the stated goals, are aspects that also need to be looked at.
Implementation of the 2006 Census QMS is currently in progress. Regular monitoring and reporting on the implementation of the quality management strategy across the census process is being carried out. Some of the observations that have emerged so far in the use of the 2006 Census QMS are that it is being used as a framework for prioritising and decision-making, it is providing guidance for monitoring and evaluation, it is reiterating good work practices, and it is providing a reminder that the main focus of the census is to produce data that is fit for use.
2001 Census QMS output quality goals
- Maximise the coverage of dwellings and people
- Produce relevant and useful outputs
- Produce outputs on time
- Produce outputs consistent with previous censuses
- Produce transparent and well-documented methodology underlying the census outputs
- Conduct the census within budget
- Control and reduce respondent load
- Evaluate the data and methodologies used.
2001 Census QMS strategies to achieve output quality goals
- Differentiate between levels of output quality
- Agree on quality standards for outputs in 2001
- Identify areas of high risk to quality throughout the census process
- Manage and reduce this risk
- Monitor and measure quality and provide feedback to users.
Level 1 - Foremost topics/variables/outputs
- Count of the population (final)
- Count of dwellings (final)
- Meshblock location of each dwelling in New Zealand
- Age of all respondents in New Zealand on census night
- Sex of all respondents in New Zealand on census night
- Location of all respondents in New Zealand on census night to meshblock level
- Usual residence to meshblock level of all usually resident in New Zealand
- Ethnicity of all respondents in New Zealand on census night
Level 2 - Defining topics/variables/outputs
- Families and Households
- Relationship to Reference Person – foundation variable for family coding, which in turn, is used for determining families and households
- Absentees – foundation variable for family coding, which in turn, is used for determining families and households
- Family Type and associated variables/derivations
- Child Dependency and associated variables/derivations
- Household Composition and associated variables/derivations
- Extended Family Type and associated variables/derivations
- Number of occupants on census night
- The number of occupants variable is used to balance households. This variable needs to be of high quality to ensure balancing proceeds relatively smoothly. Enumerators also use the number of occupants question, in addition to the relationship to reference person question, to ensure all individual forms have been collected. This variable contributes to producing accurate counts of the population.
- Person Record Type – New Zealand adult, New Zealand child, overseas adult, overseas child, absentee – helps define key subject populations
- Unoccupied dwellings – contributes to dwelling counts
- Dwelling Record Type – private dwelling, non-private dwelling
- Maori Descent
- Usual Residence Five Years Ago
- Dwelling Type
- Tenure of Household
- Ownership of Dwelling – dwelling form question – key contributing variable for the derivation of tenure of household
- Mortgage Payments – variable used in the derivation of tenure of household
- Weekly Rent Paid by Household – this variable is included as an output and may be used in the derivation of tenure of household.
- Work and Labour Force Status (WKLFS)
- Job Indicator – key contributing variable for the derivation of WKLFS
- Hours Worked per Week and associated variables/derivations – variable used in the derivation of WKLFS
- Job Search Methods – variable used in the derivation of WKLFS
- Available for Work – variable used in the derivation of WKLFS
- Seeking Work – variable used in the derivation of WKLFS
- Status in Employment
- Legal Marital Status
- Social Marital Status
- Personal Income
- Sources of Income
- Sector of Landlord
- Highest Secondary School Qualification
- Post School Qualification Level of Attainment
- Post School Qualification Field of Study
- Highest Qualification.
Level 3 - Supplementary topics/variables/outputs
- Sector of Ownership
- Number of Bedrooms
- Number of Rooms
- Number of Other Rooms
- Fertility – This variable is a cyclical topic and was not collected in 2001.
- Years Since Arrival in New Zealand
- Years at Usual Residence
- Main Means of Travel to Work
- Workplace Address
- Fuel Types Used to Heat Dwelling
- Access to Telecommunication Systems
- Number of Motor Vehicles
- Rent Indicator – This variable was not included as an output in 2001. It is used in the derivation of tenure of household.
- Unpaid Activities
- Tenure Holder – individual form question
- Religious Affiliation
- Living Arrangements – This variable supplements the determination of families and households and is used in the derivation of social marital status. It was not included as an output in 2001.
- Cigarette Smoking and associated variables/derivations – This variable is a cyclical topic and was not collected in 2001.
- Disability – This topic will not be output from the 2006 Census, but will instead be used as a filter question for a post-censal survey, subject to funding for a post-censal survey. If a decision is made to sample in the census, this topic will be exempt.
Other data not included in the three quality levels
- Telephone number – Used for post-censal surveys. Telephone numbers could potentially be used for non-response telephone follow-up.
- Name of employer – Used for industry and occupation coding.
- Tasks and duties – Used for industry and occupation coding.
- Name and address – Used to assign person numbers to each individual form, and so the collector can ensure a form has been completed by everyone in the dwelling. This information is also crucial for prosecutions. If someone refuses to complete a census form, collectors must attempt to obtain as much information as possible about the person (including name and address) and record this in a Refusal Report.)
- Archiving – If a decision is made to go ahead with this question, the responses will be captured.