|  | | |
Using Both Internet and Field Collection Methods for the 2006 Census of Population and Dwellings
|
Background
Challenges faced and lessons learned
Managing the impact on existing business process
Managing stakeholder expectations
Security / privacy and confidentiality of information
Statistical
Level of respondent burden
Understanding impact on data
Technological
Conclusion
References
The Statistics Act provides that population censuses be conducted every five years in New Zealand. As with a (declining) number of other countries, censuses in New Zealand are based on the model of collector drop-off and collect, although a small proportion is returned by mail. Since 1945, separate dwelling and individual forms have been provided for all households. These have been available in both English and bilingual (English/Māori) formats since the 1996 Census. Support for the census remains high, with the Post Enumeration Survey in 2001 showing a net undercount of 2.2 percent (+/– 0.3 percent).
It is recognised that this high level of support cannot be taken for granted. New Zealand society is changing in many ways. It is becoming more diverse, and people are busier and less available and less willing to accept the need to participate in statistical activities such as the census. Public attitudes in New Zealand and in many other countries indicate a strong demand for choice and convenience when interacting with government. This has been an important driver in deciding to proceed with the provision of an online form for the 2006 Census.
From a statistical perspective, the key drivers for an online form are:
- to improve/maintain participation in the census and give people choice
- improve aspects of census data quality.
Field tests have both proven the feasibility of this approach and confirmed that the drivers’ objectives are being met.
As the 2006 Census will be the first mixed-mode census in New Zealand, the learning gained from it will position Statistics New Zealand to take advantage of the efficiencies the technology can provide in the 2011 and following censuses. Statistics New Zealand has already made significant gains with imaging of forms since the 1996 Census, and will also make savings in both collection and processing costs in the future. Other major benefits should include timelier and higher-quality census data. There are also wider government benefits – the provision of online forms makes an important contribution to meeting e-government objectives for delivering a wide range of government services via the Internet.
The risks inherent in inaccurately forecasting Internet take-up levels would place the integrity of the 2006 Census in jeopardy. Therefore, there has been no attempt to leverage efficiency gains in any of the traditional census processes. Paper forms are being printed for all households and individuals, and there will be a full quota of field collectors engaged for the census. This is a longer-term investment in improving participation in and data quality from the census. The census experience of electronic data collection will also help Statistics New Zealand to better understand how this technology might improve other areas of statistical data gathering.
Back to Top
In developing the online census, a number of challenges were faced, from both business and technical perspectives. While the technical perspective is extremely important, this paper focuses primarily on the business requirements, exploring how these have been met rather than the technical solution implemented. The online form is a translation of the paper form into an online questionnaire, maintaining the look, feel, intuitiveness and intentions of the paper form. This, in itself, was not an inconsiderable task, but the results have been well received and gained international acclaim.
Broadly, the challenges can be categorised as:
- managing the impact on existing business processes
- managing stakeholder expectations
- statistical
- technological.
Back to Top
Integration into field collection
The most important element of the online census is successful integration into the field collection phase. Without this, there could be no online option for the 2006 Census. This option increases the expectations of the public for immediate updates to field staff of forms submitted to census processing. Our aim has been to ensure that the respondent who completes their census form via the Internet one evening is not faced the next day with the collector returning to pick up their paper form. The immediacy of the Internet heightens community expectations that the same use of technology permeates across the whole collection process.
To meet this need, information about the forms collected online will be communicated to the field using mobile phone text messages. These will be bundled and despatched at certain points in the day to coincide with field staff work patterns.
Prior to delivering forms in the field during the dress rehearsal, the collectors were given access to the online census facility, to enable them to familiarise themselves with the online option before it went live. This was invaluable as an aid to their understanding of how the option would be viewed by respondents prior to standing at their doors and offering that option. This will be key in ensuring that we gain the full potential of the option, both for maximising coverage and increasing the quality of data submitted.
Additional training was also introduced for the field collectors to ensure they understood what is required of the respondent, regardless of which option is chosen.
This is to ensure, firstly, that the online option is communicated correctly to the respondent on the doorstep; and, secondly, that the field communication systems integrating the online option with the paper option are correctly performed. This will have an impact on both the time taken to communicate procedures and the amount of information that needs to be absorbed by collectors before they enter the field.
The training theme continues with field supervisors, who need to be able to use the field operations monitoring systems. These are the mechanisms used to monitor how the field responds to the text messages they receive about the lodgement of Internet forms and related helpline actions. The complexity of these systems and the need to integrate with existing systems for the paper form collection process have posed a real challenge to ensure seamless operations, and it will be essential for the 2006 Census that this integration process performs as required.
Communication / promotion
How the option is communicated to the population will have an impact on both who the respondents are and the level of uptake. As stated earlier, promotion of the online census to the desired audience will be key to managing the eventual level of uptake.
Consideration is being given to promoting the online census to particular ‘Internet savvy’ groups and other sectors of society where we have had less success using traditional collection methods, such as people living in apartments, the Asian community (research has shown this sector to be more receptive to completing forms online), students, and people in rural areas.
Complicating matters further, there will be two separate requirements to meet the needs of users for the Internet-based solution for the 2006 Census:
- online census form completion
- online general census information.
Online census form completion option
The first requirement is to facilitate completion of the census form via an online census option. The specifications call for a highly secure, easily completed, stable online application that allows high volumes of census forms to be completed over the limited census data collection period. Each form must be completed within a single session, as there will be no capability to resume partially completed forms. The online census also has a concise bilingual context-sensitive help facility, the content of which has been based on the level of online support expected by the Internet respondents that has been identified through field testing.
Online provision of general census information
The second requirement is to provide a comprehensive source of more general information relating to the census. This will provide information relating to the current census, along with assistance in completing the paper-based census form. Information will include FAQs, multilingual help (for ‘paper-based’ respondents) and a request for forms. Statistics New Zealand will have the ability to dynamically update certain areas of information on a regular basis. This information will be available for an extended period prior to and following the census.
The online provision of general census information is consistent with research which has shown that accessing a website or using email are the most preferred sources of assistance. The 2003 Government Online survey (GO2003) identified that online information, including email enquiries, was preferred by 48 percent of New Zealanders, with telephone the next most preferred medium, favoured by 30 percent of people.
The online census option needs to be completely ‘locked down’ for security reasons, and available only for a relatively short time (six weeks). In contrast, the more general range of general census information needs to be able to be updated on a daily basis and available for much longer (seven months). This creates an interesting dilemma in terms of how to promote and communicate the availability of each without compromising either.
Availability / peak activity periods
The GO2003 survey found that 75 percent of New Zealanders had used the Internet in the month prior to the survey. New Zealand had the highest Internet usage of all participating countries and well above the global average of 45 percent.
The same survey identified that completing the census form online was perceived to be significantly useful by 69 percent of New Zealand’s population. The highest rating sectors were university-educated people (88 percent) and higher income earners (84 percent of those earning $70,000–$100,000; 89 percent of those earning $100,000 or more). This will prove useful when considering the target audience for promoting the option.
Testing has shown that actual uptake does not mirror the initial interest shown on the doorstep. Uptake depends more on the particular motivation of the respondent. Our own follow-up surveys have identified that the most common reason for using the online option is that there is no subsequent need to wait for a collector to call to pick up completed forms. From the results of our field tests in March and November 2003, we project that around 15–20 percent of people may take up this option in 2006. That's around 600,000 people, or 250,000 households.
Recognising that the online option for the 2006 Census could be New Zealand's largest online event to date, the system is being designed for stability and performance when there are high volumes of concurrent users, with back-up sites, multiple servers and substantial redundancy. In the event that there are more concurrent users than the system can handle, a message will display advising users to try again later.
As part of its communications and risk management strategy, Statistics New Zealand is also talking to telecommunications providers and Internet Service Providers (ISPs) to integrate network load planning for the 2006 Census.
Uptake
New Zealand’s census is focused on a single day, with the prime activity taking place on one night – known as ‘census night’. Statistics New Zealand is designing the system for a spike of activity on census night and is confident that the system will handle the predicted load. Testing to date has shown that while a number of people are likely to complete their online census forms in advance of census night or on the day after, the majority will complete their forms on census day, as required.
The following chart shows there was a spike of responses on the dress rehearsal test day of 8 March 2005. A large proportion of the Dwelling Forms (DF) were received on the test day (45 percent). This pattern was also shown during both of the previous field tests, conducted in March and November 2003.
|
Adding to the overall challenge is the fact that the projected peak activity of Internet traffic will coincide with the heaviest demand for help over the public telephone helpline, as witnessed during the 2001 Census. The competing demands for use of the nation’s telecommunications infrastructure need to be well managed to mitigate the risk of a telecommunications ‘meltdown’. To address this risk, Statistics New Zealand has established joint risk workshops with all of the communications providers for the 2006 Census.
Usage by hour of day
The following chart shows the website visitor sessions for the March 2005 dress rehearsal. The same trend was present in the field tests in March and November 2003.
The chart clearly identifies that most website visitor sessions were recorded from 19:00 hours to 23:00 hours, with the peak hour between 20:00 and 21:00 hours. The system is being designed to cater for this expected peak website visitor session activity. Again, the competing demands of the public telephone helpline have to be considered in this narrow window of service demand. The demand profile for the helpline in the 2001 Census shows that peak activity for the helpline was between 18:00 and 20:00. There is, therefore, an overlap from 19:00 to 20:00 when the telecommunications infrastructure would be handling a peak of both voice and data connections.
Back to Top
E-government requirements
The e-government strategy requires the following strategic alignment by 2004, 2007 and 2010 for all major government agencies, including Statistics New Zealand:
- By June 2004, the Internet will be the dominant means of enabling ready access to government information and services.
- By June 2007, networks and Internet technologies will be integral to the delivery of government information and services.
- By June 2010, the operation of government will have been transformed through its use of the Internet.
Statistics New Zealand is already strategically positioned to meet the 2004 requirement, through its own web portal which provides ready access to a wealth of statistical information and services.
When the 2007 requirement is considered, there is an additional need to look at the phases of the survey cycle that are concerned with other interactions with the New Zealand public – whether in terms of interactive information provision, or online data provision.
Meeting the requirement for 2010 will be the most challenging, as this requires that services across government are seamlessly integrated to ensure respondents do not have to provide the same information to one department that they have already provided to another. Meeting this requirement is not considered appropriate for the census, as it would lose the independent and impartial position it currently occupies. The accuracy of data supplied by individuals to census could be jeopardised if it was felt that the data supplied would be freely shared across government.
Back to Top
Public perceptions about the security of the online census option will inevitably contribute to final Internet uptake levels. It has already been established that the current perception of the public about the Internet being a secure place to transmit data is not very high in New Zealand. A Statistics New Zealand Public Attitudes Survey in April 2003 showed that only 5 percent (of 700 respondents) felt that the Internet was the most secure method of data collection compared to the other methods available. Interestingly, those who were comfortable with using the Internet and used it frequently believed it to be more secure.
The GO2002 Survey commissioned by the e-government unit (1,000 respondents) found that only about one-third (31 percent) of New Zealanders consider it safe to use the Internet to provide government with personal information; and results from the GO2003 survey showed that only 35 percent of New Zealanders consider it either ‘very safe’ or ‘safe’ to provide personal information to government over the Internet.
Although public perception of the adequacy of Internet security seems to be rising, it would take only a single large negative event to reverse that trend. Any approach involving the security of data needs to be managed with these issues in mind. For instance, what would be the expected response from the public to finding out that sensitive data they provided during census was stored on a private company’s own server housed on their own premises, and not on government premises? And if we promoted the Internet as the most secure environment to transmit data of all the options available to respondents, what effect would that have on people who did not have the option to use the Internet? Would they then mistrust the collector and consider that information provided on the paper forms might not remain confidential? The right balance needs to be found in the messages given to the public regarding use of the Internet as a secure environment.
Data provided by respondents over the Internet needs to be secure from a number of perspectives. The public already has an expectation, through online banking and shopping, that they will need to log on to the site and be authenticated, and that their transactions will be encrypted. Census needs to meet these expectations, if not exceed them. However, the balance struck between the perceptions of the public about the integrity of a secure log on and authentication process and how complex it is will be another important factor in determining the level of uptake for this option. If it is believed to be too time consuming, then respondents may abandon the online census in favour of paper forms. Testing so far has shown that our log on process is not considered onerous.
Although data will be held externally to Statistics New Zealand when it is first captured, it will be subject to stringent physical and logical controls. The data centre facilities will be physically and technically separated from all other services at the two hosting sites; under the full control of Statistics New Zealand personnel during both operation and testing; and decommissioned fully following the census. The collected data will be encrypted at all times and transferred to the census processing site via a secure virtual private network. All data collected externally to Statistics New Zealand will be deleted from data centres every 24 hours to ensure there is no possibility of unauthorised access.
Back to Top
Usability / accessibility
On the other side of security and performance risks is the requirement to ensure all online census functions are not only easily accessible to users, but also easy to use. By making access to the online form too complex, there is a risk of both lower uptake and negative publicity. If respondents are familiar with using Internet applications and transactional websites such as online banking or shopping, they will find it relatively straight forward to complete their census forms online. Statistics New Zealand has tested the design internally and in public trials, and usability has been key during the forms’ development.
The online census has been developed to represent, as closely as possible, the paper census forms in an online presence. Transferring a paper form to an electronic format is a process that, while simple in theory, is far more complex in practice. A number of processes for the paper form that are conducted manually by field collectors would fall to the online respondent. In this sense, we have both increased and decreased the burden on the respondent.
Back to Top
Increase |
Decrease |
| The respondent is unable to submit the form if one or more of the six compulsory fields have not been completed, ensuring that we have high-quality data for the foremost variables – age, sex, ethnicity and location. |
Based on responses to certain questions, the respondent is ‘routed’ past non-required fields.
|
| Field validations prevent ‘impossible’ values in respondents’ responses, thus allowing online checking of the data being entered. |
A summary of forms completed and receipt numbers are provided on the Household summary page. |
| The relative complexity of the log on process adds burden to accessing the online form. |
There is no need for the collector to return to collect the completed form. |
Back to Top
Understanding impact on data
The use of an electronic form carries with it the potential to increase or decrease the quality of the data being received, compared with that received through the traditional paper method. This creates the risk that the two sources would be so different that the electronic data would be unable to be combined with the paper-based data. Time series data would also be affected immeasurably. In practice, however, testing has shown that the sector of population choosing the online option would provide the same high-quality level of data regardless of the medium they used to supply it. Being aware of this capability to distort data, the 2006 online census will have minimal data editing and validation at the user input stage to reduce both the potential disparity between the different modes of data collected and the impact on the respondent for data input interruptions.
The completion of census forms online can be performed by anyone in a household. It may be that one person takes responsibility for the whole household and completes all forms that are due, or individuals may take responsibility for submitting their own census data. Younger people may even want to be involved and complete their own individual forms. Although Internet form completion has been shown during testing to be mainly homogeneous, there is nothing to prevent some members of a household completing paper forms instead. The collection processes will need to accommodate these eventualities.
With more than one collection mode, there is the opportunity for a percentage of respondents to duplicate their census forms, either deliberately or accidentally. There is also an increased possibility that duplicate forms may be submitted if respondents complete and submit forms prior to census night and then change their intentions on census night. Although there are checks to ensure that only one form per individual is processed for the census data, the pre-processing effort to identify and remove redundant duplicate forms is likely to be increased.
Although testing has not shown evidence of partial form completion or item non-response, this remains a risk. The online option has been designed with this in mind, as the respondent is unaware of the required compulsory fields until the end of the form has been reached, at which point the respondent is prompted to check the details provided for compulsory fields. Online forms cannot be submitted if entries are not present in all compulsory fields.
Back to Top
Unsurprisingly, a number of technological challenges have been encountered while developing the online option, not least of which has been determining and implementing the solution 18 months prior to the 2006 Census. The external environment is constantly changing; as each new virus, security flaw or software bug is addressed, a new and more persistent threat takes its place.
The technological challenges relate not only to the environment the online option is hosted within, but also to the technology used to build it. Currently, a mix of three computer operating systems and four Internet browsers need to be tested for compatibility against the online option. As these are upgraded, patched or simply replaced, so the testing cycle starts once more. Compatibility testing is a very time-consuming, but necessary element of the development process to ensure the widest possible coverage for both the respondent and Statistics New Zealand.
Back to Top
The learning gained through both the use of this technology and respondent behaviour will prove invaluable as government and public expectations increase for the 2011 Census. The expectation of government will be that future operations will be carried out using less resource, and consequently at a lower cost. The public will expect to be able to provide information to government by electronic means. It is essential, therefore, that Statistics New Zealand successfully implements the online census in 2006 and uses the knowledge gained to plan for a more efficient and effective 2011 Census, when real savings may be both expected and realised.
Data quality in the 2006 Census will be maintained, if not increased, through both the overall coverage of the online census and the quality of response.
Back to Top
2003 GO Survey.
http://www.e-government.govt.nz/docs/go-survey-2003/chapter1.html
Printable version (133 KB) - The downloadable file is in MS Word 2002 format.
Powerpoint presentation (1.8MB)
An invited paper for
The Population Association of New Zealand Conference
Ian Smith
Online Census Project Manager
Statistics New Zealand
Christchurch, New Zealand
ian.smith@stats.govt.nz
|
|
|
|
|
|
|
|