• Share this page to Facebook
  • Share this page to Twitter
  • Share this page to Google+
IBULDD project peer group review: Findings and recommendations

October 2006

  • John Blanchette (Australian Bureau of Statistics)
  • Ron Jarmin (US Bureau of the Census)
  • Felix Ritchie (UK Office for National Statistics)

Summary

This paper records the findings and recommendations of the first peer group review of the Statistics New Zealand IBULDD (Improved Business Understanding via Longitudinal Database Development) project, held in October 2006 in Wellington. The key results are as follows:

  • The review team was satisfied that the project was making sufficient progress towards its targets.
  • The team was impressed by the quantity and quality of work carried out to date.
  • The action points and recommendations raised by the review are relatively minor adjustments to the flow of work.
  • The review team offered its strongest support for the project to continue into its second year and then to become a long-term project.
  • In making this last point, the review team offered a note of caution that the project team and Statistics NZ management should not have expectations of perfection from the database. Releasing information from the database using the ‘experimental’ tag and making it available to researchers in the short term, despite its imperfections, is the best way to ensure long-term improvements.

Review findings

The IBULDD project has been active at Statistics NZ since the beginning of 2006. The project involved bringing together numerous data sources to build a single unified data source for new and expanded official statistics. It was also hoped that this dataset would be available for research use, subject to Statistics NZ’s standard confidentiality arrangements. The peer group review was organised to take stock of progress, assess the programme for the remainder of the project, and make recommendations as to the project’s development.

The review team was unanimous in its view that this was a useful and important project that should be continued in line with the original project plan. The review team expressed enthusiasm for the aims and methods expounded, and remarked positively on the way Statistics NZ had addressed many difficult problems.

The minutes of the peer group review contain 36 action points, most of which are of a technical nature. Five overall development areas can be highlighted.

Objectives

The review team agreed with the objectives of the project, in principle, but noted that the objectives required some clarification and prioritisation. Some objectives were legal requirements and some were desirable features, and not all could be solved at the same time. The IBULDD team agreed to review the objectives, identifying the short- and long-term focus and choosing two objectives to concentrate on in the short term.

Recommendation 1
  • objectives are split into short term and long term, with two objectives for the short term identified and prioritised.

Official statistics

The need to generate official statistics from this dataset was of paramount importance for ensuring that the project met the requirements of the Statistics Act 1975. The reviewers clarified their views on what should be produced, as there was a need to generate statistics that are not available from existing sources (or not available at a high enough degree of quality or detail). It was felt that the greatest addition to official statistics would be to generate more longitudinal and distributional information with appropriate industrial breakdowns, and to widen the range of economic variables described. The reviewers were particularly keen on the distributional element, as this can be highly informative but is rarely produced by national statistical institutes. Comprehensive descriptions of the characteristics of the distributions of many key statistics are possible only with universe microdata files such as those proposed in the IBULDD project. Thus, these types of statistics would be informative and not possible without the project.

Recommendation 2
  • the official statistics to be produced are clarified and approved, in principle, as satisfying the legal requirements of the project.

Longitudinal elements

The similarities between this project and LEED (the Linked Employer-Employee Data) were noted. LEED had been useful in pathfinding many of the longitudinal concepts and techniques, although the review team regretted that legal restrictions meant that LEED could not be used directly in the construction of IBULDD.

It was suggested that IBULDD could work with LEED to develop a ‘change model’ (ie showing the longitudinal elements developing over time) for the web, if possible. This would help to stimulate user interest and clarify the research demand for the dataset. The review team also supported the potential future coming together of the projects, although this is outside the scope of the present phase of IBULDD.

Recommendation 3
  • IBULDD to continue to build on experience gained from LEED and feed back its own findings, to ensure efficiency in building the longitudinal data and consistency in any future Statistics NZ projects bringing the two datasets together.

Coverage

IBULDD pulls in data from several different sources, some of which overlap. This gives rise to questions on which data should be used and how much it can be trusted; ultimately, which firms will be included in the dataset? There was a difference of opinion on this issue within the peer group, between (a) focusing on a high-quality core of data (eg only those units that are both on the frame and with GST data) and extending coverage later; and (b) keeping and using all data, but possibly making adjustments for the source.

It was agreed that the novelty of the data sources being used made this hard to decide a priori, and that at this stage a simpler, cleaner dataset might be easier to work with (for both manipulation and imputation) and explain. This is consistent with Recommendation 1, which seeks to split the project into short-term and long-term objectives. Initial work on this area will study the quality impact of reducing coverage, and will consider whether there are ‘problem’ industries or sectors. It was noted that analysis showed good correlations for those variables where the data was available.

Recommendation 4
  • IBULDD to report on the quality impact of reducing coverage to core sectors and/or data sources.

Research use

It was noted that the research use of the dataset was an aim subsidiary to the main aim of producing official statistics. However, the review team remarked that, in their experience, researchers make critical contributions to the quality assurance and improvement of datasets. Hence, research use remains an important objective, and the early involvement of external researchers (from academia, government ministries and other research institutes), subject to legal constraints, is to be encouraged. This included presentations on the project, to identify likely research uses.

One highlighted danger was of trying to design a dataset for research use, for example by the construction of additional variables or unnecessary imputed values. It was felt that the IBULDD team should focus on its core objective of building a high-quality, coherent database from disparate sources, and allow ‘research’ features to be developed as demand and opportunities arise. IBULDD should try to avoid closing off avenues for potential research, but should also not try to anticipate them too much.

Recommendation 5
  • IBULDD to investigate increasing the involvement of external researchers in the development of the dataset and the identification of wider research demands.

Conclusion

The 36 action points and five recommendations are relatively minor comments on the development of IBULDD. No significant problems were identified, and the level of commitment to the project by Statistics NZ was impressive.

Overall, the peer group review team was satisfied with the direction of the project and the progress made to date, and offered very strong support for the project to continue into its second year.

  • Share this page to Facebook
  • Share this page to Twitter
  • Share this page to Google+
Top
  • Share this page to Facebook
  • Share this page to Twitter
  • Share this page to Google+