hr-survey.com

Technical Skills used in Contracting for a Survey

Introduction

A survey is generally a rather complex activity. Furthermore, the results to be derived from a survey are usually difficult to specify, for example, types of data and analyses desired. Consequently, the task of contracting for a survey is not straightforward. Specifically, a considerable amount of effort is required to provide complete and clearly written survey requirements in a Request for Proposal (RFP), select the best vendor to receive the contract, and adequately monitor and review the progress of the survey.

Technical expertise is essential to the success of contracting for and monitoring the execution of a survey. In this chapter, technical skills that should be available when contracting for surveys are presented.

Following are areas in which technical skills are required:

The subject matter knowledge required for a particular survey will, of course, vary depending on the nature of the survey. As mentioned previously, surveys deal with many subjects, for example, demographic, economic, psychological, sociological, and medical topics. The other types of technical skills listed above are more generally applicable to all surveys.

It is very unusual to find all the skills required for a particular survey in one individual, although it is often true that some people will possess more than one of these skills. Consequently, it is usually necessary that there be a Government project team consisting of the project officer and one or more technical advisors to supply the required technical skills. The number and types of advisors needed will vary according to the size and complexity of the survey being planned and the experience of the project officer. Smaller and less complex surveys will usually require fewer, and less specialized advisors than large, complex surveys.

It is important that the need for a project team and the types of members needed be recognized early in the development of a project. At least some of the advisors should participate with the project officer in the following major phases of the survey contracting process: (1) project planning and preparation of the RFP, (2) evaluating proposals and awarding the contract, (3) and monitoring the work done under the contract.

In many cases the agency contracting for a survey will not have on its staff all the technical advisors needed. In such a case, the agency should use technical advisors as part of the project team. These advisors could be persons employed by other Federal agencies or nonfederal consultants. Nonfederal consultants who assist in development of the RFP are not eligible to respond to the RFP. If advisors are needed, they should be brought in at the beginning of the contracting process so that they can participate in planning the project, in preparing the RFP, and in evaluating the proposals. if advisors are brought into the project early, they can provide the guidance and assistance needed in designing and executing a survey.

It is generally a mistake to rely solely on the contractor to provide the technical expertise required during the execution of the survey plan and the processing and analysis of the data. If this is done, the technical quality of the survey may be diminished because the contractor's technical expertise may not be fully adequate, or the contractor may decide that it is not in his or her best interest to take steps to ensure the highest technical quality. Even with the highest quality contractor, the Government should still provide technical review and guidance as part of the monitoring of the survey process; otherwise, the value of the project may be reduced.

The technical aspects of some phases of a survey are often not fully appreciated. For example, in the design of the data collection activities of a survey, a reasonable time schedule cannot be formulated without regard to technical input on follow-up activities, such as the number and type of follow-up attempts required.

The next section of this chapter contains a discussion of the required technical skills listed earlier. The following section covers the type of technical input required at the three major stages of the contracting process. The final section includes a discussion of some of the technical aspects of current practice in contracting for surveys.

Basic Types of Skills Required

Developing Project objectives and Specifications

The formulation and definition of project goals and objectives is the most fundamental aspect of a project. Often goals and objectives are not defined adequately. In such cases there will be confusion regarding the type and amount of survey information to be collected, and hence the strong possibility that the project objectives will not be met. Sometimes a project may have multiple objectives that may be somewhat conflicting. For example, if both subgroup and total population estimates are important, a decision must be made on how to allocate the sample to these subgroups. The sample allocation that would provide equal precision, or some, other specified precision, for subgroup estimates will generally differ from the sample allocation that would maximize the precision for total population estimates. Priorities must be set for each established goal and objective at the beginning of the project in order to resolve any conflicts that may subsequently evolve.

In addition to stating the objectives clearly, an effort should be made to define the objectives to be as narrow or specific as possible. If the objectives are too broad or general, the survey might become too large in scope to manage adequately for the time and funding available.

If the goals and objectives are defined adequately and if the study population has been defined, then appropriate technical specifications can be developed for instrument design, sample design, and data analysis. consequently, these technical aspects of the project should be kept in mind as the goals and objectives are developed.

Subject Matter

In order to formulate and develop a project adequately, there should be a subject matter expert on the project team at the start of the project. Such subject matter expertise is required in order to insure that all data requirements. are delineated. The subject matter expert may not know the appropriate manner in which to phrase a question in order to elicit the desired response, but will know the fundamental issues that must be explored in order to satisfy the data requirements. For example, to develop a survey to investigate the accuracy of the amounts paid to railroad retirement beneficiaries, there should be someone on the project team with knowledge of the railroad retirement system. This subject matter expertise should allow for the formulation of the most appropriate survey questions and should enable the project to collect adequate information from respondents.

Project Scheduling and Costing

Scheduling and costing various phases of a survey are generally difficult. Typically, a gross cost for the project is set very early, particularly for one-time surveys. Agency personnel are generally limited by the initial gross cost allocated to the project.

The amount of time required for planning and writing the RFP is often underestimated. This can delay the scheduled start of the survey which, in turn, can jeopardize the quality of the work. Sufficient time for execution of the survey by the contractor must be allowed.

Since there are several aspects of a survey, such as printing, data collection, and data processing, that may require special clearances, applicable requirements should be determined as early as possible in development of a survey to be conducted under contract. The procurement office can provide a list of required clearances so that their applicability to a particular survey can be determined and provision for compliance can be made in the project schedule.

In addition, there are several survey tasks which involve aspects for which the completion time and cost are typically hard to predict. For example, a survey may involve a large frame development phase in which a number of frame sources have to be investigated. This could require working with a number of computer files involving problems of incompatibility between various computer systems, different record layouts and file definitions, and matching elements in different files. These types of activities are typically difficult to assess in terms of time and cost.

Other survey activities that are typically difficult to assess in terms of cost and time requirements are questionnaire design and testing, data collection, and data processing. Due to the potential for cost and scheduling problems as indicated above, it is important during the project development phase to include a person on the project team who has experience in developing project schedules and cost estimates for surveys.

Questionnaire Development and Testing

Proper development of a survey questionnaire and other data collection instruments is a deceptively difficult phase of a project. The knowledge, time, and effort required to complete this phase adequately is often underrated. In addition to preparing the instruments, time must be allowed to adequately test them. Testing is important since it is generally impossible to anticipate all the problems that will arise when an instrument is used. It is important to include someone on the project team who has had considerable experience in questionnaire development and testing so that the common pitfalls of instrument design can be avoided and so that the instrument may be designed to facilitate data processing.

If a questionnaire is used to collect the survey data and the questionnaire is not developed by someone with appropriate experience, it is likely that some questions will be unclear to the respondents, will be misinterpreted by the respondents, will confuse or upset some respondents, or will tend to influence or "lead" the respondents. The position of the question on the questionnaire may influence the response rate to the question. For example, sensitive questions, like income, are usually answered at a higher rate if they are asked after the respondent has become more comfortable with the interviewer. Also, the nature of the questions surrounding a question of interest may influence response proportions by several percentage points. Furthermore, the questionnaire may be difficult to administer and may not provide adequate survey information. In addition to operational difficulties, these problems can cause both response and nonresponse biases in survey estimates.

Sample Design and Selection

Even for a survey that seems straightforward in terms of the information to be gathered and the definition of the target population, there are usually some difficult questions that arise in designing the sample selection method, particularly if the survey has multiple objectives. Questions about sample size determination, frame development, stratification procedures, sample allocation to strata, or whether or not to use differential probabilities of selection arise in all surveys. These questions become particularly difficult to answer when there are conflicting objectives. For example, deciding whether subgroup estimates are more important than overall estimates or vice versa.

Many of the questions that arise are not easily answered by referring to standard textbooks. Consequently, the project team should include a person who not only has the theoretical knowledge of sampling methods, but who has also had experience in the practical application of sampling principles.

Data Collection

The data collection phase of a project may involve editing procedures to check for faulty or missing data. Procedures also have to be developed to follow-up on non-respondents or incomplete responses. These procedures are important in order to improve data quality and reduce the response and nonresponse biases.

Data Analysis

An analytic plan and strategy should be developed in the earliest stages of the survey development process. Often included in the plan are specific tables to be used in survey tabulations. This plan is the foundation for sample and questionnaire design and is essential for the survey to provide the results intended.

Often the survey analysis phase includes the estimation of population and subpopulation characteristics (for example, means, totals, and proportions). In addition, there are many types of more complex statistical analyses that may be applied to survey data, depending on the objectives of the survey. Some examples of analyses performed are the following: hypothesis testing on population means or proportions, correlation, and analysis of variance.

Before statistical analyses are applied to survey data, imputed (or pseudo) responses are often inserted in the records of respondents that have missing or faulty responses. Setting up an imputation procedure for missing data is a complex process that requires experience. Also, differential probabilities of selection should be taken into account in applying statistical analyses. This is often accomplished by assigning weights to survey respondents to reflect varying probabilities of selection. These weights are usually adjusted to account for eligible sample units that do not respond.

Once the survey estimates and statistical analyses have been made, the results must be examined. Study conclusions, and perhaps recommendations, are then made. The data analysis phase is a very important part of the project and requires the assistance of appropriate subject matter experts and of a statistician who has had experience in analyzing data from surveys.

Quality Control

During virtually any survey operation there are many points when data collection or processing errors can be made that diminish the accuracy of survey data. These instances include sample selection, completing the survey instrument, response editing, coding, and keying the survey data.

It is strongly recommended that quality control procedures be set up at each step to detect and correct as many of these survey operations errors as is feasible. At a minimum, the quality control program should cover interviewing, field editing, coding, and keying. These procedures should be set up by a person who has had experience with survey quality control procedures

Survey Documentation and Report Preparation

Prepare enough project reports to adequately document the survey procedures, results, and conclusions. A conscientious effort should be made to allow adequate time in the project schedule for ongoing documentation of the survey procedures used.

The final project report should include survey results, analyses, conclusions, recommendations based on the results, and suggestions for additional research. Project specialists, such as subject matter specialists and statisticians, should be involved in preparation of the final report.