RFPs for Contracting for a Survey
Discussions with your contracting/procurement office will enable determination of the contracting options, if any, which are available, the input expected from the program staff, the steps which should be followed from development of the RFP and a list of potential vendors through award of the contract, and the amount of time each step is likely to take.
A contract may stipulate a fixed-price which is agreed upon in advance by the sponsor and the contractor, or it may provide for reimbursement of allowable costs (up to a preset maximum) incurred by the contractor in performing the survey. The decision about which approach to use is usually made by the procurement office based on technical contracting considerations, not by the program staff. The RFP should indicate whether the proposed contract will stipulate a fixed price or will provide for reimbursement of costs incurred by the contractor.
There are advantages and disadvantages to each of these payment mechanisms. For example, fixed-price contracts assure, in advance, that sufficient funds are available for completion of the work and keep the sponsor's administrative audit expenses to a minimum. However, some qualified potential vendors will not submit a proposal for a fixed-price contract and competition is therefore reduced. it is particularly important that fixed-price contracts contain comprehensive, detailed statements of the requirements for the work to be performed by the contractor to avoid misunderstandings about what the agreed-upon price actually covers. These requirements should be based on a combination or merging of the specifications in the RFP and the contents of the successful vendor's final proposal.
If there are many unknown or undecided factors in the project plan that affect the predictability of costs, a cost-type contract may be preferable, provided adequate funding will be available. A "dollar cap" or maximum cost can be included in a cost-type contract. If a cap is specified, flexibility should be built into the survey design so that if avoiding cost over-runs, which jeopardize completion of the survey, is not possible, a limited but still useful product can be obtained.
If there are any serious doubts about the feasibility of key aspects of the survey plans because of unknown factors such as the ability or willingness of designated respondents to provide the information sought, it may be prudent to limit the scope Of the project to which the sponsor will be committed.
One approach which might be appropriate in this situation is to conduct a pilot study for an evaluation of the proposed methodology, with the stated intention of issuing an RFP for a full-scale survey if the initial effort indicates the methodology is feasible.
Another possible approach is to specify work segments which may be done during the course of the contract if the concerns are resolved in the earlier stages of the contract. If a phased survey is deemed advisable, early discussions with the Procurement office will indicate what contracting alternatives are available for consideration and their advantages and disadvantages.
Survey Design Specification
RFPs may include details of the survey design. The level of survey design specification found in RFPS, in practice, ranges from "This is exactly what we want, including the questionnaire to be used" to "We have a problem and need information to help solve it; what do you suggest?" The choice of the level of specification is dictated by a number of considerations such as the nature of the survey to be conducted, the applicable regulations and policies.
Detailed survey design specifications may be appropriate if the company has highly qualified and experienced statistical and survey staff available to develop the RFP who are aware of potential vendors' capabilities and resources, and who know exactly how the agency wants the survey to be conducted.
Maximal specification of the survey design will indicate the quality standards of the organization to potential vendors, and will also permit them to repeat the requirements in their proposals without demonstrating capability or understanding of the problem. It may be easier and quicker to evaluate the responses to a detailed RFP, but more difficult to determine the technical capabilities of the vendors. Selection is therefore more likely to be made primarily on price.
Less detailed RFPs which, invite vendors to make suggestions or develop survey designs, allow them to apply their knowledge and demonstrate their ingenuity or creativity. However, minimum design criteria or quality standards should be stated, when known, even if the vendors are asked to develop the survey design. Evaluating the proposals received in response to a less detailed RFP is likely to be more difficult and time consuming. When an RFP does not specify the details of the survey design, the successful vendor's explicit statement of the survey design specifications is commonly incorporated in the contract awarded.
An alternative that incorporates many of the advantages of both the detailed and less detailed survey design specification approaches is one in which potential vendors are invited to submit proposals based on detailed specifications but are also encouraged to suggest modifications to the detailed plans which they believe will be advantageous to the organization.
Contents of RFPs
When developing an RFP, it may be helpful to refer to RFPs for similar surveys conducted by the organization as sources of ideas for content and wording. However, these RFPs should be used together with comments or suggestions from persons familiar with those surveys about problems or misunderstandings which arose during the contractor selection process or the conduct of the study - particularly difficulties which might have been forestalled if the RFP had been written differently.
The goals of the procurement process should be kept in mind when an RFP is written. Ideally, an RFP does the following:
- Requests the development of or specifies a technically sound survey design which will meet the sponsor's information needs and is realistic, considering the time and funds available for the project and the capabilities and resources of potential contractors.
- Conveys all the information needed by potential contractors to prepare a responsive proposal, including all the terms, conditions, and provisions that the sponsor intends to incorporate in the contract.
- Asks for all the information that procurement and technical personnel will need to conduct an accurate and equitable evaluation of the proposals received.
- Attracts responsive proposals from enough of the qualified potential contractors to provide adequate competition and to permit a suitable selection.
Some organizations include in the RFP a table of contents for responses to be followed by all vendors and stipulate that staffing requirements should be specified in a common measure (for example, hours). This simplifies reading and comparison of proposals.
Specific suggestions for developing an RFP that meets these criteria are presented in the remainder of this chapter.
Statement of Survey Objectives
Every RFP should contain a clear statement of the specific objectives of the survey to be conducted under the ensuing contract. Discussing the proposed research in the larger context of the mission of the agency may also be advisable, particularly if the project is part of an ongoing integrated program. The statement of the survey objectives should also indicate what groups (such as departments or committees) will be the key users of the findings. If data from the survey or an interpretative report of the findings must be provided in time to meet a deadline, the date of and reason for the deadline should also be stated.
Any reference material that would be helpful to potential vendors in planning the broad outline of the proposed survey or in estimating time, skill, or cost requirements should be cited and also included in the RFP package. If the material cannot be provided with the RFP, information on where and when it can be accessed or obtained should be supplied. Appropriate reference material might include such information as estimated numbers of eligible respondents, geographic distribution of eligible respondents, and the methodology and results of any related research that has been conducted by or is known to the sponsoring agency. However, materials which would be of interest only to the successful vendor should not be included with the RFP.
Although RFPs sometimes state anticipated level of effort in terms of person-years or similar proxies for dollar estimates, the Subcommittee's interviews with contractors indicated that these proxies are less helpful to them than dollar estimates would be because of wide variations from agency to agency in the dollar amount one person-year signifies. If agency procurement policies permit, it may be desirable to inform potential vendors of the approximate amount of funds available for the contract. Providing vendors with this information would help avert some of the difficulties that might otherwise be encountered in the contractor selection stage, such as the problems that arise when the cost estimates for all of the technically sound proposals received by the organization substantially exceed the funds available for the contract. If some but not all of the technically sound proposals fall outside the anticipated price range, this reduction in competition may result in a real but less obvious disadvantage to the sponsoring agency.
Request for Information About the Vendor
The RFP should request the relevant information about the vendor. It is not sufficient for an RFP to simply define the requirements for the technical proposals. Before a contract is awarded, the reliability of the contractor should be established. When appropriate, the following specific requests should be responded to by the vendors before a contract is awarded.
Description of experiences. The following list contains areas where it could be important to obtain specific descriptions of the vendors' experiences:
- frame refinement processes,
- probability sample designs, especially for large-scale surveys,
- data collection methodology (including callback procedures),
- questionnaire development and testing,
- estimation procedures, including nonresponse adjustments,
- variance computation procedures, especially for complex samples,
- training and supervision of interviewers, coders, and others,
- analysis of data from large-scale surveys,
- data processing,
- quality control, and
Examples of previous relevant work. The examples of vendors' work should include questionnaires, sampling documentation, instruction manuals, and reports containing the results of the projects. The organization may request information on similar vendor clients.
Key project staff. The names of key project staff members should be required. Their resumes may be included with the proposal.
Subcontracting. If an vendor plans to use a subcontractor in any aspect of performance under the contract, detailed information should be provided on their experience, resources, and facilities. If subcontracting is anticipated but there is no identified subcontractor, standards for subcontractor selection should be specified.
Survey Design Checklist
A descriptive checklist which contains topics that should be addressed or considered when writing the survey design portion of an RFP. As was stated previously, the amount of specification is dependent on many factors. Ideally the amount of detail that each item in the checklist requires will depend on the type of study to be conducted. For this reason, the following list should be used only as a guide and the appropriateness of each item should be judged separately for each RFP. However, if a design topic is not specified in detail, it should be mentioned as an item to be discussed by the vendor. If an item is not mentioned in the RFP, the agency might not be able to use it as a criterion for evaluating the proposals. In addition, the importance of consulting a qualified sampling statistician before proceeding with this section cannot be overemphasized.
Design and selection
- Definition of the target population and key subgroups.
- A description of any subpopulations for which estimates are desired.
- Number (or expected number) of sampling units to be chosen.
- The target sample size, usually in terms of the number of completed questionnaires.
Data collection, questionnaire development and testing
Data specifications or information to be obtained. They should be listed in sufficient detail to indicate what the questionnaire content should be and what the amount of effort required in developing and testing the questionnaire will be. They should be complete and realistic. When the survey is covering a new topic, the sponsor should identify the variables for which data are needed, but should allow the contractor, once selected, to play a major role in developing the specific questions needed to get data on these variables.
Pretest. The need for a pretest is paramount in any survey. Without it the chances of achieving good results are minimal. The three following primary issues arise in a pretest which must be considered in any survey: sample size, probability sampling, and the survey time schedule.
Sample sizes should be sufficiently large to learn about significant problems that may be encountered. For this reason, specifying a pretest of nine respondents so that the pretest does not have to be cleared through OMB should be avoided. Furthermore, clearance at an early stage of development may identify possible overlap of the planned survey with work of other agencies or may bring to light experience in other surveys that will help to avoid repetition of past mistakes. It is better to determine the sample sizes needed without reference to clearance requirements and to ensure that the survey development schedule allows enough time for necessary clearances.
Tabulations and reports
The final products and quality of the completed work should be monitored very closely. To facilitate the monitoring process, certain deliverables should be specifically requested in the RFP. A schedule should be set up and rigidly followed. The following list contains several of the more important deliverables.
Progress reports. Progress reports should be required according to an agreed upon plan. The plan should specify content as well as time of delivery.
Draft reports for agency review. The RFP should indicate that a draft of each required task report will have to be submitted for approval before the final report is written and the task is considered complete.
Tabulations required. When it is feasible, the format and content of all required tables should be included in the proposals. The RFP should also state whether the contractor will be expected to perform analyses of the data and in what form they must be delivered.
Data descriptions. All data supplied should be accompanied by sufficient documentation. For example, the formats of any data tapes must be required and the formula used to calculate any weights must be supplied with the weights.
Survey documentation. The RFP should require a complete detailing of the methodology used. This report should include the contractor's impressions of the effectiveness of the survey procedures, suggestions for improved methods in future surveys, and all appropriate technical materials. Included would be the following: copies of all forms; documentation of all (especially sampling and data collection) procedures in all phases; all adjustments for nonresponse and missing data; survey estimates and estimates of sampling errors; full accounting of all data collection results, including response rates and interviewer validation results; coding error rates, along with all quality control procedures used; methodological findings; and a description of the procedures used in handling confidential data.
Analytic reports. If the agency wants the contractor to produce one or more reports analyzing the findings of the survey, such a report should be requested in the RFP. Frequently this analysis and the resulting report are done in- house. If this is to be the case, it should be stated in the RFP. Analytic reports should contain a complete analysis of the survey data along with any appropriate conclusions and recommendations.