Evaluation Plan

This is an MTC-led activity, with some coordination from the PATH, as PATH is the Caltrans-sponsored evaluator of the World Congress Innovative Corridors Initiative.

Additional inputs will come from VII California stakeholders and participants, as
technical data and operational know-how will be produced from the present effort.

The evaluation will be conducted in two steps: an initial evaluation after the World Congress, and a final evaluation at the conclusion of the VII California testbed development in 2007.

Initial Evaluation

The initial evaluation will commence shortly after the World Congress and is intended to produce lessons learned, in essence, to build a better test bed. In many respects, it will come at an ideal time – after the first, quick iteration where initial installation and operation are key but before the second, more studied iteration where long-range objectives take over. The focus will be on what is learned and some of the important questions asked in the evaluation will include:

• Did technology perform as desired?
  
• What were the limitations/constraints observed?
  
• Was there VII California interoperability with the privately-led World Congress Innovative Mobility Showcase?
  
• How successful was the organizational/management structure?
  
• Are demonstration outcomes extensible to a larger-scale and longer-term test setting?

These types of questions portend technical, institutional and process improvements for the subsequent testbed and will allow VII California to expand in scope, participants, stakeholders, and quality based on valuable lessons learned for a “quick strike” initial deployment.

Final Evaluation

The final evaluation will include technical, impact, and institutional analysis, and, where appropriate, should include before and after analysis. The full evaluation of the use cases could include agency and industry partner interviews regarding institutional and technical issues, surveys, observational analysis, interviews, and focus groups. The methodology to evaluate each use case will depend on the specific attributes of the technology and the persons exposed to the technology. For all use cases, technical, impact, and institutional variables will be assessed as appropriate.

Technical: The technical evaluation should focus on the hardware and software operation of the demonstration. Did the technology perform as expected/desired? Consideration should be given to: identification of key stakeholder partners; eliciting from the partners a meaningful set of goals and objectives for the project and their relative priorities; identifying and obtaining insight and consensus regarding which measures will indicate the degree to which project success has been achieved; and communicating changes in goals, objectives, and measures as the project progresses.

Impact: Impacts on VMT, safety, mobility, mode choice, transportation system efficiency, productivity of transportation providers, air quality, energy efficiency, etc. may be examined, as appropriate. Performance measures and corresponding metrics would be identified for study. Measures could include: reduction in the overall rate of crashes; reduction in delay; improvement in customer satisfaction; increases in freeway and arterial throughput or effective capacity; decrease in emissions levels; decrease in energy consumption; and cost savings. The level of detail and the opportunity to complete before and after analysis will be determined to a large extent by the size and duration of each use case. If the demonstration size is too small it may not be possible to study some of the broader societal impacts.

Institutional: Researchers would document lessons learned regarding institutional (State and Federal) challenges (what worked and what didn’t) and make recommendations for improvements (both institutional and procedural). This evaluation would also include a broader assessment of lessons learned regarding the public-public and public-private relationships for VII deployment.

Broadly these three evaluation components (technical, impact, and institutional) should culminate in policy recommendations for the direction of further VII California deployments, and relationships among agencies and between agency and industry, including institutional challenges, recommended solutions, and next steps.