AG employs formative, process and summative evaluation methods in its annual National Accreditation Process.

• Formative evaluation is employed when a state or local program is “forming” as a means of shaping the program to the JAG Model and organizational best practices. Formative evaluation is used in the establishment of the state organization and local programs.
• Process evaluation is conducted annually by JAG but on a continuous basis at the state and local level to describe and measure if the JAG Model has been properly implemented. The process evaluation focuses on the following questions:
  • Who are the students selected to receive the services provided by a JAG Model program?
  • What services are provided as well as: when, how, frequency, and in what settings?
  • What interactions are occurring between program staff and stakeholders?
  • What are the management strategies and philosophies?
  • What are the costs of a JAG Model program?
  • How satisfied are stakeholders with the program?students, employers, parents, administrators, etc.?
  • How effective is the JAG staff in implementing the JAG Model?
  • How are staff members trained?
  • What are the roles and responsibilities? Is staff satisfied with their role?
  • What are the program costs? Has there been a change in funding? What about future funding?
• Summative evaluation is conducted based upon the measurable outcomes tracked using the Electronic National Data Management System (e-NDMS). Among the most important summative data captured by student, school, site, and state, we are able to answer the following questions at the close of the 12-month follow-up period:

Program Performance Outcomes for Graduates:

• What is the graduation rate?
• What is the employment rate?
• What is the positive outcome rate?
• What is the full-time jobs rate?
• What is the full-time placement rate?
• What is the average wage paid to employed graduates?
• What is the rate of graduates attending a postsecondary institution?

Program Outcomes for Non-Seniors:

• What improvement was achieved in daily attendance?
• What improvement was achieved in grade point averages?
• What improvement was achieved in passing courses?
• What improvement was achieved in reducing suspensions?
• What improvement was achieved in enhancing their image with others in the school?
• What improvement was achieved in improving basic skills, i.e. math and reading?
• What improvement was achieved in participation in the Career Association?
• What improvement was achieved in the staying in school status?

THIRD-PARTY FEEDBACK

The primary purpose of the National Accreditation Process is to provide third-party feedback to state and local affiliates as to how well they have implemented the JAG Model and operated an effective statewide organization.

State affiliates may request third-party feedback to identify major or minor deficiencies in implementing the JAG Model. JAG views the National Accreditation Process as the first step in a program improvement initiative. State or local oversight bodies can request the following services:

1. Mid-Year Review - This review is conducted in the initial program year of a new state affiliate. The review is designed to provide feedback to the oversight body, management team, and staff without impacting the accreditation status of the state affiliate. The Mid-Year Review is usually followed by additional technical assistance to overcome deficiencies.
2. Targeted Site Review - A targeted review is for a designated site or schools. The review is usually requested by the Management Team as a means of improving performance in designated schools prior to a comprehensive site review. The site reviewer(s) will conduct an exit interview to explain the deficiencies that must be addressed before the National Accreditation Process is implemented.
3. Comprehensive Site Review - This is the traditional site review in which the state organization and a sample of local programs/schools will be reviewed in addition to a thorough review of the data contained in the Electronic National Data Management System. The Comprehensive Site Review is the initial step in the National Accreditation Process that concludes with the awarding of standard, provisional or probationary accreditation. JAG will work closely with the oversight body and/or the Management Team to deliver consulting and/or training to upgrade the state's accreditation status.
4. Management Review - The state affiliate's oversight body can request a management review to provide feedback to the State Director and/or Management Team to improve personal and programmatic performance. JAG is willing to conduct a 360-degree assessment for one or more members of the Management Team. The 360-degree feedback will be used to develop a Professional Development Plan to guide the continuous improvement of the targeted members of the Management Team.

JAG VALUES

The critical ingredients in raising JAG Model Programs to a higher level of results is essential for any school-based or publicly funded program. Being successful and staying successful is much more challenging today than ever before. The six ingredients for higher results are:

1. Genuine focus on at-risk youth - The JAG Model’s customer!
2. Data-driven management (e-NDMS.) Use data to reduce failures (defects.)
3. Core processes that are required to deliver JAG Model services and extraordinary outcomes! Continuous improvement!
4. Proactive management! Eliminate “quick fix” mentality!
5. Boundaryless collaboration! Smash barriers that block the flow of new ideas! Eliminate duplicity and waste!
6. Drive for perfection! Tolerate failure!

DATA COLLECTION METHODS

State and local affiliates are evaluated on the basis of their ability to operate an organization that is efficient and effective in implementing JAG Model Program Applications. The JAG Model components are the evaluation categories for which data is captured from the following sources:

• On-site Interviews by the Site Review Team
• Electronic National Data Management System - The Internet-based data system provides JAG the online capability to monitor the following:
  • Students selected for the program
  • Services delivered to program participants
  • Outcomes achieved by program participants during the follow-up period
• Destination Validation Telephone Surveys?a telephone survey is conducted with employers to determine their satisfaction of the services delivered by Job Specialists
• State Program Handbooks, Policy Manuals, Brochures, Videos, etc. JAG reviews these documents to evaluate the extent to which the state and local affiliates “walk the talk.” There must be alignment between program literature and the services delivered.
• Participation in National Network Professional Development Opportunities. JAG monitors those that participate in national training sessions/events. The training sessions that are delivered at national training sessions/events are based on the evaluative feedback provided by the site reviewers and the overall National Accreditation Report.

NATIONAL ACCREDITATION REPORT OUTLINE

Based on the data collected, the National Accreditation Report documents deficiencies in implementing the JAG Model. JAG is an organization in which affiliates and staff are held strictly accountable for process, services and outcomes. The report serves as an evaluation tool that includes the following:

• Overview - overall evaluation statements
• Affiliate Information - basic organizational information
• Program Demographics - who is served
• Program Outcomes - what is achieved
• Outstanding Performance and Outcomes - what exceeds the JAG Model
• Major Areas of Non-Compliance - what must be improved
• Minor Areas of Non-Compliance - what should be improved
• Model Enhancements - best practices to be shared with the National Network
• Observations and Recommendations - prescription to improve performance
• Accreditation Status - awarded based upon measurable inputs and outcomes
• Action Plan - the deadline for submitting an action plan to overcome deficiencies

The state affiliate is requested to submit an Action Plan in response to the National Accreditation Report. The Action Plan is reviewed to determine if deficiencies will be overcome, process improved, and/or outcomes improved. If the Action Plan is inadequate, JAG will schedule a face-to-face meeting with the Management Team to develop an Action Plan that will result in the intended areas of improvement. JAG monitors the implementation of the Action Plan until the performance areas are improved. A variety of services are delivered to state and local affiliates that are unable to improve their process and/or outcomes including: on-site technical assistance, visits to high performing states/locales, participation in the National Training Seminar, telephone consultation, to name a few.