Assessment and Using Data

The HSTW Assessment has been an integral tool in the school improvement efforts of High Schools That Work (HSTW) and participating states, districts and schools since its first administration in 1988. This assessment has provided comprehensive school-level data that disaggregate students' achievement by their perceptions of school and classroom experiences. These results have given schools, districts and states a unique opportunity to determine what is and is not working to increase student achievement.

SREB State Career and Technical Education Consortium, in conjunction with Educational Testing Service (ETS), have begun a process of test development to improve and refresh the HSTW Assessment to add more value for schools and participating students. The revised HSTW Assessment, in addition to continuing to serve its purpose as a measure of continuous school improvement, will also serve as a measure of student college- and workplace-readiness.

A panel of school, district and state leaders from HSTW states was convened to develop the framework for this assessment. This panel used the 2009 NAEP frameworks in reading, mathematics and science as the basis for establishing the content specifications for the new HSTW tests. In doing so, they tailored the frameworks to the curricular goals of the HSTW initiative and to the knowledge and skills they judged relevant to college- and career-readiness. They determined that the basic set of knowledge and skill critical to a successful high school outcome and to college- and career-readiness is one and the same. By focusing on informational passages and critical thinking problems, the new assessments can be an important measure of a student's chance of entering a postsecondary institution without having to take remedial courses or a workplace situation or training program with the ability to continue learning.

After this panel of leaders developed the general framework for the revised assessment, panels of content area specialists were convened to define the specific objectives that each test (reading, mathematics and science) would cover. A detailed explanation of these objectives can be found in the section of this document titled “Test Content.” Prior to the first administration of this revised assessment, a panel of employers and community college representatives will be convened to review the subject tests and help solidify this assessment as an instrument that will measure student college- and workplace-readiness. As this new assessment is rolled out, it is hoped that schools can work with local employers and community colleges to use assessment results to indicate that students are ready for postsecondary study and the workplace.

This document provides information on the revised HSTW Assessment. The next section, “Highlight of Major Changes,” gives an overview of the most important differences between the previous and revised HSTW Assessment that school leaders need to know. Subsequent sections go into more detail about the subject tests, surveys, administration and reporting methods.

This section highlights the major changes between previous HSTW Assessments and the 2008 HSTW Assessment.

• The new 2009 National Assessment of Educational Progress (NAEP) frameworks were used as a starting point for the reading, mathematics and science subject test content.
• The reading subject test focuses primarily on informational texts. It will include 80 percent informational texts and 20 percent literary nonfiction texts.
• The mathematics subject test now includes topics through Algebra II with some pre-calculus topics. Previous assessments, aligned to the previous NAEP framework, included topics through Algebra I and geometry.
• The science subject test continues to focus on life science, physical science and earth science. However, less of an emphasis has been placed on earth science since many high school seniors will not have covered this material since the middle grades.

The HSTW Assessment consists of three subject tests: reading, mathematics and science. Each test is comprised of multiple choice and constructed response questions. A single constructed response question carries more weight than a single multiple choice question. The content for each subject test is based on the new 2009 NAEP frameworks. The NAEP frameworks were modified to reflect the goals of HSTW. For one thing, unlike NAEP, the HSTW Assessment is designed to be completed in its entirety by every participating student. Also, the HSTW Assessment has its own purposes - to measure continuous school improvement, readiness for postsecondary education and readiness for the workplace. To reflect these purposes, test questions for the new assessment are not taken from NAEP but were developed specifically for the new HSTW tests under the scrutiny of content specialists who are HSTW stakeholders.

The reading test will be 90 minutes in length and will focus primarily on informational text types. Eighty percent of the texts will be informational while the remaining 20 percent will be literary nonfiction. Meaning vocabulary will be included but literary fiction and poetry will not be included.

The test specifications for passage length are 800-1500 words. The cognitive targets include 20 percent locate/recall, 45 percent integrate/interpret and 35 percent critique/evaluate. Paired texts will be included as they afford an opportunity to assess students on a critical type of analysis in reading.

The mathematics test will be 70 minutes in length and will focus on the following content areas - Number Properties and Operations; Measurement/Geometry; Data Analysis, Statistics and Probability; and Algebra. The percentage distribution of questions in these content areas will be 10, 30, 25 and 35, respectively.

The mathematics test will include topics from pre-algebra through pre-calculus. Calculators will be utilized for a portion of the assessment. Students may bring their own scientific or graphing calculators for this portion of the test. However, no questions in the calculator portion of the test will advantage or disadvantage users of particular types of calculators.

In the Number Properties and Operations content area, questions that assess students' understanding of proportional thinking, percentages and general number sense (i.e., knowing the effect of multiplying/dividing by a number less than 0, equal to 0, a number between 0 and 1, or a number greater than 1) will be included. Application problems involving rational numbers and ordering and comparing very large and very small numbers will be part of this content area.

In the Measurement/Geometry content area, questions that assess students' understanding of the effects of scaling on lengths, areas and volumes; indirect measurement; student's ability to use definitions of sine, cosine and tangent to solve problems; and understanding of radian measure will be included. This content area will include questions that involve conversions within or between measurement systems, reasoning in two- and three-dimensions, geometric concepts of congruency and similarity, properties of parallel and perpendicular lines, and applying the Pythagorean Theorem.

In the Data Analysis, Statistics, and Probability content area, questions will focus on assessing students' ability to represent data graphically, interpret data, draw conclusions from data and apply concepts of probability.

In the Algebra content area, questions that assess students' understanding of patterns, relations and functions will be included. Questions will focus on assessing students' ability to work with different algebraic representations (including manipulating algebraic expressions), translate between representations, understand the concepts of equality and inequality, and work with linear and nonlinear algebraic models.

The science test will be 70 minutes in length and will assess students' knowledge in Life Science, Physical Science, and Earth and Space Science as well as their knowledge of science practices. Test content will be divided as follows: 40 percent Life Science, 40 percent Physical Science and 20 percent Earth and Space Science.

The four science practices are identifying science principles, using science principles, using scientific inquiry and using technological design. Sixty percent of the science practice categories will be low (identifying or using science principles) and 40 percent will be high (using scientific inquiry or using technological design).

One overall goal of the science test will be to assess a student's ability to reason and think scientifically using general knowledge of matter, energy, and motion in Physical Science; structures and functions of living systems, and changes in living systems in Life Science; and earth structures and systems in Earth Science.

The student survey is being revised to eliminate questions that are confusing, not indicative of student achievement or not germane to school improvement efforts. Questions with historically low response rates have also been removed. A few additional questions have been added that will provide new insights into students' school and classroom experiences. The transcript analysis section of the survey is being reviewed to make sure courses mentioned in the survey are relevant to what is being taught in HSTW high schools. The career/technical concentrations are being reviewed and will be updated to reflect the 16 clusters and 81 pathways. Finally, based on results from analyses conducted by ETS's Fairness and Validity group, the HSTW indices have been refined to be even more predictive of student achievement.

The teacher survey is also being reviewed to eliminate questions that historically have been confusing or not germane to school improvement efforts.

Students participating in the HSTW Assessment will receive an individual student report detailing their performance on the assessment. This report will include the student's proficiency level for each subject, how far he/she have gone in completing the HSTW-recommended Curriculum and whether he/she earned the HSTW Award of Educational Achievement.

To earn the HSTW Award of Educational Achievement, students must meet all three performance goals, complete two or three parts of the HSTW-recommended Curriculum and complete a concentration in the humanities, mathematics/science or a career/technical area.

The student report will contain detailed information on the HSTW Assessment such that the student can take this report to placement officers at a community college or to an employer, and they will be able to interpret that student's performance with regard to college- and workplace-readiness.

Beginning in 2009, student reports will be mailed to schools by early May after the assessment window. For the 2008 administration, however, student reports will arrive in mid-June because Educational Testing Service must work with HSTW stakeholders to complete a standard setting process in order to develop and define the proficiency levels and performance goals. This process cannot be completed until data from the assessment administration are available and the panelists have student results to analyze. After this one-time event, however, schools will receive student reports prior to graduation.

The HSTW Assessment Report has been revised into a more user-friendly version featuring graphs and charts. The report now includes three major sections: a one-page dashboard that provides schools with a snapshot of their implementation of the reform design and of their performance on the assessment; a section summarizing the school's progress in implementing the HSTW design, performance on the HSTW indices and performance on the subject tests; and detailed tables breaking out student achievement by survey results and presenting teacher survey results.

The one-page dashboard contains information on the school's overall HSTW implementation; the percentage of students meeting the HSTW readiness goals with a comparison to the network and high-scoring sites; the percentage of students in each proficiency level for each subject test; and a placeholder for the school to fill in their AYP status, graduation rate and state assessment results.

The second section of the site report will contain more detailed information on the school's performance on the HSTW indices and subject tests. A more detailed explanation of the school's overall HSTW implementation ranking will be provided. The percentage of students experiencing a low, moderate and intensive emphasis on each of the HSTW indices will be provided along with the percentage of students meeting readiness goals within each level of emphasis. Finally, more detailed tables on mean scores, the percentage of students meeting readiness goals and the percentage of students within each proficiency level will be provided.

The final section of the report will include several tables breaking out student achievement by survey results. These tables will be similar to those in the existing HSTW Assessment Report. This section will also contain teacher survey results.

While the report will continue to be lengthy, it is designed in such a way that an overview of the entire report and the key points can be found within the first few pages, which will present information graphically. These pages can also serve as a stand-alone summary of the entire report. The reason the report remains lengthy is simply because HSTW wanted to continue providing detailed data tables to those schools that find these data tables useful.

The registration window for the 2008 HSTW Assessment is September 3, 2007, through October 31, 2007. Orders submitted after this window will be accepted based on availability and are subject to a late fee. Orders must be submitted online on the SREB Web site at http://www.sreb.org/programs/hstw/Assessment/assessindex.asp. Administration preparation materials (such as the administration guide, the test description with sample questions and sampling instructions) will be provided electronically upon submitting an order. They will also be available on the SREB Web site.

Assessment materials will be mailed to schools on December 1, 2007, to arrive at schools by December 15, 2007. This will allow all schools to receive materials before the winter holiday break. It will also allow for extra time to prepare documents for the administration, such as completing Section 1 of the Student Survey (transcript analysis). It is strongly recommended that schools complete Section 1 for the students. This guarantees more accurate transcript analysis results as well as reducing testing time.

The student assessment window is January 7, 2008, through February 1, 2008. Students should complete all four components - student survey, reading test, mathematics test and science test. Completion of the student survey is required for students to be included in the site report calculations. Completed assessments must be postmarked for return to ETS by February 1, 2008. Assessments returned after this date may not be scored, and both student and site reports may not be produced.

The administration window has been shortened by one week from what it was in previous years. This will allow ETS to score assessments and mail individual student reports to participating schools by early May. This decision is based on results from a survey sent to all principals and site coordinators at schools that participated in the 2006 HSTW Assessment; they favored the decision.

The teacher survey window is January 7, 2008, through February 15, 2008. Completed teacher surveys must be postmarked for return by February 15, 2008. Surveys returned after this date may not be scanned or included in assessment reports.

HSTW will be holding informational sessions on the revised HSTW Assessment during the 2007 Summer Conference in New Orleans, LA. This will be an opportunity for the presenter to explain the new assessment as well as an opportunity for states, districts and schools to ask questions and seek clarification. Please look for these sessions in the program.

For additional information about High Schools That Work, please contact:
Gene Bottoms
Senior Vice President
gene.bottoms@sreb.org

For additional information about the HSTW Assessment, please contact:
Allison Timberlake
Coordinator of Assessment
allison.timberlake@sreb.org