Proposed No Child Left Behind Report Card
�
with Additional State Indicators
€
Indicator
NCLB
Required
Current
Report
Card
Changes to State Report Card
Assessment Results by
proficiency level (disaggregated).
√
Disaggregated information is not currently
reported, but is available.
Assessment results compared to
Florida's annual objectives by
(disaggregated).
√
Currently, we do not have annual
objectives by subject or by student
subgroup. Objectives will need to be
determined.
Percentage of students not tested
(disaggregated).
√
Not currently reported, but available.
Assessment results compared to
the most recent 2-year trend in
each subject, for each grade.
√
Prior year comparison currently reported,
2-year trend data is available.
Results of Florida Writes
(disaggregated).
√
Disaggregated information is not currently
reported, but is available.
Graduation rates
(disaggregated).
√
√
Disaggregated information is not currently
reported, but is available. Report will
include PY Grad Rate.
AYP, including schools
designated for improvement.
√
AYP calculation will need to be
determined, collected, and reported.
The professional qualifications of
teachers disaggregated by high
poverty vs. low poverty.
√
√
Disaggregated information is not currently
reported. Determination of high poverty
vs. low poverty needs to be made.
The percentage of teachers in the
state teaching out of field
disaggregated by high and low
poverty.
√
√
Currently aggregated data is added to the
state report card at the district. Data to be
collected and reported.
The percentage of classes in the
state not taught by highly
qualified teachers disaggregated
by high and low poverty.
√
Data to be collected and reported.
Dropout Rate
√
PY Dropout Rate to be Reported.
Number of teachers and staff
new to the school
√
Results of Kindergarten
Readiness
√
October Membership
√
Teachers, administrators, and
staff who receive satisfactory
annual evaluations
√
Add to LEA Report Card
School advisory council
membership composition.
√
Add to LEA Report Card
School Lottery $ Budget
√
Add to LEA Report Card
Current State Report Card Indicators – Recommended
€
for Deletion
€
Indicator on State Report Card
Recommendation
Average number of days of students absent
Keep on the Indicators Report
Average number of days of teacher and
administrator absences
Keep on the Indicators Report
Graduates found by FETPIP
Keep in Reports produced by the Florida
Education and Training Placement
Information Program
Graduates found by FETPIP in the Military, in
post-secondary schools, and/or employed
Keep in Reports produced by the Florida
Education and Training Placement
Information Program
Occupational Completion Point Graduates found
by FETPIP compared to all OCP graduates
follow-up by FETPIP
Keep in Reports produced by the Florida
Education and Training Placement
Information Program
Occupational Completion Point Graduates found
by FETPIP in the Military, in post-secondary
schools, and/or employed.
Keep in Reports produced by the Florida
Education and Training Placement
Information Program
Number of diplomas awarded to Adults
Keep in Reports produced by the Florida
Education and Training Placement
Information Program
Students enrolled any time during the 180 day
year
Keep on the Indicators Report
Current State Report Card
The School Advisory Council Report is produced in accordance with State Rule 6A-
1.09982. Each district school board is responsible for developing and implementing
procedures for schools to use when issuing annual school reports. Each school must
distribute a school public accountability report to all parents, guardians, and adult
students and make it available to the general community upon request. Reports are due
on November 15 of each year.
NCLB Report Card
Not later than the beginning of the 2002-03 school year, school districts must
disseminate the NCLB Report Card to all schools in the school district and parents in the
reported school. To the extent practicable, the information should be made widely
available through public means such as the Internet and reported in a language that the
parents can understand.
Appendix B
Description of the Florida Comprehensive Assessment Test
(FCAT)
Section 1111 of H.R.-1 (NCLB), outlines the Congressional requirements for
academic standards, assessments, and accountability system. These
requirements will not be repeated herein but will be identified by Section numbers
throughout the description of Florida’s programs for the reader’s cross-reference.
Challenging Academic Standards (s. 1111(b)(1))
Section 1008.22(3)(a), F.S. (available at www.leg.state.fl.us/Welcome/index.cfm)
requires the Commissioner of Education to bring to the State Board of Education
sets of skills and competencies that will guide instruction in all of the public
schools. The specific requirement is stated as follows:
Submit to the State Board of Education a list that specifies student
skills and competencies to which the goals for education specified in
the state plan apply, including, but not limited to, reading, writing,
science, and mathematics. The skills and competencies must include
problem-solving and higher-order skills as appropriate and shall be
known as the Sunshine State Standards as defined in Section
1000.21, F.S. The commissioner shall select such skills and
competencies after receiving recommendations from educators,
citizens, and members of the business community. The commissioner
shall submit to the State Board of Education revisions to the list of
student skills and competencies in order to maintain continuous
progress toward improvements in student proficiency.
The development of Florida’s content standards began with creation of
curriculum frameworks as a resource and a guide for school districts. The
frameworks included the Sunshine State Standards (Standards) that specify the
challenging content expected of Florida students.
The development of Florida’s Sunshine State Standards was discussed in the
Title I Plan for 2001-02 and will not be repeated herein. Specific information
about the manner in which each set of standards was created is available at the
following Department of Education web site:
http://www.firn.edu/doe/curric/prek12/frame2.htm. In summary, the Standards
were developed with the involvement of practicing educators from across Florida,
reviewed by various interested parties, including the Mid-Continent Regional
Educational Laboratory (McREL), reviewed by all school districts, and adopted by
the State Board of Education in 1996.
Grade Level Expectations
The original design of the Standards did not include grade-by-grade expectations
for all grade levels. As decisions were made to expand the statewide
assessment program to include all grades 3-10 (see following discussion), it
became necessary to create “grade level expectations.” These are described at
length on the Department’s web site at www.firn.edu/doe/menu/sss.htm and will
not be repeated here.
Evaluation and Review of the Sunshine State Standards
In addition to the review of the emerging standards by McREL as previously
described, the Sunshine State Standards have been reviewed by the American
Federation of Teachers (AFT). Their commentary can be found at
http://www.aft.org/edissues/standards99/states/Florida.htm. AFT concludes that
Florida’s Standards are “clear, specific, and grounded in content.” Here are
selected statements descriptive of the mathematics, language arts, and science
Standards:
Language Arts
“The
English
standards are clear across all three levels, and the content
at the elementary and middle levels is strengthened by the addition of the
new
Expectations.
… In addition, the
Florida Writes!
assessment booklets
clarify the writing forms at all three levels and include examples of student
work that illustrate the quality and complexity of writing expected of
students at each of the levels.
Mathematics
“With the addition of the new
Grade Level Expectations,
the elementary
and middle level
mathematics
standards are quite clear and specific. …
At the high school level the standards are generally clear and specific, but
at times, they are broad.”
Science
“The
science
standards are also clear, specific, and grounded in content.
… The
Expectations
help clarify the standards for grades K-8 and illustrate
how the standards might look in a classroom.”
Education Week also conducted a review of the standards and accountability
programs across the 50 states. Florida’s program was rated “A” in the special
publication
Quality Counts
that can be seen on the web at
http://www.edweek.org/sreports/qc03/rc/rcard_frameset.htm.
The Department of Education commissioned a special mid-term review of the
Sunshine State Standards to be conducted by the Suncoast Area Center for
Educational Enhancement at the University of South Florida. The results of this
activity will be used for a more comprehensive review of the Standards over the
next three years.
The Standards represent what all students should know and be able to do as
designated by the State Board of Education. The adoption of the Standards sets
policy direction for instruction in Florida’s schools. However, the Standards do
not limit schools or school districts in what should be taught.
Local units are
completely free to supplement the instructional program with content and
objectives not included in the Standards.
Academic Assessments (Section 1111(b)(3))
Florida has had an organized statewide assessment program for more than 30
years. (A chronology of the statewide assessment programs can be found at
http://www.firn.edu/doe/sas/hsaphome.htm.) As understanding of the value of
data has increased and as testing and computer systems have developed,
especially in the last 20 years, the Florida statewide assessment program,
management information system program, and school/district accountability
program have grown and become more sophisticated. The activities during the
school year 2002-03 continue this steady evolution and improvement process.
As early as 1973, Florida showed an understanding of the importance of
measuring student achievement, measuring other educational indicators, and
reporting to the public. In 1971, Governor Reubin Askew appointed a Citizens’
Committee on Education to study education and recommend ways to improve
our schools. The report of this committee,
Improving Education in Florida
,
included such concepts as citizen participation in the educational process, public
reporting of information, state- and district-level assessment programs, and
participation in the fledgling National Assessment of Educational Progress (The
Governor’s Citizens’ Committee on Education, Tallahassee, FL, March 15, 1973).
In 1976, the Florida Legislature enacted an Educational Accountability Act that
expanded the statewide assessment program to grades 3, 5, 8, and 11 and
introduced the nation’s first requirement that students pass a high school
competency test to qualify for a regular diploma. The State developed and
implemented these tests and subsequently faced two legal challenges. The high
school competency test was challenged in
Debra P. v. Turlington
(474 F. Supp.
244 , MD Fla. 1979), and the basic skills tests were challenged in
Love v.
Turlington
(1980). The State position prevailed in both situations with the courts
ruling that competency tests can be required although the State must assure that
the students are afforded due process. These landmark rulings established the
precedent for court rulings in other states, including, most recently, a challenge to
the TAAS system in Texas (
GI Forum et al v. TEA et al,
2000).
The Legislature continued to modify and improve the statewide assessment
requirements during the 1980s and even extended the concepts to the
postsecondary level with creation of certain statewide testing requirements for
community college and university students. However, it became apparent that
the emphasis on minimum competencies for all students had its limitations.
Average and above average students were not being challenged, and the
general focus on “minimums” was not producing graduates who could perform in
today’s employment marketplace.
In the early 1990’s, the Legislature revised the structure of the state assessment
program and created the Florida Commission on Education Reform and
Accountability. Their work contributed to discussions about the importance of
challenging educational standards and the need to move away from minimum
competency tests and the traditional reliance on multiple-choice test questions.
Coming from the work of the Commission and the Department, an overall
assessment plan was adopted for 1996-97 and revised for 1997-98.
In the mid-1990s, the State moved rapidly toward creation and adoption of the
Sunshine State Standards, as previously described. The Standards and the
associated curriculum frameworks defined challenging content in seven subject
areas. In 1995, a request for proposals was issued for a new, expanded
statewide assessment program. Contracts for development of the new
assessment were issued in mid-1996, and the State began the creation of tests
that would look much different than the older minimum competency tests.
By 1992, the statewide assessment program developed and implemented a
writing assessment program in grades 4, 8, and 10. In 1999, under Governor
Jeb Bush, the program was expanded to all grades 3 – 10. The new assessment
program would contain both criterion-referenced tests measuring state content
and nationally norm-referenced tests. Performance items were to be included to
the extent that was practical. Student, school, district, and statewide results
would be reported and used as the basis for a school accountability program.
See s. 1008.22, F.S., available at http://www.leg.state.fl.us/Welcome/index.cfm.
Under the terms of the new statute, the existing High School Competency Test
(HSCT), a minimum competency test required for graduation, would be phased
out and students graduating in 2003 would have to earn a passing score on the
new grade 10 Florida Comprehensive Assessment Test (FCAT) to graduate.
Students who were enrolled in 9th grade in the Fall of 1999 thus were given
advance notification of their graduation requirement vis-à-vis the state test.
The new testing structure is shown in Figure 1 below. Reading and mathematics
are tested with both criterion-referenced and norm-referenced tests at eight
grade levels, while writing and science are measured at three grade levels each.
This combination permits the achievement of students to be measured against
two different dimensions – the State’s own challenging content as well as
national norms. Although Figure 1 does not show it, Florida also participates in
the National Assessment of Educational Progress (NAEP) at both the national
and state levels. (Due to a scheduling conflict, Florida did not participate in the
2000 state-NAEP but did participate in the Spring 2002 testing and will continue
to participate in the future as required by
No Child Left Behind
(
NCLB
).
The Florida Comprehensive Assessment Test exceeds the minimum
requirements of NCLB that only requires student assessment in reading and
mathematics once in grades 3-5, 6-9, and 10-12. Because the FCAT measures
student achievement in reading and mathematics in all grades 3-10 and uses a
coordinated vertical score scale, Florida is able to track student achievement
over time from one grade level to another. This generates powerful information
with which to monitor progress as is required by NCLB.
The writing assessment component and the High School Competency Test
component have been in existence for many years. Each of these tests was
developed through the efforts of commercial contractors and school district
curriculum content committees to both develop and validate items. For additional
information, the reader may refer to the Department of Education’s web site at
www.firn.edu/doe/sas/sasshome.htm, which contains a description of these
programs as well as a chronology of the development of the state assessment
program from 1976 to the present.
Through the 1995 competitive bid previously mentioned, the Department
contracted with CTB/McGraw-Hill (CTB) for the development and implementation
of the FCAT tests in grades 4 (reading), 5 (mathematics), 8 (reading and
mathematics), and 10 (reading and mathematics). Initial development took place
from May 1996 through the census field test in March 1997. The first full-scale
census assessment occurred in February 1998 followed by the second
administration in February 1999.
The test development process began with selection of those portions of the
Standards that would be measured. Since the Standards themselves are very
broad goals and were not specific enough to define the assessment system,
benchmarks within the Standards were selected for this purpose. The test
blueprint was created to show how many items would be needed for the test and
how they would be distributed. In general, about 20% of the items would be
either short- or extended-response items. In mathematics, gridded response
items also were to be used.
Item specifications were drafted by writers from CTB, reviewed by Department
staff, and validated by committees of practicing Florida classroom teachers and
curriculum supervisors. The specifications can be seen at the Department’s web
site at http://www.firn.edu/doe/sas/fcat/fcatis01.htm . Items and performance
exercises were drafted by CTB writers, reviewed by Department staff, and, again,
validated by committees of Florida educators. Pilot tests of the items and
exercises were conducted with small groups of students, not to gather statistical
information, but, instead, to see whether students understood the directions and
the item content. Pilot test participants were interviewed to gather feedback
information. A community sensitivity review committee and an item bias review
committee also reviewed all of the items prior to their use.
CTB psychometricians worked cooperatively with Department assessment staff
to select the measurement model that would be used. In this case, since the
performance items were being merged with the multiple-choice items, it was
decided to use both 2- and 3-parameter item response theory techniques to
analyze the data, create the score scale, and equate the tests from year to year
and horizontally across operational forms.
The field test of the test items and performance exercises was conducted in 1997
with students in all schools across Florida participating. An item sampling
methodology was used, so not all students took all items although all students at
the assessed grade levels were tested. Exceptional education students (those
with disabilities and those who are gifted) and Limited English Proficient (LEP)
students were included in the field test. The test items were calibrated, and
items were selected for the 1998 operational tests.
In 1998, the tests were administered and the results were reported. Since
achievement levels (i.e., performance standards) had not yet been adopted,
student performance was reported in terms of scale scores and content
subscores. In addition, student performance was displayed in terms of whether
the score was within the lower, middle, or upper third of Florida examinees.
graduation requirement. The FCAT system also incorporates a national norm-
referenced test in reading comprehension and mathematics problem-solving.
Higher-Order Thinking Skills
To understand how the new FCAT measures higher order thinking skills and
understanding, reviewers should inspect the Sunshine State Standards, the
benchmarks being measured, the item specifications, and the sample exercises.
One will immediately see that students are not being asked simple, one-step,
minimal skills items. They must think, analyze, and explain, answering questions
that require original thought and multiple steps, cast in a framework that crosses
all subject areas. In other words, the stimulus material in the mathematics test or
the reading test can come from any appropriate material from any content area
(e.g., science). Reading skills and mathematics skills are thereby applied in
other content domains.
Comprehensive Writing Assessment
The Department has initiated steps to revise the current writing assessment so it
will be more comprehensive. In addition to the existing single essay prompt, the
revised test would include machine scoreable items measuring editing, language
mechanics, and other writing skills. By including new content and additional
items, the content will be broadened, and it will be possible to more closely
equate each year’s test form to that of the preceding year. Current plans call for
the new writing test to be implemented in the spring of 2005.
Alignment of Assessment and Standards
The FCAT is not an off-the-shelf test product; it has been built to Florida’s
content standards and expectations from the first day.
In all cases, the FCAT items and performance exercises are written to match the
Department’s approved item specifications, which match the designated
benchmarks. This linkage has been built into the system and is verified at every
stage of the test development process. Both the specifications validation
committee and the item validation committees reviewed the given materials in
terms of the degree of match to the benchmarks.
The FCAT is developed with the assistance of subject area committees of Florida
educators who teach or supervise mathematics, reading, writing, and science.
These practicing classroom teachers and curriculum supervisors assist in
approving the overall test design, the benchmarks to be assessed, the test
specifications, and the test items themselves. Their work guarantees that the
tests are aligned with the Standards. It also guarantees that consideration is
given to the measurement of content areas not currently included in the test so
that changes can be made in future editions of the tests.
The task of alignment is built into the test development system rather than
determined by some outside source. In effect, since the initial materials are
developed by the test contractor, the Florida-based committees are the outside
reviewers and validators.
The FCAT is not a basic skills test, and the items include a range of difficulty.
The test measures more complex skills and requires the students to think, solve,
and explain.
Information about the Sunshine State Standards, the test specifications, sample
items, and the FCAT are available on the Internet and through various printed
publications. (See the assessment program’s web site at
www.firn.edu/doe/sas/sasshome.htm.) The FCAT item specifications are public
documents and are disseminated to all school districts for their use.
No Child Left Behind
requires that the statewide assessment program:
•
Specify what children are expected to know and be able to do;
•
Contain coherent and rigorous content; and
•
Encourage the teaching of advanced skills.
The Florida system meets each of these three criteria.
Individual Student Assessment Reports
The FCAT program is a census-based assessment, although sampling
procedures are used for some statistical analyses and for field-testing new items.
This design provides complete data reports for each student, school, and district.
A publication titled Understanding FCAT Reports, 2002 includes a description of
various report forms. The FCAT Individual Student Report provides the usual
identification information about the student and then describes the student’s
performance. The data show the student’s total score for each subject area,
compare the student against the established performance criteria, compare the
student against statewide averages, show how the student performed in each
subcontent area, and provide a measure of growth over a two-year period.
Beginning with the 2000 assessment, a separate report included how well the
student performed on the national norm-referenced component (the SAT-9). The
reverse side of the individual reports includes descriptive information for the
parent written in English and Spanish. This document is available upon request.
Disaggregated Reports
Florida’s student assessment program has a long history of providing
disaggregated reports of student data. The current FCAT provides a variety of
reports of data for subpopulations.
NCLB
requires that assessment results be provided by school district, school,
racial and ethnic group, English proficiency status, migrant status, gender,
students with disabilities compared to non-disabled students, and economically
disadvantaged compared to those not economically disadvantaged. Florida is
committed to reporting these categories of data. However, it is not possible to
produce them in the initial reports of FCAT data because several data files must
be merged to generate some of the reports (e.g., economically disadvantaged).
It, therefore, may be necessary to create the reports after the initial releases of
data.
All required reporting subgroups and reporting specifications will be provided.
Disaggregated reports are generated for ALL schools, not just Title I schools.
Disaggregated data reports from the FCAT are provided directly to each school
district as well as administrators within the Department of Education. Each
school district provides further dissemination to school administrators, teachers
and parents as appropriate. State and district school improvement personnel
regularly utilize disaggregated data for planning and achievement monitoring
purposes. To promote the appropriate use of disaggregated data for monitoring
progress and to aid strategic planning for school improvement, regional
workshops were held to train district testing, evaluation, and school improvement
staff how to read, interpret, and use the data reports.
Technical Quality of the FCAT
The FCAT is designed to be reliable, valid, and free from bias. Considerable
effort was devoted to and is being devoted to the technical quality of the
assessment program.
Validity has many dimensions, but in its most fundamental sense, a test is not
said to be valid but, instead, one speaks about whether an interpretation of an
examinee’s score is valid. This is, perhaps, a subtle distinction and not one that
the average consumer clearly understands. This is why many people ask, “Is
this test valid?”
Validity is not a single judgment but is a conclusion reached by looking at
different pieces of evidence. At the same time, the developing agency, in this
case the Florida Department of Education, bears a responsibility for stating the
intent of the test and how the scores are to be used. The FCAT was designed to
be used to measure whether or not students have demonstrated skills proficiency
to meet the State’s academic standards. This does not preclude use of FCAT
scores in some other ways such as to predict a future performance, but any such
uses would have to be individually validated, as they were not part of the original
test design.
With the above principle in mind, the development of FCAT is founded on content
validity as indicated by the match between the test and the benchmarks the items
purport to measure. In other words, the question is, “Do the items match the
content that the State desires to measure?”
The content validity of FCAT was built into the developmental process from the
very first steps. The item specifications were created by the test development
contractor and reviewed, revised, and validated by committees of Florida
classroom teachers and curriculum specialists. The overall test blueprint was
likewise reviewed, revised, and validated by subject area content committees in
Florida. During these reviews, the materials provided by the contractors were
and are heavily edited; items are rejected or modified to make certain they meet
the test item specifications. Reviewers use worksheets that track their
acceptance or rejection of each item.
All items are pilot-tested on small groups of students and the students are
interviewed after each sitting. This permits the test administrator to learn first-
hand what difficulties the examinee has with the instructions, the items, or the
materials. All test items are field-tested with large random samples of Florida
students, accomplished by administering statewide field tests or by imbedding
items within operational forms. All items in such field tests are subjected to
statistical item analyses and further review by staff of the Department and the
contractor. Such analyses routinely include 2- and 3-parameter IRT approaches,
calculation of classic psychometric indices, dimensionality studies, and DIF
analyses. Content validity thus is established by a thorough and professional
quality control process.
As the FCAT was initially developed in grades 4, 5, 8, and 10, data were
gathered to compare student performance on the test with their performance on
district norm-referenced tests and grades earned in courses. The data collected
in 1998 from a sample of districts revealed a reasonably strong correlation
between the Stanford 8 and FCAT scores. In 2000, the FCAT included use of
the Stanford 9 (SAT-9) and, thus, the comparisons could be done by the
Department as soon as the results were available. These studies generally show
that the FCAT-SSS and the FCAT-NRT are correlated at about the 0.83 – 0.85
level.
Comparing grades earned in courses is more problematic since teachers are
known to assign grades for reasons other than academic proficiency.
Furthermore, students have many different course selection possibilities and a
simple correlation between FCAT and grade point average is often indistinct.
The Department conducted a few studies of the relationship between grades and
FCAT scores. Generally, the results show that low scores on the FCAT are
associated with poor performance in courses and vice versa.
For grade 10 students in Florida, there is an additional source of interesting data.
These students have the option of taking the entering freshman college
placement test. The State offers this test to high school students with the intent
of inspiring all students to aim toward postsecondary education and to select
courses that will prepare them for college work. Scores of over 10,000 high
school students who took this test have been collected, and correlation and
predictive studies of the data have been conducted.
Florida also is creating a “value-added” accountability system that will track
students over time as they move through the educational system.
Another important dimension of interest is that of “instructional validity,” the
degree to which the content measured by the test is being taught. The State is
obligated to consider instructional validity as one important dimension of the
provision of due process to students. (See the findings of the
Debra P. v.
Turlington
case.)
Florida districts and schools have been on notice for many years about the
development and, later, the adoption of the Sunshine State Standards. The
Standards were adopted by the State Board of Education after public hearings
and much discussion and review. Various memoranda were sent to district
superintendents and other educators about the importance of teaching the
content defined by the Standards. Publications and other informational and
educational tools have been developed to assist districts in adopting the
Standards.
The State conducted an instructional validity study in the 2000-01 school year to
guarantee that all students are having the opportunity to learn the desired
content. This is particularly important because the 10
th
grade students of 2000-
01 will be the first class to be required to pass the FCAT for graduation. The
results showed that Florida school districts have implemented the Sunshine State
Standards into the instructional program.
The consequential aspects of validity require long-term review and consideration.
Certain impressions are available at this time. First, the test results are
improving over time. This may be interpreted as schools spending more time
emphasizing the benchmarks being measured by FCAT. Since the test is secure
and is not released and since the tested content is broad, rather than narrow as
in older minimum competency days, it may be reasonably concluded that
students are making good progress toward the challenging standards adopted by
the State Board of Education.
Second, comments from instructional leaders and
supervisors across the state articulate their beliefs that the FCAT, with its reach
into higher content and its use of performance items, is moving the instructional
program in the directions they desire. Third, there are always those who
complain that the state assessment tests are unnecessary and are an
undesirable intrusion into the daily classroom life. Teachers object that they
have to “cease their normal activities and teach FCAT.” The response to this is
that “FCAT-prep” activities are not needed and are not desirable since the
Sunshine State Standards are to be woven into the curriculum and instructional
program in a seamless manner.
In summary, based on immediate information, the program appears to be
working as designed and results are being obtained. Longer-term information
will be needed for clarification of other dimensions of impact.
Reliability of the FCAT is also a matter of psychometric concern and interest.
The technical data describing the FCAT are shown in the
2000 FCAT Technical
Report
. This includes a description of the standard errors of measurement and
the rater consistency for the performance items. In regard to the latter, while the
performance scoring is being conducted, the Department receives daily statistical
reports showing rater consistency for each performance item. Other data show
the consistency of each individual rater. If a rater is not performing up to the
established standards, the individual is retrained or discharged.
Florida places great emphasis on good test administration procedures, test
security, and ethical behavior of students and test administrators. Readers
should pay particular attention to the FCAT Test Administrator’s Manual,
provided as an Attachment, which includes copies of test security statutes and
administrative rules. It is expected and demanded that Department staff,
contractor staff, advisory committees, content committees, and district educators
follow instructions relative to maintenance of test security. Procedures are in
place to investigate any allegation of a breach in test security. This includes
criminal prosecution and referrals to the Professional Practices Commission for
action against the professional license. In addition, a paper on ethical behavior
in the administration of assessment tests has been prepared and given wide
circulation. See the Department’s web site at
http://www.firn.edu/doe/sas/pdf/ethics.pdf.
The Department’s Office of Assessment and School Accountability convenes a
committee of district test administrators each year to debrief following each
annual assessment cycle. The Section Administrator personally tours the State
after the spring assessment administration and makes from eight to ten
presentations about the score reports. At the same time, feedback is gathered
from workshop participants concerning the program and problems occurring
during test administration. Each test administrator with every school completes a
feedback form to describe any difficulties with test administration and
suggestions for improvement. This feedback is analyzed by the test support
contractor and given to the Department.
In addition, the Office of Assessment and School Accountability statistical
analysis staff provides intense quality control over the processing of test answer
sheets by the test support contractor. There is an independent audit of each step
of the contractor’s work and approximately 40 separate computer programs are
run against the various computer files provided by the contractor to identify errors
of various types. Equating and calibration analyses are separately run by two
and sometimes three groups to triangulate results and confirm accuracy. No test
results can be processed by the contractor until the quality control staff agrees
that there are absolutely no errors present. This process requires about ten
calendar days to complete. In addition, staff members are present at each site
where performance items are being scored. They monitor activities, provide
guidance, participate in training, and solve problems as they occur. This requires
that each person be on-site from four to six weeks in out-of-state locations.
The State has a formal and regular operation to maintain a high quality
assessment program through analyses and input from various external sources.
There are technical advisory committees, curriculum content committees, and
external
ad hoc
committees of advisors. These groups either are convened to
review and update the assessment instruments or they are convened to solve a
particular problem or critique some aspect of the assessment. The test support
and development contracts include thousands of dollars in resource money to
provide travel and consultant fees where needed. Research projects are
routinely commissioned with state universities to explore issues related to
scaling, dimensionality, IRT questions, plans for vertical scaling, etc. The
program and its tests are reviewed and evaluated every year and are in a
constant process of improvement.
We believe that Florida’s extensive quality control process excels among the
various state assessment programs.
Additional explanatory information about the FCAT program including a
chronology of development, sample items, test specifications, and other
documents can be found on the web at the following address:
http://www.firn.edu/doe/sas/fcat.htm.
National Assessment of Educational Progress
Florida has participated in the state-level administrations of the NAEP tests, with
one exception when the first version of FCAT was being implemented. The state
will participate in the state-level NAEP in the future, as required by
NCLB
.
Challenging Student Academic Achievement Standards
No Child Left Behind
requires all states to adopt challenging academic
achievement standards for the tests in mathematics, reading/language arts, and
science. This has been accomplished in Florida for the mathematics and
reading/language arts assessments, although, in Florida, such standards are
referred to as FCAT Achievement Levels. Their development and current status
is described in the section below.
The Department of Education created definitions for five Achievement Levels that
would be the basis for describing student performance on the FCAT. The
definitions are shown below in Figure 2.
Figure 2
Definitions of the FCAT Achievement Levels
Level
5:
Performance at this level indicates that the student has success
with the most challenging content of the Sunshine State Standards.
A Level 5 student answers most of the test questions correctly,
including the most challenging questions.
Level
4:
Performance at this level indicates that the student has success
with the challenging content of the Sunshine State Standards. A
Level 4 student answers most of the questions correctly but may
have only some success with questions that reflect the most
challenging content.
Level
3:
Performance at this level indicates that the student has partial
success with the challenging content of the Sunshine State
Standards but performance is inconsistent. A Level 3 student
answers many of the questions correctly but is generally less
successful with questions that are most challenging.
Level
2:
Performance at this level indicates that the student has limited
success with the challenging content of the Sunshine State
Standards.
Level
1: Performance at this level indicates that the student has little success
with the challenging content of the Sunshine State Standards.
To operationalize the five definitions, it was necessary to select performance
standards or “cut-scores” for each level and have them adopted by the State
Board of Education as administrative rule. The Department has, at this time,
engaged in two separate standard-setting operations.
The first standard setting exercise was implemented for the initial tests in reading
(grades 4, 8, and 10) and mathematics (grades 5, 8, and 11). In the fall of 1998,
a statewide committee of practicing teachers and curriculum leaders was
designated for the purpose of advising the State on the selection of achievement
levels (i.e., performance standards). This committee of about 80 people was
divided into elementary, middle, and senior high working groups. They were
convened at a location near Tampa, Florida, for a four-day working session.
The participants engaged in a five-step process built around the “bookmark”
standard-setting procedure suggested by CTB/McGraw-Hill, the first FCAT
development contractor. Participants were given workbooks containing over 100
items that represented the range of difficulty of FCAT items. At the earliest
stage, each person reviewed the items and selected the location where a
“bookmark” or standard was to be defined. At each subsequent stage, the
participant was provided more information and opportunity for discussion. Five
votes were taken before the conclusion of the meeting.
Department staff then took the proposals for achievement levels to other groups
for review. Three committees were convened – one of business leaders, one of
citizens, and one of educators other than classroom teachers and curriculum
specialists. Further reviews were conducted within the agency, and in December
1998, the State Board of Education adopted cut-scores for an initial stage
followed by a higher, second stage.
In the fall of 2001, it was necessary for the Department to initiate development of
recommendations for Achievement Levels for the new grades in 3-10 that had
been added to the assessment program. With the assistance of the new test
development contractor, Harcourt Educational Measurement, the Department
again implemented a process involving committees of teachers, curriculum
leaders, business leaders, parents, and citizens. The “book mark procedure”
was used as in 1998; however, it was possible to add a new dimension to the
standard-setting procedure since the Department had successfully completed a
vertical scaling analysis for the assessment tests. Since the 100-500 FCAT
scale could be converted to a continuous developmental scale spanning grades
3-10, it was possible for corrections to be made in the specification of the cut-
scores to smooth out the Achievement Levels and make them more consistent
across the grades.
In December 2001, the State Board of Education considered the issue of passing
standards for the new grade levels. The Board adopted the recommended cut-
scores and, in addition, decided to extend to 2004 the date at which Stage Two
would become effective. Tables 1-2 display the FCAT Achievement Levels
adopted by the Board.
Table 2
€
FCAT Mathematics, Grades 3-10
€
Stage 1 for tests administered in 1999-2003
Grade
Level 1
Level 2
Level 3
Level 4
Level 5
3
100-252 253-293 294-345 346-397 398-500
4
100-259 260-297 298-346 347-393 394-500
5
100-287
288-325
326-354
355-394
395-500
6
100-282 283-314 315-353 354-390 391-500
7
100-274 275-305 306-343 344-378 379-500
8
100-279
280-309
310-346
347-370
371-500
9
100-260 261-295 296-331 332-366 367-500
10
100-286
287-314
315-339
340-374
375-500
Stage 2 for tests administered in 2004 and beyond
Grade
Level 1
Level 2
Level 3
Level 4
Level 5
3
100-265 266-306 307-358 359-410
411-500
4
100-272 273-310 311-359 360-406
407-500
5
100-300 301-338 339-367 368-407
408-500
6
100-295 296-327 328-366 367-403
404-500
7
100-287 288-318 319-356 357-391
392-500
8
100-292 293-322 323-359 360-383
384-500
9
100-273 274-308 309-344 345-379
380-500
10
100-299 300-327 328-352 353-387
388-500
For the writing assessment, the State Board of Education has not officially
adopted a “cut-score” since there is no high-stakes decision required of this test.
However, the Department considers a student score of “3” on the scale of 1-6 as
being the lowest acceptable score. This is driven by the definition of the scoring
rubric itself as can be seen at the Department’s web site:
http://www.firn.edu/doe/sas/fw/fwaprubr.htm.
The State Board of Education has not yet considered the issue of performance
standards for the science assessment since the test was not administered on a
statewide basis until 2002. The Department expects to undertake standard-
setting operations in August 2003.
The grade 10 FCAT passing scores adopted by the State Board of Education in
December 2001 specified two levels. For students tested in March and October
2001, the passing scores would be 287 in reading and 295 in mathematics.
Beginning in March 2002, all 10
th
graders who are initially taking the FCAT for
graduation will be required to earn scores of 300 in reading and mathematics.
The Commissioner of Education is required to review the passing score levels in
mid-2002 and determine whether to make further adjustments in the passing
scores. This step is necessary because the objective of the overall accountability
program is to keep moving the educational system forward. Making regular
adjustments in the required passing scores (or definitions of the FCAT
Achievement Levels) is viewed as essential.
No Child left Behind
Academic Standards
As stated in Section 1111(b)(1)(D) of
NCLB
, each state Title I program is
required to include challenging academic standards with at least three levels of
proficiency –Advanced, Proficient, and Basic. States may have more than three
levels, but must define the levels and explain their relationship to the levels
required for Title I purposes.
Florida deliberately did not use value-laden words to describe its achievement
levels because of the lessons learned in the
Debra P. v. Turlington
case. The
original high school competency test was called a “functional literacy test,” and
so it was easy for someone to mistakenly assume that a failing score labeled a
student as being “functionally illiterate.” It has, therefore, been decided to use
only numbers to identify the five different levels describing performance on
FCAT.
The National Assessment of Educational Progress (NAEP) utilizes the labels of
“Advanced,” “Proficient,” and “Basic.” There has been considerable discussion
about these labels and whether another category of “Below Basic” should be
added. Table 3 below shows the percentage of students attaining various
achievement levels for grades 4 and 8 in reading and mathematics (
NAEP 1998
Reading State Report for Florida
, 1999;
NAEP 1996 Mathematics, Cross-State
Data Compendium for the Grade 4 and Grade 8 Assessment, 1998
). This is
followed by Table 4 that shows the percentage of students in each of the five
achievement levels of FCAT from the 2002 administration.
Limited English Proficient Students
In Florida, there are 215,777 students classified as limited English proficient and
being served. These students speak 207 different languages and come from 257
different countries. The four largest language groups are Spanish, Haitian-
Creole, French and Portuguese. See 2000-2001 ESOL Annual Report, State
Synopsis for a listing of all native languages represented in Florida’s K-12
educational system in 2000-01, available via the Internet at
<http://www.firn.edu/doe/bin00011/0001esol/files/0001ESOLState.pdf>.
As described in Principle 5.4, all LEP students are to be assessed. An LEP
student may be exempted only when he/she has been receiving services in a
program operated in accordance with an approved district LEP plan one year or
less and a majority decision is made by an LEP committee, on an individual
student basis, to exempt him/her.
In this context, the term “LEP committee” is defined in Rule 6A-6.0902, F.A.C., as
meaning:
“...LEP Committee means a group composed of ESOL teachers and home
language teachers, and an administrator or designee plus guidance
counselors, social workers, school psychologists or other educators as
appropriate for the situation.
The parent(s) would also be invited to attend
any committee meetings.”
The LEP committee, in making its decision, shall consider the following factors:
(1) level of mastery of basic competencies and skills in English and home
language according to appropriate local, state and national criterion-referenced
standards; (2) grades from the current or previous years; or (3) other test results.
(See Rule 6A-1.09432, FAC, available via the Internet at
<http://www.firn.edu/doe/rules/6a-1-8.htm#6A-1.09432.>)
Rule 6A-6.09091, F.A.C., Accommodations of the Statewide Assessment
Program Instruments and Procedures for Limited English Proficient Students,
ensures accommodations in the administration of the FCAT to LEP students.
(See Rule 6A-6.09091, FAC, available via the Internet at
<http://www.firn.edu/doe/rules/6a-69.htm#6A-6.0901>.) The FCAT
Administration Manual contains the complete description of the accommodations
for LEP students. Following are selected pages from the 2003 FCAT
Administration Manual.
For school year 2002-2003, assessment results for all LEP students will be
collected, analyzed and reported. The scores of LEP students participating in
FCAT and those assessed by other methods shall be used in the calculation of
AYP. The scores for LEP students who did not participate in the FCAT will be
collected and reported by the number scoring proficient and those not scoring
proficient.
A comprehensive plan has been developed for both assessment of academic
progress and English language proficiency. The June 2002 FDOE submission of
the NCLB Consolidated Application describes in detail the process and reporting
of results for English language proficiency and for academic achievement of all
LEP students.
READING, MATHEMATICS, AND SCIENCE
€
TEST ADMINISTRATION MANUAL
€
MARCH 2003
€
FLORIDA DEPARTMENT OF EDUCATION
www.myfloridaeducation.com
30
€
TEST ACCOMMODATIONS FOR LIMITED ENGLISH PROFICIENT (LEP)
STUDENTS
Districts are required to offer accommodations to LEP students who are currently receiving services in a
program operated in accordance with an approved District LEP Plan. Permissible modifications for these
LEP students are listed below. The test may be administered with any one of these modifications or a
combination of accommodations that are determined to be appropriate for the particular needs of the LEP
student. However, all testing, with or without accommodations, must be completed during the prescribed
testing dates shown on the inside front cover of this manual.
Flexible Setting. LEP students may be offered the opportunity to be tested in a separate room with the
English for Speakers of Other Languages (ESOL) or heritage language teacher acting as test administrator.
Parents must be informed of this option for students not of legal age and shall be given the opportunity to
select the preferred method of test administration.
Flexible Scheduling. LEP students may take a part or session of the test during several brief periods within
one school day; however, a session of the test must be completed within one school day.
Flexible Timing. LEP students may be provided additional time; however, a session must be completed
within one school day.
Assistance in the Heritage Language. For the mathematics and science tests, LEP students may be provided
limited assistance by an ESOL or heritage language teacher using the student’s heritage language. The
teacher may answer specific questions about a word or phrase that is confusing the student because of
limited English proficiency, but is prohibited from giving assistance that will help the student solve
mathematics and science test questions. A student’s questions must not be answered in a way that would
lead the student to unmistakably infer the correct answer to a question. If the FCAT is administered to a
group of students, the teacher may answer questions about directions for the benefit of the group; questions
of clarification from individual students must be answered on an individual basis without disturbing other
students taking the test.
For the reading test, the ESOL or heritage language teacher may answer student questions about the general
test directions in a way that the student would not be unmistakably led to infer the correct answer to any of
the questions. The teacher is prohibited from reading words to the student from the passages, test items, and
performance tasks, and from answering student questions about the passages, test items, and performance
tasks.
For the writing test, the ESOL or heritage language teacher may answer student questions about the general
test directions in their heritage language. The teacher may answer specific inquiries concerning a word or
phrase in a writing assessment prompt that is confusing the student because of limited English proficiency.
In no case shall assistance be given to the student in responding to the writing assessment prompt. The
teacher is prohibited from reading the prompt to the student. All student responses
must be written in
English.
Responses written in languages other than English will not be scored.
Dictionary.
LEP students may have access to an English-to-heritage language translation dictionary and/or
heritage language-to-English translation dictionary, such as those made available to LEP students in an
instructional setting. However, a dictionary providing definitions written exclusively in the heritage
language or in English may not be provided.
31
€
FLORIDA DEPARTMENT OF EDUCATION
www.myfloridaeducation.com
32
€
WRITING
€
TEST ADMINISTRATION MANUAL
€
FEBRUARY 2003
€
33
€
TEST ACCOMMODATIONS FOR LIMITED ENGLISH PROFICIENT (LEP) STUDENTS
€
Districts are required to offer accommodations to LEP students who are currently receiving services in a
program operated in accordance with an approved District LEP Plan. Permissible modifications for these LEP
students are listed below. The test may be administered with any one of these modifications or a combination of
accommodations that are determined to be appropriate for the particular needs of the LEP student. However, all
testing, with or without accommodations, must be completed during the prescribed testing dates shown on the
inside front cover of this manual.
Flexible Setting. LEP students may be offered the opportunity to be tested in a separate room with the English
for Speakers of Other Languages (ESOL) or heritage language teacher acting as test administrator. Parents must
be informed of this option for students not of legal age and shall be given the opportunity to select the preferred
method of test administration.
Flexible Scheduling. LEP students may take a part or session of the test during several brief periods within one
school day; however, a session of the test must be completed within one school day.
Flexible Timing. LEP students may be provided additional time; however, a session must be completed within
one school day.
Assistance
in
the Heritage Language. For the mathematics and science tests, LEP students may be provided
limited assistance by an ESOL or heritage language teacher using the student’s heritage language. The teacher
may answer specific questions about a word or phrase that is confusing the student because of limited English
proficiency, but is prohibited from giving assistance that will help the student solve mathematics and science
test questions. A student’s questions must not be answered in a way that would lead the student to unmistakably
infer the correct answer to a question. If the FCAT is administered to a group of students, the teacher may
answer questions about directions for the benefit of the group; questions of clarification from individual
students must be answered on an individual basis without disturbing other students taking the test.
For the reading test, the ESOL or heritage language teacher may answer student questions about the general test
directions in a way that the student would not be unmistakably led to infer the correct answer to any of the
questions. The teacher is prohibited from reading words to the student from the passages, test items, and
performance tasks and from answering student questions about the passages, test items, and performance tasks.
For the writing test, the ESOL or heritage language teacher may answer student questions about the general test
directions in their heritage language. The teacher may answer specific inquiries concerning a word or phrase in
a writing assessment prompt that is confusing the student because of limited English proficiency. In no case
shall assistance be given the student in responding to the writing assessment prompt. The teacher is prohibited
from reading the prompt to the student. All student responses must be written in English. Responses written in
languages other than English will not be scored.
Dictionary. LEP students may have access to an English-to-heritage language translation dictionary and/or
heritage language-to-English translation dictionary, such as those made available to LEP students in an
instructional setting. However, a dictionary providing definitions written exclusively in the heritage language or
in English may not be provided.
34
€
Appendix C
Discussion of NCLB “Starting Point”
Section 1111(b)(2)(B) of NCLB requires each state to create an accountability
program to ensure that all schools are making adequate yearly progress (AYP).
Each state has the flexibility to define how it will approach this task and hold
schools responsible for the progress of the students.
The law sets forth various requirements that will not be quoted herein. Instead,
the discussion that follows will provide Florida’s solutions and will include a
discussion of all relevant points.
Florida’s student assessment tests measure the same high standards for all
students, are valid and reliable instruments, seek continuous improvement of
students’ educational attainments, measure all schools against the established
standards, and report disaggregated test results for all groups required by NCLB.
The discussion that follows will address (1) the starting point for measuring
progress, (2) the timelines for improvement, (3) other indicators that will be used,
(4) annual measurable objectives, and intermediate goals for improvement.
Since the Florida Comprehensive Assessment Test has been in place for several
years, the Department of Education has complete information on the current
status of student achievement in reading and mathematics in grades 3-10 and
writing achievement in grades 4, 8, and 10. Information on student achievement
in science will not be available until after the spring 2003 assessment has been
conducted.
Because Florida has not established different levels of performance for the
writing assessment as is required by NCLB, this test will not be used to meet the
requirements of Section 1111(b)(3)(C)(v)(I) or (II). The writing assessment
results will be used as an additional indicator as required by Section
1111(b)(2)(C)(vii). See the discussion in the following pages about “other
indicators.”
The Department analyzed the results of the FCAT administered in the spring
2002 and the results are presented in the following discussion. In considering
what should be the starting point for AYP, the FCAT data could be presented in
several ways: (1) as mean scale scores on the FCAT 100-500 scale, (2) as
mean scale scores on the FCAT 0-3000 vertical scale, or (3) as percents of
students in the “Proficient and Above” category. Since the latter is considered to
be the easiest to understand and is consistent with the overall objective of getting
students to be Proficient or better, the “starting point” data were analyzed and are
presented as percentages. This does not preclude the Department from using
35
€
the student performance in terms of the vertical scale for the purpose of tracking
progress over time.
Section 1111(b)(2)(E) provides that the starting point shall be, at a minimum,
based on the higher of the percentage of students at the proficient level who are
in –
“(i) the State’s lowest achieving group of students described in
subparagraph (C)(v)(II); or
(ii) the school at the 20
th
percentile in the State, based on enrollment,
among all schools ranked by the percentage of students at the proficient level.”
Florida’s data were analyzed both ways, separately by grade level and subject
area (reading and mathematics). The FCAT nationally-normed test (the SAT-9)
was not used in this analysis since it is not part of the State’s school
accountability program. Instead, only the portion of FCAT that is constructed
around the Sunshine State Standards was used. (This is commonly identified as
the FCAT-SSS.)
36
€
Table 7 presents the results of the analysis method specified in (ii) for reading and
mathematics. The percent of students scoring Level 3 and above was calculated for
each school, and the schools were ranked. Counting upward from the lowest scoring
school, a school containing the 20%-tile of student enrollment was located. This
analysis depends on counting the student population within each school without regard
to how many grade levels are present in each school. Thus, the population being
counted is not the population of students in the tested grade level who earned ratings of
“Proficient or Above” but is, instead, the total enrollment of the school itself.
Table 7
Identification of Starting Points Based on Achievement
and School Enrollment
Reading Mathematics
30.68% 37.54%
NCLB specifies that the starting points will be the HIGHER of the results of the two
analyses. Since these starting points are higher than those derived from the first
analysis, the starting points will be those shown in Table 7: 31% for reading and 38%
for mathematics.
83
€
Intentionally left blank
84
€
Florida NCLB Consolidated Application
Appendix D
Annual Progress Objectives
NOTE: Year 1 = 2001-02 base year.
Source Data:
Reading
% Prof.
Year
€
2001-02
0
31
€
2002-03
1
31
€
2003-04
2
31
€
2004-05
3
48
€
2005-06
4
48
€
2006-07
5
48
€
2007-08
6
65
€
2008-09
7
65
€
2009-10
8
65
€
2010-11
9
82
€
2011-12
10
82
€
2012-13
11
82
€
2013-14
12
100
€
Starting Point and Annual Objectives for
Reading, 2001-02 - 2013-04
0
20
40
60
80
100
120
1
4
6
9
10
11
12
13
Years
% Proficient or Above
3
2
5
8
7
85
€
Starting Point and Annual Objectives for
Mathematics, 2001-02 - 2013-04
0
20
40
60
80
100
120
1
10
11
12
13
Years
% Proficient or Above
9
8
7
6
5
4
3
2
NOTE: Year 1 = 2001-02 base year.
Source Data:
Math
% Prof.
Year
€
2001-02
0
38
€
2002-03
1
38
€
2003-04
2
38
€
2004-05
3
53
€
2005-06
4
53
€
2006-07
5
53
€
2007-08
6
68
€
2008-09
7
68
€
2009-10
8
68
€
2010-11
9
83
€
2011-12
10
83
€
2012-13
11
83
€
2013-14
12
100
€
86
€
Florida NCLB Consolidated Application
Appendix E
Florida’s AYP Plan
As an introduction to Florida’s AYP proposal, first consider how the NCLB
requirements for accountability are structured. As Congress and the President
planned for NCLB, they faced a dilemma in that the 50 states and territories have
different academic standards and student assessment programs which are not
equivalent, interchangeable, measure the same content, administered at the
same grade levels, given to students at the same time of year, generate the
same kind of information, or have the same impact. Indeed, some states prefer
not to emphasize centralized testing programs and assign that responsibility to
each district. This is a direct result of the provisions in the Constitution that give
states the responsibility of implementing a public education system.
Congress sought an effective way to measure the success of the Title I
provisions in NCLB given the realities of such a diverse national system. A good
evaluation system would try to accumulate information across all of the states,
districts, and schools to provide the means of making sensible comparisons
among the various delivery agents. For years, however, Congress has had little
success in gathering such common information.
To illustrate the approaches used by Congress in its attempts to gather
meaningful common data, consider the “anchor test study” and the “NCE” units
approach. In the former, an attempt was made to link together several different
commercially available norm-referenced tests in reading. The attempt proved to
be extremely difficult and was abandoned by the mid-1970’s. The latter
approach tried to create a type of single scale that could be used in place of a
single test administered in all schools. The Normal Curve Equivalent scores
were useful but did not solve the problems associated with different instructional
programs, different curriculum expectations, and differences in content being
assessed.
As Congress created the language for NCLB, it was faced with the task of
requiring improvement in student achievement while not imposing a national
testing program or a national student identification number to track progress over
time. The model that evolved, therefore, is basically the same approach as has
been used in the past—a “status” model rather than a “growth” model. Student
achievement within a school, district, or state is to be measured during the
current academic year and the results compared to the achievement in the
following year. In other words, fifth grade students in 2002-03 will be compared
to fifth grade students in 2003-04 even though the students are not the same!
The assumption is that one year’s group of students is not significantly different
than the next year’s group of students, an idea that is extremely tenuous in these
days of high family mobility. It is further exacerbated in a state like Florida where
87
€
Florida NCLB Consolidated Application
there is tremendous in-migration of students every year, making each year’s
class different in many ways than the one last year.
Now consider what NCLB requires each state to do in measuring AYP. The
following five concepts summarize the important steps.
1.
?
Starting points are determined based on 2001-02 reading/language arts
and mathematics achievement data. With the goal of 100% “Proficient or
Above” in 12 years, each state stipulates annual measurable objectives
(growth targets). One or more “other indicators” are selected for
measurement.
2.
?
Each state gathers assessment information (including other indicators)
and reports the % “Proficient and Above” for: all students, economically
disadvantaged, students with disabilities, limited English proficient
students, and five categories of race/ethnicity (white, African-American,
Hispanic, Asian, American Indian/Alaskan). This is a status measurement
at a point in time in which current performance is compared to the growth
targets.
3. Each
school’s performance is compared to the state measurable
objectives. Each subgroup in #2 above must meet or exceed the
objectives. However, if one or more subgroups do not meet the
objectives, the school will meet AYP if the performance of the low
performing subgroups increases by 10% compared to the previous year
and if those subgroups made progress on one or more of the other
indicators and if not less than 95% of each subgroup of students
participated in the assessment. While the first comparison is one of
status, the second could be either a status measurement comparison or a
growth comparison for a cohort of students because the law does not
prescribe which it will be.
4. Each
district’s performance is compared to the state measurable
objectives. Each subgroup in #2 above must meet or exceed the
objectives.
However, if one or more subgroups does not meet the objectives, the
district will meet AYP if the performance of the low performing subgroups
increases by 10% compared to the previous year and if those subgroups
made progress on one or more of the other indicators and if not less than
95% of each subgroup of students participated in the assessment. While
the first comparison is one of status, the second can be either a status
measurement comparison or a growth comparison for a cohort of
students.
88
€
Florida NCLB Consolidated Application
5.
?
Compare the state’s performance to the measurable objectives. Each
subgroup in #2 above must meet or exceed the objectives.
In addition, the state must meet its objectives relating to the development
and attainment of English proficiency for LEP students.
Assuming that these steps have been completed, the results of the assessments
will be displayed for the public and the educational community. The conditions
for meeting AYP under NCLB are challenging in that the school and district must
meet the state targets in each of several separate comparisons. This can be
illustrated by the following table.
The data display would appear as shown below for elementary, middle, and
senior high schools.
Elementary and Middle Schools
Reading
Reading
Participation
Rate
Math Math
Participation
Rate
Other
(Writing)*
All
students
Econ.
Disadvantaged
White
Black
Hispanic
Asian
Am.
Indian
SWD
LEP
* In accordance with Section 200.19 of the final regulations, the “Other Academic Indicators” will
be disaggregated by subgroup for reporting purposes but will not be used for determining AYP.
89
€
Florida NCLB Consolidated Application
Senior High Schools
Reading
Reading
Participation
Rate
Math
Math
Participation
Rate
Other
(Graduation
Rate)*
Other
(Writing)*
All students
Econ.
Disadvantaged
White
Black
Hispanic
Asian
Am. Indian
SWD
LEP
* In accordance with Section 200.19 of the final regulations, the “Other Academic Indicators” will
be disaggregated by subgroup for reporting purposes but will not be used for determining AYP.
For a senior high school to meet all of its targets requires a number of separate
Yes/No conjunctive decisions. Scoring relatively higher in reading will not
compensate for low scores in mathematics as would happen in a compensatory
model.
The reporting of assessment information in the previous tables is subject to the
following conditions:
1.
?
The cells in the above table will be reported subject to the limitations on
cell sizes previously described.
2.
?
The school’s values in each cell of the above table will be the average of
the current year’s performance and the previous year’s performance, with
the exception of the percent tested and school grade which always shall
be the current year’s values. The averages will be unweighted.
2.
2.3.
Students who take an alternate assessment will have their results
reported in categorical classifications that include the designation of
“Proficient,” thereby making it possible for their performance to be counted
with those of other students.
3.4.
Any school that is in its first year of operation will be included in the
system but with only one year of data to report. Schools that include K-2
90
€
Florida NCLB Consolidated Application
and do not take the statewide assessment (FCAT) will be assigned the
proficiency ratings earned by the school their students attend in grade 3.
4.5.
In the event that a school district selects additional indicators with
which to determine a school’s AYP, as is authorized in Section 1116 of
HR-1 (NCLB authorization), the district shall provide these data to the
Department of Education for use with the State Report Cards required by
law.
5.6.
The required NCLB data analyses and reports will be prepared for
each school regardless of the grade level configurations. That is, a K-5
school will generate data displayed as shown on the previous page as well
as a grades 9-12 high school or a K-12 school. Each can be reported in
terms of the percent of students who are “Proficient or Above” in reading
and in mathematics. At the district level, the data will be reported for all
students in the district who are “Proficient or Above” in reading and
mathematics without regard for school-by-school distinctions.
NCLB includes several important concepts such as the following.
(1) All students must be held to the same, challenging standards;
€
(2) All students are to be assessed;
€
(3) The progress of students is to be consistent and forward-looking with the
€
goal of moving all students to at least the Proficient level within 12 years;
(4) Assessment results are to be aggregated and reported to parents
annually;
(5) Assessment and accountability results are to be disaggregated by seven
major subgroups and
(6) Student progress is monitored annually and improvement is noted when
performance improves with a specific grade level(s) over time. This is a
“status system” on a very large scale.
While these principles are admirable, the NCLB model can be improved in those
states that have launched an effort to implement “value-added achievement
monitoring” in which the progress of individual students is monitored across time.
If Johnny’s reading achievement is measured in fourth grade and he is measured
again in fifth grade with a test that has been vertically linked across the grade
levels, his absolute growth, or lack thereof, can be measured and reported. The
keys to such a system are: (1) a student testing program in all grade levels, (2)
test content keyed to an established set of curriculum expectations, (3) a vertical
measurement scale that allows student scores to be reported from the least
grade to the highest, and (4) a student identification system that assigns a unique
number to a student through his/her lifetime within the public school system. All
of these elements are found in the existing Florida public school assessment and
accountability system.
91
€
Florida NCLB Consolidated Application
The existing Florida school accountability program produces school grades
based on student performance in reading, writing, and mathematics. Special
attention is paid to students who are in the lowest 25% of students in FCAT
Levels 1, 2, and 3 in each school. Further, the program features the
measurement of academic growth of individual students through the FCAT
vertical score scale in reading and mathematics.
Florida proposes to fully implement NCLB in all schools and deliver analyses and
results exactly as the law specifies. However, in order to link the NCLB status
system to Florida’s existing status and gain system, it is proposed that no school
will be allowed to be designated as meeting AYP if it has been graded “D” or “F”
under the A+ school grading system.
Under the terms of the system Florida is proposing, all of these objectives will be
met with a system that is more challenging than the NCLB requirements.
Congress enacted a law that meets the constraints posed by the vast majority of
states who do not have the student assessment and accountability traditions
found in Florida. If Florida changes its A+ system to serve as the NCLB
accountability system, it actually will cause our system to regress, not move
forward with its measurement of individual student learning gains.
Florida already has a tremendous investment in its A+ Plan for Education, and
educators and citizens are familiar with it. To make changes would require
amendments to existing statutes, administrative rules, computer programs,
administrative infrastructure, and information dissemination to all public schools.
The Florida school accountability program grading system is illustrated in
Appendix F of this Plan. Close inspection demonstrates how the program holds
schools accountable for scoring high on the FCAT, specifically, having increasing
numbers of students earning FCAT Achievement Level scores previously
identified as being the equivalent of “Proficient or Above” as required by NCLB.
The program requires students to make learning gains in reading and
mathematics if they presently are earning less than “Proficient.” If they already
are achieving at the “Proficient or Above” levels, the school earns points to the
degree that the students do not regress.
Specific attention is paid to the achievement of those students who are
demonstrating the least achievement, below “Proficient.” Schools earn points for
all students in the lowest 25%-tile who make “adequate progress,” defined as
gaining in achievement as much as the norm group for the State.
The existing system already incorporates the results of the writing assessment
that we have proposed be the “other indicator” for grades 4, 8, and 10.
92
€
Florida NCLB Consolidated Application
If a school does not meet its annual growth targets, it can meet AYP under the
“safe harbor” provisions of NCLB for improving the performance of students in
various subgroups:
1.
?
The percentage of students in that group below the State’s proficiency
achievement level “decreased by [at least] 10 percent of that percentage
from the preceding school year”; and
2.
?
That group made progress on the other indicator of writing or, for high
schools, the graduation rate; and
3.
?
Not less than 95% of the students enrolled in each group takes the
statewide assessment.
However, if a school does not meet the State’s annual objective growth target for
two years in a row or if the school otherwise earns a grade of “D” or “F,” it will be
designated a school in need of targeted assistance and additional services or
sanctions will be identified.
If a school does not meet the 10% growth “safe harbor” as described above, the
progress of its non-Proficient students will be examined. If the school
demonstrates that more than half of the non-Proficient students have grown more
than their individual growth expectation from last year to the current year, AYP
will have been met. The State will establish the growth expectations on the basis
that non-Proficient students should be Proficient or Above within a specified
period of time.
Schools and districts will be evaluated separately for reading and mathematics
performance. A school or district could fail to meet its AYP requirements in
reading one year, improve in reading the second year, and become deficient in
mathematics the second year. If this occurs, the school or district will not be
subjected to the requirements of Sections 200.32-200.34 of the NCLB rules
because it has not had two consecutive years of poor performance in the same
content area. If a school or district fails to meet its AYP requirements in the
same content area (e.g., reading) for two consecutive years, it will be subjected
to the requirements of Sections 200.3-200.34.
93
€
Florida NCLB Consolidated Application
Intentionally left blank
94
€
Florida NCLB Consolidated Application
Appendix F
€
A+ School Grading System
€
The following charts explain how school grades are calculated for the A+ school
grading system.
95
€
Florida NCLB Consolidated Application
96
€
Florida NCLB Consolidated Application
97
€
Florida NCLB Consolidated Application
Intentionally left blank
98
€
Florida NCLB Consolidated Application
Appendix G
Annual Learning Gain Targets to Proficiency
The FCAT was originally developed to test reading in grades 4, 8, and 10, and
mathematics in grades 5, 8, and 10. The test results were reported in terms of a
succession of annual “status reports” that revealed the performance of Florida students
who were in different cohorts. The data were reported for each subject using a score
scale that ranged from 100-500, and each scale was separately computed for each
grade level. Progress over time was reported as changes in the performance of each
grade level group – i.e., this year’s fourth grade students were compared to last year’s
fourth graders to see if the average score changed or, for example, if more students
were earning “Level 3” scores.
In 2001, the FCAT program was expanded so that the tests now are being administered
in all grades 3-10. This offers the opportunity to make use of a new score scale that
links adjacent grades together and permits progress to be tracked over time, based on
what is commonly called a “developmental scale.” The effect of this improvement is that
student performance across the grades can be tracked across this scale. As a student
moves from grade to grade, his/her performance can be monitored and compared to the
performance of other students in Florida. Most importantly, the yearly progress of each
student can be reported by the change in the developmental scale scores.
Florida will use this new developmental scale to develop a plan with annual learning
gain targets for all students below proficiency.
Four-Year Plans
By using the developmental scale, districts and schools can chart their students’ growth
as they move across the grade levels. Such charts, or data plots, make it possible to
answer the question, “Is this student making adequate progress for each year in
school?” and, “If this student keeps making the same amount of progress, is he/she
going to be ready for the grade ten graduation testing requirement?” For those students
achieving below proficiency, we are able to set annual targets for growth that will set the
student on a trajectory to proficiency.
For example, a third-grade student with a FCAT Reading score of 150 would be
designated as scoring in Level 1, clearly below proficiency. The Developmental Scale
Score (DSS) for this student would be 389 in the conversion to the vertical scale. To
bring this student to a level of proficiency by the end of 7
th
grade, the student would
need to have a DSS score of at least 1719 and his/her plan would map out the growth
needed each year to reach the goal of proficiency, with identified annual benchmarks.
Instructional plans would tie to the content scores on the FCAT Reading test, so that
specific strategies link to content areas needing attention.
99
€
Florida NCLB Consolidated Application
Example Developmental Scale for Reading
The following chart illustrates the relationship between the developmental scores and
the FCAT achievement levels.
Developmental Scale for Reading with
Achievement Level Cut Points
0
500
1000
1500
2000
2500
3000
3500
3
7
10
Grade
Scale Score
Level 1
Level 2
Level 3
Level 4
Level 5
6
5
4
9
8
100
€
Florida NCLB Consolidated Application
Appendix H
The Florida Department of Education
Return on Investment (ROI)
Currently the Department of Education has available a multitude of reports which
provide information using student, staff and finance data. For the purpose of assessing
the quality and efficiency of the various education delivery systems, the Department is
in the process of using the available information to develop a comprehensive set of
measures for the purpose of comparisons and trend analysis of schools, districts, and
postsecondary institutions. By integrating the goals of highest student achievement and
quality and efficient services, this effort will establish an accountability system for the
use of public education resources for all delivery systems.
A comprehensive system for calculating “return on investment” based on indicators of
institutional efficiency and effectiveness is under development. The system will have
many benefits, including:
•
?
The use of data to influence decision-making – Good decisions are based on
inquiry and analysis. Information technologies are available to make this
possible for school-based administrators, as well as external users of
education information such as legislators and researchers.
•
?
The use of data to target specific areas for improvement – Timely and
accurate data can assist decision makers at all levels in focusing on
improvement strategies.
•
?
The use of disaggregated data to examine wide-ranging goals –
Disaggregating data for analysis allows for identification of programmatic
and/or fiscal inequities and assists in the establishment of baselines for
improvement.
•
?
The use of data in rapid program evaluation – In order to have an impact,
program evaluation must be timely as well as complete. By compiling and
linking program and other data in an accurate and well-designed retrieval
system, program evaluation can be effectively and efficiently accomplished.
•
?
The use of data to examine the relationship between cost and program
effectiveness.
101
€
