CATS Task Force Meeting 3
Summary of CATS Task Force Meeting 3
September 18, 2008
There wasn’t much consensus during the group’s third meeting.
As of the close of this meeting, task force agreement on writing is limited to a general acceptance that writing instruction is important and that writing portfolios are useful instructional tools. However, group consensus quickly breaks down on the very important question of keeping writing portfolios in the state’s CATS school assessments. Opinions generally center in one of two very different camps.
One group believes that if the portfolio program is removed from CATS, then Kentucky’s teachers will stop teaching writing. This group firmly believes that only what gets assessed will be taught. The “Keep Them” group thinks the portfolio program mostly just suffers from administrative and management problems and only needs improvement in things like better instruction to teachers about how to manage this CATS element. Key spokespeople for this position include State Representative Harry Moberly and Marion County Schools Superintendent Roger Markum, who also represents the Kentucky Association of School Superintendents.
A second group believes that writing portfolios should be removed from CATS. The “Remove Them” group believes awkward rules intended to prevent teachers from giving students too much help on portfolios actually wind up inhibiting effective instruction. This group does believe that portfolios should remain in the instructional program. The group thinks teachers can be encouraged to do this by non-assessment policies such as an audit program. The group also points out that if, after 18 years of KERA, the writing portfolio proponents still have not educated teachers about how to effectively teach writing while coping with the rules imposed by having portfolios in accountability, then it is unlikely to ever happen. Key members of the “Remove Them” group include State Senator Dan Kelly and Kentucky Education Association President Sharron Oxendine.
Unfortunately, both groups argued more from beliefs and assertions than actual evidence during the meeting. In particular, the report on NAEP writing results requested by task force member Steve Stevens at the previous task force meeting wasn’t presented, or even mentioned. However, some other information presented at the meeting also raised questions about Kentucky’s accountability program with portfolios.
For example, Ken Draut, and associate commissioner at the Kentucky Department of Education, discussed an audit of the accuracy of the 2006-07 writing portfolio scoring. The audit shows an old problem continues – portfolio scoring continues to be inflated compared to the on-demand writing samples that are collected during the CATS academic tests. The audit found inflation is particularly evident for the higher CATS portfolio scores of “Proficient” and “Distinguished.” Draut reported that out of 588 portfolios originally awarded the top score of “Distinguished,” the audit found only 37, just six percent of those 588 portfolios, actually deserved this top grade. Incredibly, out of the same original “Distinguished” score group of 588, 176, or 30 percent, should have received the much lower score of “Apprentice,” according to the auditors.
There was another important message hidden in the CATS writing data that Draut presented. Across all school levels, the overall writing academic indexes and the portfolio scores increased from last year. However, the on-demand writing proficiency rates in 2007-08 for both elementary and middle schools actually dropped from the previous year. On-demand writing is assessed under test conditions during the CATS academic test periods. The results are scored by independent graders, unlike portfolio scoring that is done by teachers, who in turn are held accountable for the results.
As a note, the audit situation provides a clue to an unspoken reason why some may be fiercely protecting keeping writing portfolios in the accountability program. Because portfolio scores are inflated, if portfolios are removed from CATS, the overall school accountability index scores will decline.
Views expressed by several task force members indicated they do not want to do very much with CATS before 2014. Though he was absent from the meeting due to illness, Commissioner of Education Jon Draud previously stated he doesn’t want to make many changes. At this third meeting of the task force, State Board of Education Chair Joe Brothers also raised concerns that making any significant changes to CATS now would require a new three-year cycle to develop new scoring targets for schools. These hold-the-line advocates argue that changes will make it harder for schools to meet their CATS goals in 2014. Left unsaid, however, was the really critical question – are the current CATS goals truly valid targets worth preserving for another six years?
Although portfolios took up the lions’ share of the time, there was some discussion on another significant CATS issue – assessment of arts and humanities. It was pointed out that the CATS arts and humanities assessment may actually be working to diminish instruction in the performing arts. The test questions apparently have degenerated into little more than a vocabulary test while actual demonstrations of students playing instruments or showing off sculpture and paintings are totally absent from CATS. There seemed to be fairly widespread interest among task force members in a pilot program to change the CATS arts assessment, but issues of funding could be a significant impediment to a statewide implementation.
Interestingly, no-one seemed concerned that changing the arts assessment might require a new three-year adjustment time to develop new scoring targets. That argument only came up during the portfolio debate.
At the close of the meeting, discussion facilitator Dave Spence was unable to get any consensus beyond a universal desire for more information.
Finally, there was a request from the committee for all interested parties to submit comments and ideas to the committee. The announcement was later made in a formal press release from the department of education that was carried in many state newspapers and on line in several education-focused blogs, including the Bluegrass Institute. 
- CATS Task Force
- CATS Task Force Members
- CATS Task Force Meeting 1
- CATS Task Force Meeting 2
- CATS Task Force Meeting 4
- CATS Task Force Meeting 5
- CATS Task Force Meeting 6
- Task Force Asks For Public Input Bluegrass Policy Blog accessed October20. 2008