Leaving aside for now the consensus of the scientific community that software based voting machines can never reach acceptable levels of reliability and security, Mr. Kellner's letter, outlining examples of specific problems with these machines, reveals the low to non-existent quality control standards and irresponsible business ethics of the vendors, an assessment which is perfectly consistent with the pattern of behaviour we have discussed previously in these pages.
We can also discern from this letter that the State may soon be under increasing pressure to lower or ignore its own testing standards in the interest of time. We urge the Legislature and the Attorney General to continue to stand firm against such pressure and to use this opportunity to re-examine the legal and/or practical necessity for abandoning the immutable technological genius of our lever machines.
-- Joanne Lukacher
From: Douglas A. Kellner
Date: Wed, Jul 23, 2008 at 6:36 PM
Subject: Systest Report on NY Certification Progress
To: "Douglas A. Kellner"
As most of you know, the New York State Board of Elections is doing certification testing on two precinct based digital optical scanning systems, the Dominion Voting System ImageCast (marketed in New York by Sequoia Pacific) and the ES&S DS-200. I believe that ours is the most comprehensive testing anywhere, including that done by the EAC.
The court ordered timetable provides for completion of certification testing by October 1, 2008, which appears to be increasingly unrealistic. The weekly report from SysTest, New York's independent testing authority, excertped below shows just how difficult it would be to meet this target date.
What is particularly distressing is discussion that New York should consider overlooking what some are describing as hundreds of "de minimis" discrepancies from the VVSG or NY regulations. In my view, if they are really de minimis, I do not understand why the vendors cannot address them. I am trying to keep an open mind, but I believe that this may become a major issue within a couple of months as pressure mounts to have us overlook the shortcomings in the interest of replacing the lever machines in 2009. The industry and the Department of Justice will argue that if every other state is using equipment that does not comply with current federal standards, why should New York be the exception?
I believe that there is still strong bi-partisan consensus within New York that we should stick to our policy that newly purchased voting equipment meet all of the currently applicable standards.
This is an excerpt from the SysTest Weekly Report to the New York State Board of Elections:
SCHEDULE RISKS: Time and schedule continue to be the major constraint and risk to the successful completion of this project. SysTest Labs remains committed to providing thorough and complete functional testing, as well as initiating the run-for-the-record test pass in accordance with the current schedule and timeline. However; NYSBOE's ability to meet its court-mandated timeline for complete and thorough testing is at significant risk for the reasons described below.
Issue 1: Test Case Execution. With only 16 working days until the commencement of the run-for-the-record test pass, there remains insufficient time for a complete test pass through the entire test case suite for both initial and regression test passes. Our risk mitigation strategy is to configure five test labs for each vendor, allowing us to execute test cases in parallel. The likely outcome, however, is that not all of the test cases will be completed within the remaining allotted time, and the Vendor's will be unable to address all of the discrepancies discovered.
Issue 2: Documentation Discrepancies. Numerous documentation discrepancies, caused in large part by missing information from both Vendors' documentation, prevent us from finalizing the test procedures in all of the test cases. As of close of business on Tuesday, July 22, Sequoia/Dominion has 148 open documentation discrepancies and ES&S has 414 open documentation discrepancies. Without timely receipt of this missing information and documentation discrepancy fixes, thorough and complete testing cannot be accomplished per the schedule.
Issue 3: Functional Discrepancies. Based on our experience, both as an ITA/VSTL and with the current NYSBOE test project, we anticipate that a large number of discrepancies will be discovered as part of the functional test effort. There is a significant risk, given the remaining allotted time, that the Vendors will be able to address all of the expected discrepancies and SysTest Labs will have sufficient time to regression test the fixes. If a significant number of discrepancies are uncovered, the run-for-the-record test pass will have to proceed with these remaining open.
Issue 4: Hardware/Software Deliveries. SysTest Labs does not yet have all of the necessary hardware and software, which is still being delivered by the Vendors. Moreover, as we configure the voting systems, we are discovering additional documentation and procedural discrepancies.
Issue 5: NYSBOE Approval of Deliverable 7. Based on direction from the NYSBOE, SysTest Labs cannot begin testing without formal approval of Deliverable 7, Voting System Specific Test Plans. Therefore, without timely approval, the start of functional testing will be delayed.
Issue 6: Required NYSBOE Policy Guidance. SysTest Labs requested from NYSBOE policy guidance with respect to the configuration files for both Sequoia/Dominion and ES&S. At issue here is whether the State will allow changes to configurations in set-up parameters for individual counties or whether a single configuration will be mandated. This decision impacts vendor-specific test case development and thus will effect test schedules.
The following is a sample of the Vendor-specific delays that are keeping us from beginning functional testing:
Sequoia/Dominion:
It was discovered while configuring the Sequoia/Dominion EED computer that the election setup procedures do not match the documentation. Sequoia/Dominion reports that the procedures were changed, but the documentation was not updated. Discrepancies have been entered.
Because we do not have documentation from Sequoia/Dominion that tells us how to set up a single tower as a standalone system (all documentation still references the network), the Sequoia/Dominion rep had to walk us through several hours of the installation to get to a point where we could proceed with the current documentation. This document does NOT address several necessary printer drivers and Adobe configurations. Discrepancies are being entered.
Sequoia/Dominion is still delivering and building the voting systems for Lot 1. This is impacting the configuration of the test labs and the start of functional testing.
We are still waiting for Sequoia/Dominion to deliver the COTS software to be installed on the voting test computers. This is impacting the configuration of the test computers and the start of functional testing.
Please reference the risks in the Vendor-specific progress report below. SysTest Labs is still waiting for Sequoia/Dominion to send source code mapping, information concerning crypto algorithms, additional test hardware, and additional documentation.
ES&S:
ES&S provided expired licenses for RM COBOL. They are in the process of activating the licenses, but this is impacting the configuration of the test computers and the start of functional testing.
ES&S has pulled all of the VAT A200-00 units from testing. We are waiting for ES&S to deliver updated A200 units for testing.
Some of the equipment for the test computers delivered from ES&S failed (i.e. 2 test computers, keyboards, mouse, and switches). ES&S is replacing the failing equipment, but this is delaying the configuration of the test labs.
The installation of COTS software has been slowed because of missing and/or incomplete documentation and instructions. This is resulting in multiple emails and calls to ES&S and is delaying the configuration of the test computers.
There is no documentation or instructions from ES&S concerning what COTS software is to be installed on the AIMS computer.
Please reference the risks in the Vendor-specific progress report below. SysTest Labs is still waiting for ES&S to send compilers for augmented source code review, information concerning crypto algorithms, additional test hardware, and additional documentation.
I have no objection if you post or forward this e-mail. If you want a copy of the full Systest Weekly Report, please send me an e-mail.
Douglas A. Kellner
Co-Chair
New York State Board of Elections
Tel. (212) 889-2121
Fax (212) 684-6224
Date: Wed, Jul 23, 2008 at 6:36 PM
Subject: Systest Report on NY Certification Progress
To: "Douglas A. Kellner"
As most of you know, the New York State Board of Elections is doing certification testing on two precinct based digital optical scanning systems, the Dominion Voting System ImageCast (marketed in New York by Sequoia Pacific) and the ES&S DS-200. I believe that ours is the most comprehensive testing anywhere, including that done by the EAC.
The court ordered timetable provides for completion of certification testing by October 1, 2008, which appears to be increasingly unrealistic. The weekly report from SysTest, New York's independent testing authority, excertped below shows just how difficult it would be to meet this target date.
What is particularly distressing is discussion that New York should consider overlooking what some are describing as hundreds of "de minimis" discrepancies from the VVSG or NY regulations. In my view, if they are really de minimis, I do not understand why the vendors cannot address them. I am trying to keep an open mind, but I believe that this may become a major issue within a couple of months as pressure mounts to have us overlook the shortcomings in the interest of replacing the lever machines in 2009. The industry and the Department of Justice will argue that if every other state is using equipment that does not comply with current federal standards, why should New York be the exception?
I believe that there is still strong bi-partisan consensus within New York that we should stick to our policy that newly purchased voting equipment meet all of the currently applicable standards.
This is an excerpt from the SysTest Weekly Report to the New York State Board of Elections:
SCHEDULE RISKS: Time and schedule continue to be the major constraint and risk to the successful completion of this project. SysTest Labs remains committed to providing thorough and complete functional testing, as well as initiating the run-for-the-record test pass in accordance with the current schedule and timeline. However; NYSBOE's ability to meet its court-mandated timeline for complete and thorough testing is at significant risk for the reasons described below.
Issue 1: Test Case Execution. With only 16 working days until the commencement of the run-for-the-record test pass, there remains insufficient time for a complete test pass through the entire test case suite for both initial and regression test passes. Our risk mitigation strategy is to configure five test labs for each vendor, allowing us to execute test cases in parallel. The likely outcome, however, is that not all of the test cases will be completed within the remaining allotted time, and the Vendor's will be unable to address all of the discrepancies discovered.
Issue 2: Documentation Discrepancies. Numerous documentation discrepancies, caused in large part by missing information from both Vendors' documentation, prevent us from finalizing the test procedures in all of the test cases. As of close of business on Tuesday, July 22, Sequoia/Dominion has 148 open documentation discrepancies and ES&S has 414 open documentation discrepancies. Without timely receipt of this missing information and documentation discrepancy fixes, thorough and complete testing cannot be accomplished per the schedule.
Issue 3: Functional Discrepancies. Based on our experience, both as an ITA/VSTL and with the current NYSBOE test project, we anticipate that a large number of discrepancies will be discovered as part of the functional test effort. There is a significant risk, given the remaining allotted time, that the Vendors will be able to address all of the expected discrepancies and SysTest Labs will have sufficient time to regression test the fixes. If a significant number of discrepancies are uncovered, the run-for-the-record test pass will have to proceed with these remaining open.
Issue 4: Hardware/Software Deliveries. SysTest Labs does not yet have all of the necessary hardware and software, which is still being delivered by the Vendors. Moreover, as we configure the voting systems, we are discovering additional documentation and procedural discrepancies.
Issue 5: NYSBOE Approval of Deliverable 7. Based on direction from the NYSBOE, SysTest Labs cannot begin testing without formal approval of Deliverable 7, Voting System Specific Test Plans. Therefore, without timely approval, the start of functional testing will be delayed.
Issue 6: Required NYSBOE Policy Guidance. SysTest Labs requested from NYSBOE policy guidance with respect to the configuration files for both Sequoia/Dominion and ES&S. At issue here is whether the State will allow changes to configurations in set-up parameters for individual counties or whether a single configuration will be mandated. This decision impacts vendor-specific test case development and thus will effect test schedules.
The following is a sample of the Vendor-specific delays that are keeping us from beginning functional testing:
Sequoia/Dominion:
It was discovered while configuring the Sequoia/Dominion EED computer that the election setup procedures do not match the documentation. Sequoia/Dominion reports that the procedures were changed, but the documentation was not updated. Discrepancies have been entered.
Because we do not have documentation from Sequoia/Dominion that tells us how to set up a single tower as a standalone system (all documentation still references the network), the Sequoia/Dominion rep had to walk us through several hours of the installation to get to a point where we could proceed with the current documentation. This document does NOT address several necessary printer drivers and Adobe configurations. Discrepancies are being entered.
Sequoia/Dominion is still delivering and building the voting systems for Lot 1. This is impacting the configuration of the test labs and the start of functional testing.
We are still waiting for Sequoia/Dominion to deliver the COTS software to be installed on the voting test computers. This is impacting the configuration of the test computers and the start of functional testing.
Please reference the risks in the Vendor-specific progress report below. SysTest Labs is still waiting for Sequoia/Dominion to send source code mapping, information concerning crypto algorithms, additional test hardware, and additional documentation.
ES&S:
ES&S provided expired licenses for RM COBOL. They are in the process of activating the licenses, but this is impacting the configuration of the test computers and the start of functional testing.
ES&S has pulled all of the VAT A200-00 units from testing. We are waiting for ES&S to deliver updated A200 units for testing.
Some of the equipment for the test computers delivered from ES&S failed (i.e. 2 test computers, keyboards, mouse, and switches). ES&S is replacing the failing equipment, but this is delaying the configuration of the test labs.
The installation of COTS software has been slowed because of missing and/or incomplete documentation and instructions. This is resulting in multiple emails and calls to ES&S and is delaying the configuration of the test computers.
There is no documentation or instructions from ES&S concerning what COTS software is to be installed on the AIMS computer.
Please reference the risks in the Vendor-specific progress report below. SysTest Labs is still waiting for ES&S to send compilers for augmented source code review, information concerning crypto algorithms, additional test hardware, and additional documentation.
I have no objection if you post or forward this e-mail. If you want a copy of the full Systest Weekly Report, please send me an e-mail.
Douglas A. Kellner
Co-Chair
New York State Board of Elections
Tel. (212) 889-2121
Fax (212) 684-6224
