BJCP exams are graded by volunteer members of the program who receive very little compensation other than to give back to this hobby, to help assure that the next generation of beer judges are of high caliber, and to make progress towards completion of one of the service requirements needed to qualify for the Grand Master levels.
Since 1994, the grading process has involved three discrete rounds of review: grading, review, and close-out. The initial grading, scoring and report writing round is conducted by two graders working to develop consensus scores and feedback. The lead grader is responsible for composing the Report to Participant (RTP) incorporating comments provided by both graders. The second round of grading is conducted by an Associate Exam Director (AD), who checks over the grading process for accuracy and consistency, and provides an editorial review of the RTP. Finally, a tertiary grading and report assessment are provided by a regional Exam Director (ED), who finalizes the scores and report and provides these materials to program administrators for database entry and delivery of the RTP to the examinee. The three tiers of exam grading are designed to provide accuracy, consistency, and detail to the grading process. Just as beer evaluation is subjective, so to is skill evaluation. The exam program prides itself on being thorough and fair above all else, and the performance and dedication of graders are fundamental to that process.
Graders are chosen and assignments are made at the sole discretion of the Exam Directors. General qualifications for being a grader include:
- An active BJCP member in good standing
- Typically National or higher rank with a minimum exam score of 80
- Demonstrated effective evaluation, communication and interpersonal skills
- Timeliness in grading the exams and completing the Report to Participant forms
The first two qualifications reflect the role of the BJCP exam graders as mentors to and role models for prospective and current judges. Graders represent the BJCP, and their names explicitly appear at the top of each RTP, so it is important that exam graders be respected members of their local and regional judging communities. The third and fourth qualifications are an acknowledgment that our volunteer graders need to honor their commitment to completing grading tasks in a timely manner. This has become an increasing challenge in recent years as the written feedback level on typical RTPs has increased from a single paragraph to approximately two pages. At the same time, we have endeavored to turn around the BJCP exam results in twelve weeks or less, so these two goals often conflict.
Graders are expected to treat exam information as confidential data. The scores are only released after the complete review process and then only by the Exam Director and program administrators. It is inappropriate for a grader to share or comment on that information. Confidentiality must be maintained throughout the entire grading process.
Grader performance is assessed by co-graders and the Exam Directors. Instructions provided to the graders form the basis for grader evaluation. These instructions include specific methods of evaluation, grade report products, and what timelines are expected. As the process for initial grading involves the development of consensus (agreement) scores between two individuals, it is expected that each grader work collegially and collaboratively through this interaction. If persistent issues of difficulty arise between graders, then a written report detailing these issues should be sent to the Associate Exam Director of record. The Associate Exam Director will conduct an initial investigation and attempt to resolve the problems. In the event of intractable disagreement, a written report must be filed with both the AD and the ED of record.
Graders’ technical performance will be evaluated by both Associate Exam and Exam Directors, as assessed against the instructions provided to the graders. In cases where grader work products (generally RTPs and scoring accuracy) require substantial and material editing, rewriting and revising on the part of the Directors, the Directors are instructed to file and archive a written feedback narrative to the grader indicating the issues with grader performance and suggested ways to improve. Issues with the timeliness of the exam grading will be handled in a similar manner. While we recognize that there has to be a balance between volunteer grading activities and other aspects of our lives, it is much easier for us to transfer grading assignments than to accommodate delinquent exam sets. Graders are typically given four to six weeks to complete the grading assignment, so there is ample opportunity to work with the Directors if conflicts arise. In recent years, there have even been several instances where an exam grader became incommunicado without completing any of his or her grading responsibilities.
In the case of minor performance issues, the grader will remain in the grader pool under review status, and the AD will be provided with records of the recommendations made by the ED, and instructed to pay particular attention to those issues described. In the case of persistently poor performance or failure to complete grading assignments, it is the Exam Director’s discretion to remove individuals from the grader pool.
In cases where graders have been removed from grading assignments following the policies outlined above, and the individual wishes reconsideration, the party is instructed to file an appeal report to the BJCP President. This appeal report should document past grading assignments and performance, including copies of grader assignment letters from the ED and close-out letters and grader forms/reports from graders to the AD indicating duration of grading period and completeness of tasks. Any additional records of discussions between graders and Directors are also encouraged to provide specific context into grader performance. Consequently, it is advised that thorough record keeping associated with all grader assignments and interactions be kept. Similar to documents associated with exam grading, all grader review materials are confidential.
Upon receipt of a formal grader appeal, the BJCP President will initiate a formal investigation, which will involve a minimum of three sitting Representatives, and result in a formal response/report within four weeks to the party in question detailing their findings and recommendations.