Numeric Score
For each criterion, you will provide a score between 1 (lowest) to 10 (highest) for each criterion. Do not score the General Comments. Use this scoring guide to assist you in making your scoring decisions.

Select the score that best reflects the applicant overall in each criterion. Assume that the applicant should get a 10 and then reduce the score to where it predominantly fits within the scoring guide. The applicant does not need to achieve every item in the category; score them where they predominantly fit. The score should align with your comments.

If the applicant does not achieve an acceptable level based on the scoring guidelines, you may score them less than six. This score should be consistent with the proportion of the criteria achieved. Applicants with a total average score of less than six are not funded and may not apply for other funding for the same program in that fiscal year. However, if they meet some part of the criterion, they should get some points. For example, a score of one in management ability suggests that there is no clarity to their application, management ability is demonstrably lacking, and they have considerable organizational issues. To be fair, the score should be proportional.

Written Evaluation
For each criterion and application, please provide one to three sentences on how the proposal addresses the criterion. Address the proposal’s strengths and weaknesses with constructive criticism focused on criteria. Avoid simply repeating narrative elements or comparing to other applicants.

The comment should be comparable with the numeric score. For example, a primarily positive comment should have a good score, while a primarily negative comment should accompany a lower score.

Common Errors

Reiterating the application narrative
This is the most common error. The comment should be relevant to the applicant. Connect the description to the proposal’s strength/weakness or provide constructive criticism.
Example: “The applicant presents to 600 audience members each season.”
This is a non-evaluative statement. It could be positive (strength) or negative (weakness), depending on the applicant’s community and target audience. You can turn this description into an evaluative statement regarding the Community Involvement criterion. For instance:
– (weakness) “Concerned that only 600 are reached: Encourage the applicant to increase their outreach for their programs, such as by including more group ticket sales.”
– (strength) “Pleased to see the strong effort to reached audiences. This is an achievement for this rural community.”

Discrepancy between the numeric score and written evaluation
This is another common error. Sometimes written evaluation ends up more critical/negative than the score reflects. Other times, the initial score was too low compared to the comments. To ensure an accurate evaluation, review the proposal, numeric score, and written evaluation, and adjust the score or written evaluation accordingly. Here are some discrepancy examples. (The ≠ symbol means “doesn’t equal.”)
– Artistic Quality: 9 ≠ “The artistic presentations have variable quality. Need formal evaluation process.” A lower score is more appropriate for these comments.
– Community Involvement: 6 ≠ “The film series has an impressive amount of collaborative support from various organizations. Selected films appeal to different cultures that are represented in the community. A higher score is a better match for these comments.
– Management Ability: 7 ≠ The grant is extremely well written. I am especially impressed to see this progression from a monthly express project grant. They have excellent support materials. A higher score is more compatible with these comments.

Select Missouri Arts Council Policies
For complete policies, check Apply for Funding. 

Definition of Quality: While it is the Missouri Arts Council’s goal to fund programs that are of high quality, the we believe that applicants cannot be measured by a single standard of quality. A high-quality performance in one area of the State may not be considered to be of high quality in another area.

University/College programs are funded to be available to their community. Involvement by students, staff, or faculty is not why they receive public funding. At least 30% of the audience should come from the community and be explicitly marketed to them.

Reviewing average scores and compiled comments
Your program specialist will send compiled comments and scores from the other panelists two to three days before the panel meeting. When you review the written evaluation and average scores, be sure to take these steps:
– Make sure that the meaning/intent hasn’t changed when edited for spelling and clarity.
– Decide if you want to revise any of your scores and/or comments.
– Identify any application that you’d like to discuss at the panel meeting.
– Consider suggestions to improve the application or issues that you would like the Missouri Arts Council to address.

The panelists’ assessments vary. This is normal, and the Missouri Arts Council does not expect a consensus.

Access the MAC Grant Panel System

Logging In
– Log in at missouriarts.smartsimple.com.
– Enter your email and password that you received from your program specialist. If you are a current applicant in the SmartSimple system AND a panelist using your applicant email, use the same email and password you enter as an applicant.
– If you are a panelist AND an applicant you have two portals: Organization Contact and Reviewer.
– Select the Portals drop-down on the top, right side of the screen Select Reviewer.
– If you are a panelist only, you will see the review page after logging in.

The Review Page and Conflict of Interest
– You will see two sections on the screen, one for Pending Reviews and one for Completed Reviews.
– You can sort each column by selecting the heading: Application ID, Application Type, Organization, Primary Contact, or Status.
– In Pending Reviews, the status for each application will be Conflict of Interest until you begin the review process.
– Click Open to review the application. The Declaration of Conflict of Interest will appear. Click on Application Summary to view the application and determine if you have a conflict.
– If you have a conflict, select “Yes – Conflict Does Exist,” and your conflict will be recorded. A message will appear directing you to select the Home button in the top right corner of the screen to return to the Home page. If selected, the screen will show the organization in the Completed Review section with a Status of Failed Conflict Check.
– If you select “No – Conflict Does Not Exist,” it will bring you to the Review Form to begin the review and scoring process.

Review Form 
– Read the Instructions. Remember to select Save Draft button at the bottom of the screen.
– Click on Application Summary to view the application form and attachments. You may use the Download Arrow at the top right to save the application.

Adjusting Your Pop-Up Settings
If you experience problems accessing the application, adjust your browser’s pop-up settings.

Google Chrome: disable the pop-up blocker
– Open Chrome.
– At the top right, click More. Settings.
– At the bottom, click Advanced.
Under “Privacy and security,” click Site settings.
– Click Pop-ups and redirects.
– At the top, turn the setting to Allowed or Blocked.

Firefox: access the pop-up blocker settings
– In the Menu bar at the top of the screen, click Firefox and select Preferences.
– Click the menu button. and select Settings.
– Select the Privacy & Security panel.
– Go down to the Permissions section. Uncheck the box next to Block pop-up windows to disable the pop-up blocker altogether.

Microsoft Edge: add exceptions to the pop-up blocker
– Click the Settings and More button (three dots) on the upper right side of the browser.
– Click Settings.
– Click Site Permissions.
– Click Pop-ups and Redirects.
– Click Add.
– Enter https://missouriarts.smartsimple.com into the Address of website to allow field.
– Close and relaunch Edge.

Score Your Application
– Provide a score between 1 (lowest) to 10 (highest) for each criterion except General Comments.
– Add your comments under each criterion and any General Comments. Format the text by using the icons above the blank comments box.
– Select Review Summary to view your scores and comments for this application.
– You will not be able to complete the review without scores and comments.
– Should you need to leave an application at any time without completing the review, select Save Draft and the Home button. The Status under Pending Reviews will change from Conflict of Interest to Draft. At this point the review is incomplete. Select Open to resume the process.
– The total score will update after you select Save Draft.

Final Review
– Review everything one final time. Select Complete to finish the review process. A message will direct you to select the Home button in the top right corner of the screen to return to the next organization for review.
– The application moves to the Completed Reviews section of the Home page.
– Please note that the online grant system will automatically log out after 30 minutes of inactivity. Save frequently.

Now that you have completed your review of the assigned applications, you can see your scores and comments as well as your fellow panelists’ information online. Your name will not appear by the completed scores and comments. If a panelist had a conflict of interest, there will not be any scores and comments shown.

– Log in at missouriarts.smartsimple.com.
– Enter your email and password that you received from your program specialist. If you are a current applicant in the SmartSimple system AND a panelist using your applicant email, use the same email and password you enter as an applicant.
– You will see two sections on the screen, one for Pending Reviews and one for Completed Reviews.
– You will be using the Completed Review section on the right side of your screen.
– Begin with the first Completed application on the list. Click Open next to the Completed Status.
– Click All Reviews Summary (the third blue button). This will list the reviews for this application by all panelists unless there is a Conflict of Interest. Clicking Review Summary will only show your review of the application.
– Scroll down the page to the first panelist’s scores and comments. Since this is one continuous document, each time the word Criteria appears, it is the start of the next panelist.
– Each panelist’s review ends with the Total Score. It is calculated using the scores and percentages above.
– IMPORTANT NOTE: The Total Score is an average based on the number assigned for each weighted criteria by the panelist for each application. It is not the overall Average Score for the application.
– The Average Score is provided by the Program Specialist on a separate spreadsheet. This number is determined by averaging all panelists’ scores together.
– Continue scrolling down the page to review the remaining panel scores and comments for this application.
– When you are finished, click on the X in the upper right corner to close this review.
– Click on the blue arrow in the right corner to go to the next application reviews.
– If you would like to make any changes to your scores and comments, contact the program specialist. You cannot make any adjustments in the online panel review system at this time.
– To log out of the system, click on your name in the upper right corner of the screen.

Missouri Arts Council staff will coordinate the panel meeting conference call using Zoom.

The panel chair, a MAC Board Member, will begin the meeting by asking panel members and staff to identify themselves. The panelists on the phone will identify themselves when they speak to help staff record the panel meeting minutes. During the orientation, panelists will have an opportunity to update the conflicts of interest list. We use the most recent edition of Robert’s Rules of Order to guide the meeting.

The program specialist will present each application in application number order, announcing the application number, applicant name, and initial average score. If necessary, the program specialist may ask for clarification and/or corrections to the presented panelists’ written evaluation. They may also note any stipulations or recommendations.

The panel chair will moderate discussion and ask the following question after each application: Are there any comments or changes that would affect the average score, or any motions for recommendations or stipulations? If a panelist wants to change their applicant score they will announce that they are modifying their score to a specific score. No motion, second, or discussion is necessary. We will adjust the score in the system and calculate a new total average score.

After presenting all the applications, the panel makes a formal motion to approve all of the final average scores. This requires a second and then a vote.

The program specialist will review issues raised during the meeting. We will hold any ideas or questions come up during the panel meeting about policies, process, programs, services, or issues critical to the discipline, for later discussion. We welcome your suggestions and ideas.

The meeting ends with a formal motion to adjourn, marking the end of your duties as volunteer panelist. Your responsibility is completed. If applicants approach you for information about the panel review process, how they performed, etc., please advise them to talk to their MAC program specialist. This will protect you and the process, as well as ensure that we deliver a consistent message to all of the applicants.

Panelists will receive a survey by email asking for your feedback on the panel process, the applications, and recommendations for future panelists. The grant process has evolved with input from participating panelists.

Calculating award amount: MAC staff calculate the award amounts for MAC Board approval in June. The grant amount is determined by application request, applicant score as recommended by the panel, and available budget. This formula directly relates to the application scores, rewarding those applications that most clearly and directly address the criteria.

Notification: After the MAC Board approves the budget and panel-recommended funding and the Governor signs MAC’s budget bill, MAC sends written notification of the MAC board funding decisions to applicants. This usually occurs in early/mid-July. The award notification will include panel comments (not attributed to the panelist) to help applicants understand how their application was reviewed and how to make a stronger application. Also, if applicable, any panel recommendations and/or grant stipulations. The program specialist will also send the results to the panel.

Questions? If you have any questions, contact your program specialist directly. We are grateful to have your expertise as a panelist.

Avoid Conflict of Interest
Panelists who serve on a board of an organization applying for a grant or that would benefit from receiving a grant will declare conflict of interest and will remove themselves from the discussion. Panelists will be asked to declare any conflict of interests at the beginning of each application. Do not score any grant with which you have a conflict of interest. If there is a question about what constitutes a conflict, please consult with the panel chair or appropriate Missouri Art Council staff person.

Avoid Bias
Harmful attitudes or stereotyping affect our understanding, actions, and decisions in an unconscious manner. Discrimination can take the form of age, ability, ethnicity, religion, philosophical leaning, philosophical leaning, or political leaning. Both negative and positive bias exist.

Bias awareness and examples
– “Like Me” Effect where you identify characteristics of the application in yourself.
  Example: You favor a discipline or style, such as a personal preference for contemporary sculpture or classical jazz.
– “Halo / Pitchfork” Effect where your prior positive or negative impression influences the entire application.
  Example: “I was in a play with so and so and they are the house manager at X venue; I favor them.”
– “Stereotyping” where you form an opinion about an applicant’s gender, race, artistic discipline, etc. that could influence your evaluation.
  Example: “Art featuring the military is just thinly veiled advertisements.”
– “Academic or Nonverbal Bias” where your positive or negative evaluation is made based on the presentation of the application, for instance organized, instructions followed, media used, punctuation and sentence structure adherence, vocabulary.
  Examples: “This spreadsheet is terrific!” “I can’t find visual documentation on YouTube—what’s Vimeo?” “I’m really having trouble reading this locally hosted festival application; their leadership must not be good.”

Where does bias reside?
– Arts discipline
– Budget
– Fine art / Community art
– Gender
– Geography / Region
– Race and/or ethnicity
– Rural / Urban demographics

How do we avoid and limit bias?
– Be aware: be mindful and take your time.
– Slow down decisions: focus on the content of the application.
– Question rationale and reconsider.
– Declare the conflict if bias exists.