Virginia Inter-Rater Reliability (IRR) Program

Tools and Resources

Virginia Medicaid – Department of Medical Assistance Services (DMAS) contracts with Maximus to develop a baseline reliability study on its Universal Assessment Instrument. This study will demonstrate the degree of consistency and reliability of long-term care screenings for children and adults, verifying that the tools and training are effectively and consistently applied for the benefit of individuals across the Commonwealth, regardless of region or screener. This is a study only, and outcomes for individuals will not change. 

The Inter-rater Reliability (IRR) Study seeks to demonstrate the degree of consistency and reliability of long-term care assessments for children and adults, as well as verification that individuals have the same opportunity to obtain services regardless of where they live or when a screen is conducted and ensure that screeners are identifying the most integrated and appropriate settings, minimizing unnecessary institutional placements.

Stay connected to important program updates and helpful program resources. If you aren't currently on the Virginia IRR contact list, please reach out to the Help Desk with your name, title/role and facility name: VAIRR@maximus.com

Contact the help desk
Phone & Email

833.73 VAIRR (833.738.2477)

VAIRR@maximus.com

Business Hours

8:00 am - 5:00 pm EST

Announcements

  • Virginia Inter-rater Reliability (IRR) Study: Screener Leave Behind Reminder | Meet Clinical Reviewer, Ruth Robertson-Gouge

    REMINDER - PLEASE SHARE THE LEAVE BEHIND DOCUMENT AFTER EACH SCREENING TO ENCOURAGE PARTICIPATION IN THE STUDY

    You can find the link for that document here.


    MEE THE TEAM: Ruth Robertson-Gouge, Clinical Reviewer

    Responsibilities: Ruth covers screenings in the Piedmont and some in Central VA and Southwest VA.

    Background: Ruth has lived in several different places in Southwest VA: Saltville, Fancy Gap, Pulaski, Richlands, and Catawba/Roanoke. She was raised in Florida and has previously lived in Georgia and Tennessee as well.

    Fun Fact: Ruth enjoys meeting all the people that we encounter in our work. Seeing what others are experiencing gives you compassion and respect for people. She has admitted to being called a “Cat Lady” in the past, but as of now, she only has one, who showed up at her door on Christmas Eve. The new kitty has ever won over husband who claims to hate cats.


    SUPPORT: IRR Study Help Desk

    Have questions about the IRR study? Contact the IRR Study Help Desk team for more information:

    Email: VAIRR@maximus.com
    Phone: 833.73 VA IRR (833.738.2477)

  • Virginia Inter-rater Reliability Study: Observational IRR Screenings Now Being Conducted & Meet Clinical Program Supervisor, Lakel Gurley-Upshaw

    Observational IRR Screenings Have Now Begun
     
    Maximus, in partnership with the Virginia Department of Medical Assistance (DMAS), launched the conducting of Observational IRR screenings earlier this month. These are LTSS screenings performed with an IRR Screener from Maximus observing the completion of the individual LTSS screening. 

    How many LTSS screenings will be selected? Only a small percentage of overall LTSS Screenings will be selected for Observational Screenings. Should one of your LTSS screenings be selected, a Maximus Screener will contact you to coordinate a time to complete the Observational screening with you. It is very important that you participate in any requests for he Observational IRR process, which is a part of the overall IRR Screening study that Maximus is conducting for DMAS. 

    Will this affect the LTSS screening determination? No. No Observational screening will not in any way impact the LTSS Screening determination, nor will there be any evaluation of the LTSS screener.


    REMINDER - PLEASE SHARE THE LEAVE BEHIND DOCUMENT AFTER EACH SCREENING TO ENCOURAGE PARTICIPATION IN THE STUDY.
     
    You can find the link for that document here:
    https://maximusclinicalservices.com/sites/default/files/pasrr/documents/VA-IRR-Original-Screener-Leave-Behind-122021.pdf
     

    Meet the Team: Lakel Gurley-Upshaw, Clinical Program Supervisor

    Responsibilities: Completing screenings in the eastern region of VA.

    Background: Lakel lives in Newport News, VA. She loves meeting so many different people, hearing their stories, and seeing the impact waiver services has on in their quality of life. 

    Fun Fact: In her free time, she loves to travel and is currently working on writing a book. 


    SUPPORT: IRR Study Help Desk

    Have questions about the IRR study? Contact the IRR Study Help Desk team for more information:

    Email: VAIRR@maximus.com
    Phone: 833.73 VA IRR (833.738.2477)

  • Virginia Inter-rater Reliability (IRR) Study: Maximus to begin Observational IRR

    New Observational IRR Screenings to Begin July 2022

    Starting in July 2022, Maximus will begin conducting Observational IRR screenings. What is an Observational IRR screening? An Observational IRR Screening is a LTSS screening that is done with an IRR screener from Maximus observing the completion of the LTSS screening. It is very important that you participate in the Observational IRR process which is a part of the overall IRR Screening study that Maximus is conducting for DMAS. A very small percentage of overall LTSS screenings will be selected for Observational Screenings. Should one of your LTSS Screenings be selected, a Maximus Screener will contact you to coordinate a time to complete the Observational screening with you. Please note that this Observational screening will not in any way impact the LTSS Screening determination, nor will there be any evaluation of the LTSS screener.
     

    SHARE YOUR FEEDBACK:  Take the Screener Survey

    Maximus is still collecting Screener input on the Screener Survey. Your anonymous answers will be used to help make recommendations to DMAS in the final study. The survey should take less than 10 minutes to complete.
     

    CONTACT: IRR Study Support

    Have questions about the IRR study?  Contact the IRR Study Help Desk team for more information:

    Email: VAIRR@maximus.com
    Phone: 833.73 VA IRR (833.738.2477)

  • Virginia Inter-rater Reliability (IRR) Study: Take the IRR Study - Screener Survey + Review Support Resources

    Work continues in the first phase of the Virginia Department of Medical Assistance (DMAS) Inter-rater reliability (IRR) study with clinical services company, Maximus. As shared earlier in 2022, this study was mandated by the General Assembly. 

     

    SHARE YOUR FEEDBACK: Take the Screener Survey 

    Maximus is also collecting Screener input on the assessment process through a survey. Your anonymous answers will be used to help make recommendations to DMAS in the final study. The survey should take less than 10 minutes to complete. Click here to take the survey

    Tool: Screener Leave-Behind | Download
    This handy printout (2 copies per page) provides the persons you assess with some brief information about the IRR study and lets them know that a follow up screening may potentially be scheduled, as well as offering options for them to learn more.
     
    Guide: Virginia IRR Study Overview | Download
    A one-page overview including the purpose, processes and planned goals for the study.
     
    Guide: Frequently Asked Questions (FAQs) | Download
    This resource guide covers the questions most often fielded about the IRR study and its related processes. This will continue to be updated as we move through the year.
     
    Recording: DMAS Webinar | Listen
    A recording of a recent webinar with DMAS/Maximus reviewing the Inter-rater Reliability Study being conducted by Maximus is available.
     

    CONTACT: IRR Study Help Desk

    Have questions about the IRR study? Contact the IRR Study Help Desk team for more information:

    Email: VAIRR@maximus.com
    Phone: 833.73 VA IRR (833.738.2477)

     

  • Virginia Inter-rater Reliability (IRR) Study: Review Recording of Recent DMAS Webinar | Take Screener Survey

    REVIEW: Recording of Recent DMAS Webinar Now Available

    A recording of the recent webinar with DMAS/Maximus reviewing the Inter-rater Reliability Study being conducted by Maximus is available. 

    Click here to review.
     

    SHARE YOUR FEEDBACK: Take the Screener Survey 

    Maximus is also collecting Screener input on the assessment process through a survey. Your anonymous answers will be used to help make recommendations to DMAS in the final study. The survey should take less than 10 minutes to complete. Click here to make your voice heard. 
     

    CONTACT: IRR Study Support

    Have questions about the IRR study?  Contact the IRR Study Help Desk team for more information:


    Thank you so much for your support working with Maximus screeners to provide contact information and supporting documentation used during the original screening.

  • Virginia Inter-rater Reliability (IRR) Study: Take Screener Survey | Register for Upcoming Jan 26 - 27 Webinars | Review Available Resources

    We are now a few weeks past the recent 1/3/22 launch of the Inter-rater reliability (IRR) study by the Virginia Department of Medical Assistance (DMAS) and state partner, Maximus. A sincere note of thanks for your continued support and assistance during this successful launch process and as we look ahead toward the future of this impactful study. As shared in recent communications, this IRR study was mandated by the General Assembly for: 
     

    • Validation of the children's criteria used with the UAI instrument to determine eligibility for Medicaid Long Term Services and Supports (LTSS
    • Design and implementation of an inter-rater reliability test for the pre-admission screening process

    Continue reading below for opportunities to learn more, including details on a new survey developed to learn more about your processes, registration details for IRR Overview Webinars taking place on January 26-27, as well as some insightful FAQs, handy resources and support options. 
     

     


    ENGAGE: Share Your Feedback in Screener Survey

    The IRR Study team would like to hear from you. A new survey has been developed to bring more insight into your existing screening processes. Click here to share your feedback


    LEARN MORE: Attend IRR Overview Webinars 
    Wednesday, Jan 26 - Thursday, Jan 27


    If you’ve yet to do so, be sure to register to join the Maximus Training Team for one of two insightful webinars, which will take attendees through the key aspects of the IRR study. The end of each hour-long session will be set aside to field any additional questions. Click the link below and select the date/time that work best with your schedule. 

    Inter-rater Reliability (IRR) Overview | 9:30 a.m. ET Wed, 1/26/2021

    Inter-rater Reliability (IRR) Overview | 2:00 p.m. ET Thu, 1/27/2021


    FAQs: Existing Screens and the Re-Screening Process

    In the weeks leading up to the recent launch, the Program Team developed a comprehensive list of frequently asked questions (FAQs) that screeners and other state stakeholders might have about general processes related to the IRR Study. Read on to review some specific details related to the existing screens and the re-screening process or click here to download a complete list of FAQs

    Q: If an IRR re-screening is conducted, is the original screener present? 
    A: No, the IRR re-screening is conducted by Maximus staff alone in the initial part of the study. Observational IRR will include silently observing a live screen with the initial screener present. That portion of the study is not expected to begin until mid-year 2022. 

    Q: How will Maximus know if answers are the same or different from the original screening? 
    A: The Maximus IRR re-screener will not know the original LTSS screening results. The initial screening is not to be shared with the Maximus IRR screener. The data from the original screening will be compared electronically after completion of the IRR rescreen. 
     
    Q: Will the results of the IRR re-screen by Maximus be provided to the original screener?  
    A: No. The focus is on systemic, big picture screening trends to demonstrate the degree of 
    consistency and reliability of LTSS screenings for children and adults, not on individual 
    screenings. 

    Q: When the study is complete and the report is published, will specific screeners be identified as performing well or not well? 
    A: No individual screeners will be identified. The report will address LTSS screening observations by region and statewide. 
     

    RESOURCES: Reference Materials for the Screener

    Ensuring that you have access to all the necessary tools and supports is critical to help ensure the successful implementation and the findings that will be gathered as a result of the IRR study. Check out a few helpful guides below that have been developed with Virginia’s Screeners in mind.  
     

    Tool: Screener Leave-Behind | Download
    This handy printout (2 copies per page) provides the persons you assess with some brief information about the IRR study and lets them know that a follow up screening may potentially be scheduled, as well as offering options for them to learn more.

    Guide: Frequently Asked Questions (FAQs) | Download - Updated 1/14 
    This resource guide covers the questions most often fielded about the IRR study and its related processes. This will continue to be updated as we move through the year.

    Guide: Virginia IRR Study Overview | Download
    A one-page overview including the purpose, processes and planned goals for the study. 


    CONTACT: IRR Study Support

    Have questions about the IRR study? Contact the Help Desk team for more information:

    Email: VAIRR@maximus.com

    Phone: 833.73. VA IRR (833.738.2477)

  • Virginia Inter-rater Reliability (IRR) Study: DMAS Study Officially Launched on January 3 | Register for Upcoming Webinars | Review Supports & Reminders

    Monday, January 3 marked the official launch of the Inter-rater reliability (IRR) study through the Virginia Department of Medical Assistance (DMAS) and contractor, Maximus. As shared in previous communications, this IRR study was mandated by the General Assembly as a review of Virginia’s LTSS screening process with several key goals in mind, including:

    • Validation of the children's criteria used with the UAI instrument to determine eligibility for Medicaid Long Term Services and Supports (LTSS) 
    • Design and implementation of an inter-rater reliability test for the pre-admission screening process


    Continue reading below for insights into some upcoming learning opportunities, frequently asked questions (FAQs), available resources and Help Desk support pertaining to the IRR study. 

     

    Timeline-VA-IRR-2.png

     


    REGISTER: IRR Overview Webinars | Jan 26 - 27

    Would you like to learn more about this important new study? Be sure to join the program team at Maximus for one of these hour-long overview sessions, as they walk through the key aspects of the IRR study. There will also be a few minutes set aside to answer attendee questions. Click the link below and select the date/time that work best with your schedule.

     



    FAST FACTS: Understanding the IRR Study Process

    In the coming months we'll continue to share some frequently asked questions about the IRR study and your role in the process. Below are four fast facts focused on how this new initiative will work.

     

    1. How will the study be conducted? Individuals will be randomly selected across regions, to include both adults and children.  
       
    2. How will rescreens be selected? Rescreens will be randomly chosen from a statistically significant sample of conducted screens and analyzed based on assessment type (child vs. adult), region, and screening entity (NF, hospitals and CBTs).   
       
    3. Who will perform the secondary screening? All screeners working with the IRR study will have taken and passed the VA LTSS modules.   
       
    4. How will the rescreening process work? Maximus will conduct face-to-face screenings for the Baseline IRR Study using two techniques:
    • Screen/Rescreen—Entails repeating a subset of required questions within a short time frame after the initial screen. This will begin in January 2022 
       
    • Observational IRR will include silently observing a live screen with the initial screener. This is expected to begin mid-year 2022.

     

    TOOLKIT: Screener Resources

    Ensuring that you have access to all the necessary tools and supports is critical to help ensure the successful implementation and the findings that will be gathered as a result of the IRR study. Check out a few helpful guides below that have been developed with Virginia’s Screeners in mind. 

    Tool: Screener Leave-Behind | Download
    This handy printout (2 copies per page) provides the persons you assess with some brief information about the IRR study and lets them know that a follow up screening may potentially be scheduled, as well as offering options for them to learn more.

    Guide: Virginia IRR Study Overview | Download
    A one-page overview including the purpose, processes and planned goals for the study.

    Guide: Frequently Asked Questions (FAQs) | Download
    This resource guide covers the questions most often fielded about the IRR study and its related processes. This will continue to be updated as we move through the year.



    CONTACT: IRR Study Support

    Have questions about the IRR study and can't find the answer in the resources listed above? Contact the Help Desk team for more information:

    Email: VAIRR@maximus.com

    Phone: 833.73. VA IRR (833.738.2477)

  • New DMAS Program Implementation with State Partner, Maximus, Coming January 3

    The Virginia Department of Medical Assistance (DMAS) has contracted with clinical services company, Maximus, to launch an Inter-rater reliability (IRR) study beginning on Monday, January 3rd
     
    The General Assembly has mandated this initiative as a review of Virginia’s LTSS screening process. Once implemented, this study will demonstrate the degree of consistency and reliability of long-term care screenings for children and adults, verifying that the tools and training are effectively and consistently applied for the benefit of individuals across the Commonwealth, regardless of region or screener type.  
     

    As the state prepares to implement this program, below are answers to some key questions about the IRR study, the project’s timeline, as well as available supports and resources you might find helpful. 
     

    What is IRR?  
    IRR, or Inter-rater reliability, refers to the extent to which two or more individuals agree. The process will be used to evaluate the consistency of processes, rating decisions and outcomes. The study will validate the children’s criteria and determine the consistency of screening across populations. 
     

    How will the study be conducted?  
    Individuals will be randomly selected across children, adults, regions. 
     

    How will rescreens be selected?  
    Rescreens will be randomly chosen from a statistically significant sample of conducted screens and analyzed based on assessment type (child vs. adult), region, and screening entity (NF, hospitals and CBTs).  
     

    How will the rescreening process work? 
    Maximus will conduct face-to-face screenings for the Baseline IRR Study using two techniques: 

    1. Screen/Rescreen—Entails repeating a subset of required questions within a short time frame after the initial screen. This will begin in January 2022 
    2. Observational IRR will include silently observing a live screen with the initial screener. This is expected to begin mid-year 2022.  


    Who will perform the secondary screening? 
    All screeners working with the IRR study meet or exceed requirements and will have taken and passed the VA LTSS modules. 
     

    How many individual screenings are estimated to go through a secondary screening? 
    Out of approximately 40,000 LTSS screenings a year Maximus anticipates sampling just over 3% 
     

    Will the secondary screenings affect findings from the initial screen? 
    No. DMAS and Maximus recognize the importance of minimizing disruptions to normal processes. This is a study only, and outcomes for individuals will not change. No part of this study will negatively impact a completed screening and no changes will be made to any previously completed screenings, outcomes or services received. 
     

    What are key dates ahead for this initiative? 
    There are a few milestone dates to keep in mind for the IRR study including: 

    • IRR activities (screen/rescreen) will begin on January 3, 2022 
    • Observational IRR-related activities will take place mid-late 2022 
    • Baseline study recommendations due in the fall of 2022 


    How will the study results be used? 
    At the close of the IRR study, the results of this instrument-focused initiative will serve as a foundation for several important next steps:

    • Ensure reliability across different settings 
    • Assure confidence in the LTSS screening process  
    • Inform best practices ad develop training materials (if required) 
       

    What will Maximus need from screeners?
    Maximus may contact you to request certain information and copies of the supporting documentation you used during the original screen.
     

    Are we to add Maximus to our consent to exchange information, or are they covered under DMAS?  
    Maximus is included in Business Associate Agreement (BAA) for DMAS. 
     

    Should initial screenings be faxed to Maximus?
    No, the initial screening should not be faxed or emailed to Maximus.
     


    Program Support

    Ensuring that stakeholders across the Commonwealth have access to the information they need about the IRR study will remain a critical goal throughout this process. The Help Desk contacts and supports shown below are now available for any questions regarding IRR study processes, next steps and expectations. Communication updates and will be shared by the state and Maximus as the study moves forward in the months ahead. 

     
    Help Desk Contacts 
    VAIRR@maximus.com | 833.73 VAIRR (833.738.2477) 


    Virginia IRR Program Overview
    Download a one-page overview here to learn more.
     

    About Maximus

    Maximus is nationally recognized for its work with states across the country to manage complex programs and ensure that individuals receive the appropriate care and supports necessary to improve their quality of life. Enduring commitment to process innovation and service excellence has led the company to a role as the industry’s leading provider of high-quality, conflict free assessment services for persons with intellectual, behavioral health, and developmental disabilities, as well as older adults and individuals with physical disabilities and complex medical conditions. Learn more about Maximus services and capabilities at: https://maximus.com/clinical-services