ROMA: Frequently Asked Questions (FAQs)
The information presented in this FAQs section is based on ROMA orientation/training sessions held for CAA boards at agency locations around the state in the summer of 2002.
The FAQs are organized by topic. Use the list below to jump to FAQ groupings of particular interest:
ROMA Real Fast: Just the Essentials
Q: In two sentences, what is ROMA?
Q: What does the term ROMA mean?
Q: Is ROMA just another fad?
Q: Is ROMA required for CAAs?
Q: Aside from requirements, why is ROMA important for CAAs?
Q: In two sentences, what is ROMA?
A: It's an approach for determining whether a CAA's programs are actually improving peoples lives. It emphasizes results over merely counting the number of services provided or people served.
Q: What does the term ROMA mean?
A: Results-Oriented Management and Accountability: It concerns managing programs to make it easier to conduct quantitative assessments about whether the expenditure of program funding helped those targeted by the program. It involves setting program goals, establishing baselines for where things stand, and then measuring progress on a regular basis.
Q: Is ROMA just another fad?
A: Only time will tell, but there's mounting evidence to the contrary. ROMA is a variation of the well-established "performance measurement" concept. This management approach--widely used in government and private industry--seeks to involve the entire organization in working toward broad goals and related interim targets that are focused on results (also called outcomes). In the case of CAAs, performance measurement (i.e., ROMA) is about generating data that indicate whether the services provided are improving people's circumstances.
Q: Is ROMA required for CAAs?
A: Implementation of the ROMA-approach to program management
is required of CAAs only for programs funded by Community Services
Block Grants (CSBG). These federal funds--managed at the national
level by the Office of Community Services (OCS) of the U.S. Department
of Health and Human Services (HHS)--are subject to requirements
under the Government Performance and Results Act of 1993. GPRA mandates
that federal agencies establish standards for measuring their performance
and effectiveness. In some cases, CSBG funding accounts for a relatively
small portion of a CAA's overall budget. In such cases, the ROMA
management and reporting requirements may be relevant for only a
handful of programs. But CAAs are encouraged to consider the benefits
of results-oriented management for their other programs. See Recommended
ROMA: Is It Good for You? below.
Q: Aside from requirements, why is ROMA important for CAAs?
A: Two words: clarity and learning. When applied across your CAA, this management approach can help the agency sharpen up its thinking about the desirable impacts of programs on the people and community served. And because ROMA relies on collection and use of measurement data, it makes it easier for the agency to learn from results and make programmatic adjustments while working toward stated goals. The resulting clarity about program objectives and effectiveness can help the CAA make a more persuasive case when marketing for future funding, especially in times like these when government budgets are tightening.
Required ROMA: When It's Mandated
Q: When is ROMA required for CAAs?
Q: What in general is required of CAAs reporting on use of CSBG funding?
Q: Are ROMA requirements likely to constrain CAA autonomy?
Q: What is the deadline for ROMA implementation?
Q: How are the CAAs doing on ROMA implementation?
Q: What about the specifics of reporting on CSBG-funded programs?
Q: When is ROMA required for CAAs?
A: CAAs are required to implement the ROMA-approach to program management only for programs funded by the federal Community Service Block Grants (CSBG) program, which is managed at the national level by the Office of Community Services (OCS). The results-oriented program management approach is required by the Government Performance and Results Act of 1993 (GPRA), a law mandating that federal agencies establish standards for measuring their performance and effectiveness.
Q: What is required of CAAs reporting on use of CSBG funding?
A: To comply with federal requirements, Massachusetts CAAs must provide performance-related program information about the use of CSBG funding on an annual basis to the state's Department of Housing and Community Development (DHCD). As the state's administering agency for CSBG funds, DHCD collects this information from the CAAs and incorporates it into its reports to the OCS. The CSBG-related data that CAA must provide annually include fiscal, client, demographic, and program outcome data.
Q: Are ROMA requirements likely to constrain CAA autonomy?
A: The requirements merely establish a framework for measuring program performance. It's up to each CAA to establish program-specific goals and identify desirable outcomes that would address the needs of the particular community.
Q: What is the deadline for ROMA implementation?
A: The OCS has been working with CAAs around the country with the objective of achieving full implementation of ROMA by September 30, 2003.
Q: How are the CAAs doing on ROMA implementation?
A: CAAs in Massachusetts are in the process of providing the state with the required information for CSBG reporting. And the state is on track for achieving full compliance by the end of fiscal year 2003. As a follow-up effort, the state is providing the CAA directors and boards with information for determining whether the results-oriented management approach might make sense for overall management of agency programs.
Q: What about the specifics of reporting on CSBG-funded programs?
A: For details on meeting CSBG reporting requirements, refer to your CSBG grant documentation or contact:
Sandra Hawes, Director of the Bureau of Neighborhoods
Massachusetts Department of Housing and Community Development
One Congress Street, 10th Floor
Boston, MA 02114
Phone: (617) 727-7004 x521
Fax: (617) 727-4259
Email: sandra.hawes@state.ma.us
Recommended ROMA: Is It Good for You?
Q: Why use the results-oriented management approach when it isn't required?
Q: Have we been counting the wrong things all these years?
Q: What about program assessment criteria specified by funding organizations?
Q: Are you suggesting that ROMA-type measures be handled separate from criteria specified by a funding organization?
Q: How rigorous should the measures be?
Q: Don't we run the risk of being held hostage by a framework of our own creation?
Q: What's in it for the CAA's Board of Directors?
Q: What's in it for the program manager?
Q: What's in it for the local community?
Q: How long does it take to start getting useful data?
Q: What's the key to presenting the data?
Q: What would be an example of presenting data in a hierarch for a direct-service program?
Q: Why use the results-oriented management approach when it isn't required?
A: Because results-oriented management seeks to answer a question that is fundamentally important for all CAA programs: Do the things we do work? That is, are programs bringing about a change in people's conditions or behaviors? For example, are people in computer training programs getting jobs that call for these new skills? Are people in drug-prevention programs saying "no" to drugs? If so, how many successes in relation to the baseline or population of people being targeted for such programs?
Q: Have we been counting the wrong things all these years?
A: Not the wrong things. But maybe not the things that provide the greatest clarity and understanding about whether program efforts are changing peoples lives for the better. Typically, program assessments are based on a count of how many units of service have been provided. An important difference with the results-management approach involves establishing agreed-upon goals that speak to desirable outcomes, and then regularly measuring progress toward achieving those goals. It might be necessary to make adjustments along the way based on interim results, but the focus stays on effecting change in people's circumstances. In that way, ROMA is about much more than counting people served or services provided.
Q: What about program assessment criteria specified by funding organizations?
A: While important, such criteria often are not what the CAA itself would use to define program success in terms of the needs of the specific community served and the longer term outcome goals of the agency. That's why it's important to go beyond such generalized measures. Indeed, often the funding organizations themselves are uncertain about what a program's outcomes should be. To achieve that kind of clarity, the program manager needs to step back from near-term requirements and consider what sorts of results would be meaningful. That is: What outcomes would indicate that the circumstances of individuals within the targeted group would be improved by the program?
Q: Are you suggesting that ROMA-type measures be handled separately from criteria specified by a funding organization?
A: It might need to be. But, not necessarily. Some requests for proposals are more flexible than others. In such cases, the program manager may be able to write results-oriented assessment criteria into the agreement. Establishing community-specific measurement targets that are performance related can be a way of impressing upon the funding source that the money will be well spent. A forward-thinking plan can indicate that the program manager has a clear understanding of goals and objectives and is interested in learning whether the program is effective. And, taking this approach will allow the funding organization to anticipate receiving outcome-related information from the investment-such information can be equally valuable for the program and the funder.
Q: How rigorous should the measures be?
A: Outcomes are about settings targets. Experience indicates that agencies that set ambitious targets often get better results from their programs. And setting ambitious targets tends to drive quicker learning about what does and doesn't work.
Q: Don't we run the risk of being held hostage by a framework of our own creation?
A: Results management done well is liberating. The challenge is to step back, think about what outcomes have meaning for your community, establish goals and targets, and then rewrite work plans so that program assessments reflect what is or isn't working. What's liberating about this approach is that the more the agency knows about what it's doing and why, the stronger its position. It's far preferable for an agency to determine what it should be doing and what it does well, than for an outside entity to be dictating to the agency how it should work with its local community.
Q: What's in it for the CAA's Board of Directors?
A: Information generated from results-oriented program management enables board members to efficiently grasp the effectiveness of operations across the agency. With an understanding of what is or isn't working on the program level, they are able to ask informed questions of the agency director and the program managers, help guide policy making, and focus the agency's strategic goals. In this way, board members can help CAA staff be better managers. Board members can be thought of as internal investors who use outcomes-related data to intelligently assess community needs and to make informed determinations about how best to target available resources. Also, knowing more about program results, helps board members fulfill their responsibility to advocate for the agency within the community and to market the agency to funding organizations.
Q: What's in it for the program manager?
A: Clarity and understanding about the program. Asking tough
questions--Are we focusing on the right problems? Is the program
making a difference for people's lives? Are adjustments needed to
be more effective? Should we refocus our efforts?--allows program
management to be a dynamic and engaging process. Having quantitative
data makes it possible to assess program effectiveness and react
to the results-oriented information. Ultimately, this kind of knowledge
is likely to yield greater satisfaction in working with the local
community. And, it enables the program to tie in with the larger
goals and objectives of the agency. This is true whether the program
is providing direct services or conducting advocacy.
Q: What's in it for the local community?
A: Measurable goals and targets can establish a context that supports the community's commitment to the success of agency efforts. When the community understands how issues of common interest are being addressed and what progress is being made, individuals are more likely to get involved. Even individuals who are the focus of direct-service programs may be motivated by understanding how they fit into the larger picture.
Q: How long does it take to start getting useful data?
A: It can take 2 to 3 years to start generating numbers that tell you something about the effectiveness of a program. Once baseline data reflecting agency and program goals are established, it's usually necessary to collect a couple years of follow-up data in order to start identifying trends.
Q: What's the key to presenting the data?
A: When reporting results, it's a good idea to provide different slices of the data. But be organized and concise, so that you present the picture you intend. For instance, present data in a hierarchy that places big-picture or aggregate goal measures in relation to more specific supporting measures. This presentation of data will reflect the underlying framework of the results-oriented management approach, which consists of broad goals that have been translated into performance targets and milestone measures. Also, rather than reporting every bit of data collected, a presentation of summary data can be the most effective approach for telling the story, without overwhelming the audience with a flood of numbers.
Q: What would be an example
of presenting data in a hierarchy for a direct-service program?
A: An employment-related program might show results measurement values such as the following: 100 individuals were advised over the phone; 80 were advised onsite; 75 participated in skills training; 50 developed resumes; 45 responded by mail to job openings; 30 had job interviews with prospective employers; 20 were hired; and 15 were still in the job after six months. The outcome of interest might be the last number (individuals still employed after six months), with the other numbers supporting this result as milestone measures.
More on ROMA
Q: How do I get more information about ROMA and its implementation?
A: To learn more, see the Outside
Links area of this Web site.
You also can send questions/comments about ROMA to MASSCAP from this Web site, using the form on the Contact Us page.
Also, Sandra Hawes at DHCD can be contacted at: sandra.hawes@state.ma.us
|
