Why Is RESNET Limiting Its Options for Improvement?
In case you haven’t heard, last year RESNET created a task force to look at ways to improve the consistency of results among certified home energy raters (aka HERS raters). One week ago, executive director Steve Baden sent an email about the latest developments in the process. He also attached a document titled Quality Improvement Working Group RESNET QA Options (pdf) that outlines the areas of consensus and gives three options for changing the way quality assurance (QA) is done. Unfortunately, there’s a glaring omission from the list.
In case you haven’t heard, last year RESNET created a task force to look at ways to improve the consistency of results among certified home energy raters (aka HERS raters). One week ago, executive director Steve Baden sent an email about the latest developments in the process. He also attached a document titled Quality Improvement Working Group RESNET QA Options (pdf) that outlines the areas of consensus and gives three options for changing the way quality assurance (QA) is done. Unfortunately, there’s a glaring omission from the list.
The need for improvement exists
We’ve been a HERS rating provider, now called HERS QA provider, since 2009. We serve independent raters from Arizona to Maine, and we make sure our raters know what they’re doing and do things right. Because we don’t let them get away with not meeting the HERS Standards or ENERGY STAR new homes program guidelines, we hear their complaints about raters with other providers who aren’t such sticklers. Here’s one I got last month:
It’s getting more and more frustrating on our end and is to the point where [one local HVAC company] is not wanting to bid on projects with a builder that has [our company] as the raters. Meanwhile, [that HVAC company] is pushing paperwork / installation issues through other raters that are apparently not having any issues getting Energy Star.
Yes, there are problems. Not all are egregious violations. Part of what’s motivating RESNET is that if you hire several raters to come into a home to do a rating, you’ll get a spread of results that’s broader than it should be. Apparently, some of the big production builders that build homes in different markets have experienced this when they build the same home in a similar climate and get different results when different raters rate the home.
When a provider does QA on their raters, the results for the HERS Index are supposed to be within 3%, but different providers and QA Designees (QADs) don’t always interpret the standards the same way. Each one has their ways of doing things and might emphasize or de-emphasize certain aspects of the rating process. So you can have variance in the results even without malfeasance.
RESNET’s oversight of providers has improved but…
The HERS rating industry is set up to have QA of raters done by their providers. Providers are accredited by RESNET, which oversees and does QA on the providers. Energy Vanguard Energy Ratings has to submit an annual report to result documenting the results of the QA we do. In our first three years as a provider, however, RESNET wasn’t doing a lot of oversight. One of those years, we never even got a report back from RESNET about how we did with our annual report. Also, despite RESNET having guidelines calling for enhanced oversight or field visits to providers, we haven’t had any of that. (We’re getting our first online review this fall.)
That’s definitely changed, though. Laurel Elam took over as QA Manager for RESNET, and she’s on the ball. In the past two years, she’s not only looking carefully at the reports submitted by all providers, she’s acting on what she finds. Here’s what a recent email from RESNET announced about the process:
The reviews found numerous instances where providers fell short of complying with RESNET quality assurance provisions. As a result 43 Rating Quality Assurance Providers were placed on administrative probation. An additional 56 Providers had findings that needed correction to be compliant with RESNET Standards.
That’s great! I’m happy to see that RESNET is starting to hold providers to the requirements in the HERS Standards.
But, it’s still not enough. Laurel is doing a wonderful job and RESNET is lucky to have her, but she’s not a technical person. She’s never been a HERS rater and isn’t a QAD. I don’t think the QA Manager needs to have those qualifications, but RESNET has to have technical people involved in the oversight process. They do have at least one technical person, Abe Kruger, who assists on a contract basis, but he’s not fulltime.
What RESNET has been doing over the past two years since Laurel took over is the important step of making sure the t‘s are crossed and the i‘s dotted. What RESNET has not been doing is looking at the words and seeing if there should even be a t or an i in them or maybe if they should have used a different word altogether.
In other words, RESNET needs to dive in deep enough to determine, not only if providers are doing enough QA and keeping records properly, but also if their QA process is good enough to assure the quality of ratings. A big step here would be to repeat some of the rating data file QA done by providers on raters.
Providers are required to do this for 10% of all ratings done by a rater, so RESNET, it seems to me, should be doing this for some percentage of every provider’s file QA. RESNET wouldn’t have to travel for this, and I think it would turn up a lot of problems and force all providers to make sure they do it right.
The 3 options
Back to the Quality Improvement Working Group now. This process has been going on since last year, and they recently released a document outlining the issues and their options for improvement (pdf). Here they are:
- RESNET staff do all QA on raters and ratings.
- RESNET creates a pool of contractors to do all QA on raters and ratings.
- RESNET contracts with a pool of third-party regional QA contractors to do all QA on raters and ratings.
I can’t see RESNET going with number 1. They probably won’t want to do the work involved with number 2 either. I think, if they continue down this path, they’ll probably go with number 3, and we’ll have ICF, CSG, and ClearResult doing all the QA.
All three of these options take QA away from the current providers. The task force and working group have said they want all QADs to act as “agents of RESNET.” Providers will still exist but will not do QA anymore.
It’s an important document for the future of the HERS rating industry. Please read it, think about it, discuss it, and participate in the public comment process. Unfortunately, today is the deadline for public comment, so don’t delay.
The glaring omission
Let me state again that this process is necessary. RESNET definitely needs to do something improve the consistency and quality of the HERS ratings being done. The list of consensus items in their recent document has some good points. For example:
“QA contractors should not be performing QA on Raters and ratings where a conflict of interest exists.”
“The RESNET Home Energy Rating and QA processes must be more clearly
defined to achieve greater consistency.”
“Qualifications of the companies and/or individuals who provide QA services,
including accountability, must be more stringent and more clearly defined to
create greater consistency and quality in how QA is delivered.”
Unfortunately, they seem to think that because the majority of providers also do ratings, no providers should be doing QA. I beg to differ. There are a few of us who are providers but do little or no ratings. Energy Vanguard does a handful of ratings per year, nearly all for Habitat for Humanity, many of them pro bono. We made the decision early on not to compete against our raters. Building Efficiency Resources (The BER) is another company with this model, and so is Southface.
Why is RESNET ignoring the benefits of this model? Rather than including only options that take QA away from providers, why not have a fourth option that allows providers to do QA but not on any ratings they do themselves? This is the fox-in-the-henhouse issue people have been talking about for years because of all the big providers that do their own QA.
This model can work, but it requires RESNET to take the next step I described above and look not only at providers’ administrative compliance but also technical compliance. As a member of the RESNET QA committee, I’ve been saying this over and over. The current structure can work but RESNET needs to do more thorough QA on providers.
What happens when QA is taken away from providers?
If RESNET continues down the current path and chooses one of the three options above (probably the third), why would anyone want to be a provider? The big rating companies will continue, of course, but there’s not much in it for small providers like ours. We’ll become document chasers and discipline enforcers, and I have no interest in that.
If RESNET chooses one of the three options above, I think QA could actually get worse for independent raters. Many providers do more than the minimum QA. In our providership, we look at nearly every REM/Rate file that gets sent to us, not just the 10% required by RESNET. Will the new QA companies do that?
And providers won’t know as much about QA if they’re not doing it. When raters call them up to ask questions, providers won’t know as many of the answers. They may even steer raters in the wrong direction because the QA companies may do things differently.
The collapse of QA
Another problem with the three options above is that it could lead to the collapse of the QA process from three levels to two. Right now it’s RESNET => providers => raters. It seems like RESNET thinks that handing all the QA off to “independent” QA companies (or RESNET staff) will solve all the problems so the collapse won’t matter. I think they’re wrong. Let’s keep three levels and beef up RESNET’s QA, which will reverberate all the way down.
I don’t think the folks on the task force and the working group have fully considered all the options or the consequences.
Two changes that could make all the difference
I hope you’ll comment on this issue before the deadline (tonight, 8/29). If you want to comment but can’t spend a lot of time, you could simply tell RESNET to consider the following two changes:
- Providers should not be allowed to do QA on their own ratings.
- RESNET should do more technical QA on providers.
The latter is what I was talking about above when I said RESNET should redo some percentage of each provider’s rating data file QA. Combined with the great job Laurel Elam has been doing with administrative QA, this could would allow RESNET to discover problems that might otherwise never be found.
Let’s get it right
If you’re a home energy rater, QAD, or provider, please get involved in this process. Read the documents. Check out the resources I’ve listed below. And most of all, participate in the public comment process. It ends today.
External Resources
Submit a comment to RESNET — Deadline 8/29!
RESNET quality improvement process
The BER’s webinar on an alternative path
The BER’s white paper on quality improvement
Texas HERO’s response to RESNET’s quality improvement process
Related Articles
RESNET Bungles the Adoption of Combustion Safety Testing
The Tail-Wagging Labradors of RESNET
Everything You Ever Wanted to Know about the HERS Index
NOTE: Comments are moderated. Your comment will not appear below until approved.
This Post Has 13 Comments
Comments are closed.
We have once again come full
We have once again come full circle. Since we can’t trust unlicensed contractors … we have licensing … since we can’t trust licensed contractors we have raters … now we can’t trust the raters.
I’m reminded of the final scene from the movie ‘Enemy of the State” …. “Who’s going to monitor the monitors?”
I appreciate the personal
I appreciate the personal attention my Ratings get from a provider that knows me. I know the QAD may check some items that I consistently have problems with and not pay as much attention to others. A 3% variance should be easy for almost anyone.
The idea that, as an independent business, I would be assigned to some large company for QA is scary at best. A smaller QA provider that knows me, has done some of my training, and can work with me to offer a quality service to my builders is important.
Things like standardized libraries, check lists, and spot checks on the QA process would help move individual raters to a more standardized result.
We hear a lot about QA on raters. What about QA on the QAD. This seems to be completed, and to most in the field this is a huge black box. We don’t know what happens. Some guesses may be made when we receive feedback on requests for revisions on a file occur. If a rater had an understanding of what the QA on the QAD was looking for, we could help meet those requirements. Yes, that is a form of teaching to the test. The results depend on the quality of the test.
Part of this decision making process should include some type of webinar presentations. If all we, as raters receive, is Blog Posts and notifications from Steve via email – we are not well served.
In my opinion, I think many
In my opinion, I think many problems can be attributed to the fact that no one is licensed to perform these tests and this includes insulating homes. Another problem is the men who install energy saving products are not over seen by anyone other than themselves and the fact that many large insulation companies have internal raters who do not follow through with inspections after the products sold are installed. DOE reports suggest the same. Lack of after installation inspections or simply falsifying paperwork. There needs to be enforceable penalties associated with those persons or companies who do not perform. Fact is, as long as there is money to be made there will always be some form of corruption all the way to the top.
Allison,
Allison,
At the risk of broadcasting my ignorance, I’m puzzled by this observation:
“Laurel is doing a wonderful job and RESNET is lucky to have her, but she’s not a technical person. She’s never been a HERS rater and isn’t a QAD. I don’t think the QA Manager needs to have those qualifications, but RESNET has to have technical people involved in the oversight process.”
Best wishes
I agree that with a bit of orientation and familiarity, an individual with solid management skills can manage anything, but the second part is a mystery. Could you please elaborate?
Allison, I think your
Allison, I think your proposal is good and that a small percentage of the ratings be, on site, checked by a technical person from RESNET — not someone else.
As we have talked (and watch this space for more) ratings within 3% of each other would be very rare indeed!!!
In Santa Fe we have greatly
In Santa Fe we have greatly benefited by having knowledgeable local providers who know our unique style of building. I wonder if a large regional provider would be able to explain the intricacies of modeling a trombe wall, for example.
Ernest M.:
Ernest M.: There’s definitely some truth in that.
John N.: Good point. I hope you’ll submit that in the public comments.
Richard B.: Good points, but I do believe we can make this work.
Steve W.: OK, I need to revise the article a bit because RESNET does have at least one technical person (Abe Kruger) helping with QA. They contract with him to provide assistance to Laurel, but it’s not nearly enough.
John P.: Yes, RESNET absolutely needs to have technical people checking up on all providers with field QA. As far as I know, that’s something they’ve done next to nothing of.
Amanda H.: Good point.
If there was a regional or
If there was a regional or national provider, then I think they could, and should, hire people for each region from that region, people with knowledge of building styles in the area.
I think this conversation is
I think this conversation is long overdue on multiple levels.
The provider/rater business model is one that should either not exist or have a different set of QA regulations. Not only in the case of the provider performing QA on their own people and files but also in competing with the independent rater. The large McRating companies are racing to the bottom in their earnings-through-volume business models that inevitably drives the true value of the rating into the ground forcing raters who can’t compete with the Provider-Mart in a bid war to sell out and eventually subcontract with them. As a result the rater is paid so little they either look the other way, sub out to another person (likely not a certified), or not even show up to test at all. I’ve actually gotten into arguments with Ronald McProvider himself because “none of the other subcontracted raters have a problem with what we’re doing.” There also needs to be improved QA on providers and QADs. I will leave it at that and refrain from sharing my personal experiences.
Another thing about RESNET that baffles me is this…HERs ratings (Energy Star certifications) are all points of data. If we are all collecting the same data so why is there no universal data base or standardized system to collect it? Providers issue their own data collection and field testing forms which, if it’s in Word format, is so out dated I feel as if I need to get out my chisel and hammer. If the provider needs to see the pressure readings from my single point test, on any house I’ve ever rated, they should be able to access it. The Energy Star data collection process is even more cumbersome and I know companies like ICF have paid to create a data collection app to streamline the process. Have any raters seen it?
Next: My understanding is that RESNET performs QA’s on providers through a webinar? I hope this is only in the interim because it’s just not a sustainable model. Again, the information they need to see should be accessible on any given day at any time. Seems to me that having a app or software that allows for immediate upload of data collected in the field as well as a place to store .bld files would decrease the burden of the provider, improve the RESNET QA process on both the provider and the rater, and make the rating process that much smoother.
This is just my two cents…I definitely don’t claim to have the answers but like I said, I’m glad we’re having the conversation.
My point exactly Rich! The
My point exactly Rich! The current system of Providers and QADs is not broken. It is just not uniform. If providers require different documentation then it is easy to game the system.
RESENT could fix a lot of it real fast.
1) Include Raters in the communications loop with QADs regarding QA requirements.
Allowing raters to learn in any other way creates less uniform results.
2) Require each rater to take pictures of some data points. Ex: Elevation Views; AGW insulation; FDN wall insulation; framing; duct blaster probe location with tube in the grill; duct blaster set up; duct blaster manometer reading; blower door set up from outside; blower door manometer reading; attic insulation showing level and with a ruler showing thickness.
That alone will guarantee at least one, if not two visits to the building by every rater.
Require elevation shots and one other of the requirement image to be sent in for archive purposes. Make the one other shot a request from the QAD after receipt of the file. They can randomly choose which shot.
How about random email questions from RESENT to raters about the QA process. Ex: Send a copy of the certificate on your manometer showing the calibration date. Show us which manometer you used for the Blower Door and which one for the duct blaster, if you have more than one manometer. Show the documentation that you are in compliance with the field calibration checks on your manometers, blower door and duct blaster fan. The QAD is supposed to have this stuff, why depend on them. Collect the information direct and then check with the QAD to see they also have it.
Require QADs to do desk audits on files, Collect all images taken at the job site.
Implement desk audits of QADs, require them to randomly submit desk audits of raters. See they are complying with the data collection documentation that we all know is required.
All of these things can be done with 30 days notice or less. It just takes will power and bucking some people.
Word document <
Word document
After going through the process it became obvious to me that a RESNET energy rater could not make a reasonable living following the RESNET requirements. In 2010 I decided to retire and have a home built to a LEED gold standard. I went on the RESNET training class so that I would know what the requirements were for both energy star and LEED and so I could be the LEED project leader.
My intention was to become a rater after my home was completed. Unfortunately the bureaucratic pyramid scheme required for RESNET certification made it impossible to support a financial reward. Some of the hurdles included completing a training class and passing a test, finding a provider and fulfilling numerous RESNET ongoing training requirements. It was also necessary to purchase a number of expensive instruments to complete an audit and an expensive licensed software package.
The first obstacle was that providers also did audits and saw other raters as competition and therefore didn’t want to be a provider to them. Second they controlled the software license and charged what they liked (could get away with), third they charged the rater for each compulsory audit review and fourth they performed expensive on-site quality audits each year on a percentage of rater audits. All this added up to a lot of cost to the rater. Fee’s for an audit varied a lot, but a consensus was that the rating should be under $1000.00 and as low as $600.00. This was still seen as high for some builders and homeowners ignored it until the utility companies introduced their own $99.00 program, which in fact was a $400.00 program that the utility customers were unknowingly paying for.
I concluded that I would have to perform over thirty five audits a year just to break even. It was obvious the rout to go was to subcontract the audit to someone else; unfortunately, many others had come to the same conclusion and changed career direction; even the provider I had trained with went out of business.
And now RESNET is trying to add another layer of bureaucracy and cost to the process. If a rater is trained correctly and passes the test; and if the software tools are easy to use and the input data is strait forward why should you need all this checking and double checking. The problem lies in the complexity of the software; as with all software after a time it becomes too complex. Engineers love to add stuff, more and more details that cloud the simplicity and create error mechanisms; I know I was an electronic engineer.
Building science is not rocket science and RESNET is treating it as rocket science. There are two many layers and costs to overcome for the reward, and LEED is even worse. The new 2012 IECC standard is coming close to Energy Star and is being adopted by local building authorities over the next two to three years. These standards are being verified by home inspectors with contractors providing the data. It will take very little effort to record these numbers and put them in a MLS file. It costs nothing beyond that of the building fees and it provides most of the same information insulation and leakage levels, HVAC and appliance specifications etc as energy star. The only missing specification is the HERS rating and I am sure if that is important to the industry it will be resolved with the DOE NREL BEopt package or one of the many others. If RESNET doesn’t streamline the process and get ride of the multilayer pyramid they will go the way of other organizational monstrosities.
Just my observation!
I feel bamboozled. For years
I feel bamboozled. For years, I’ve been teaching people about Resnet’s ‘3rd party system’. Raters check the houses, Providers check the raters, Resnet checks the providers.
And now you’re telling me that Resnet hasn’t been checking the providers. I’m dumbfounded. Every organization that I know who has a certification program has yearly audits of their ‘providers’. I simply don’t understand how Resnet thinks it can play in the big boy certification sandbox and yet miss this integral step. Yay Laurel for dragging them in.
And now, instead of going the tried and true route that every other certification program does, it’s going to undercut the business model for their boots on the ground people? There is plenty of cheese to go around, folks! Contract with an audit company and get the data for how these rating are off wildly! enforce your own system before changing it!
I sent in a comment, but I fear that I wasn’t nearly articulate enough and will therefore be ignored.
Insightful comments. I agree:
Insightful comments. I agree: there are major issues with HERS and EP5. That said, it’s also worth noting that we are Building Scientists. As such, we should be familiar with the scientific process: observe, hypothesize, test, exam results, repeat. We are still at the beginning. Let’s give our industry a well-deserved pat on the back and get back to work. We have much to do and the clock is ticking. We’ll get there – but only by following the unforgiving and tedious scientific process.