2024 RTO Roundtable Series: Part 1 - with Lauren Hollows and Maree Thorne - Standard 1.3 to 1.4 All about Assessment

In this riveting episode of the RTO Superhero Podcast, we delve deep into the upcoming shifts in the Standards for RTOs, set to come into effect on January 1, 2025. Joining forces with Lauren Hollows of Aniwaya Education Services, and seasoned consultants Maree Thorne from Compliance Assist and the effervescent Angela Connell-Richards from Vivacity Coaching and Consulting, we unpack the nuances of assessment—the cornerstone of RTO operations.

Our conversation kicks off with a warm welcome from Lauren, followed by insightful introductions by Angela and Maree, who share their extensive experience and perspectives on the evolving landscape of vocational education and training. As we dissect the proposed changes to assessment standards, particularly the transition of 1.8 to 1.3, our experts debate whether these modifications signal a transformation in approach or a mere linguistic reshuffle.

Maree highlights the emphasis on assessment systems over tools, and the nuanced requirements for pre-use testing of assessment tools, underscoring a potential shift towards a more holistic assessment strategy. Angela brings to the table her concerns regarding the practicality of these changes, especially for smaller RTOs, and muses on the broader implications for the sector amidst ongoing qualifications reform.

Lauren raises critical questions about the operationalization of these standards, including the role of AI and technology in testing and validating assessment tools, and the importance of tailoring assessment practices to diverse learner cohorts.

This episode is a must-listen for RTO professionals seeking to navigate the forthcoming standards with confidence and creativity. Stay tuned for the continuation of this enlightening discussion in our next episode, where we will dive even deeper into the implications and implementation strategies for RTOs.

Don't miss this deep dive into the future of vocational education and training standards. Subscribe and follow the playlist for more insights from the frontline of RTO management and compliance.

Transcript

Lauren Hollows: 0:01

Hi guys, this is Lauren Hollows Aniwaya Education Services, and we are back with our YouTube Roundtable series, and today we are going to be talking all things assessment one of the RTO's favourite things to talk about and I am lucky enough to be here with Maree from Compliance Assist and the vivacious Angela from the Vivacity team. Thank you very much, ladies, for joining me. I'll give you guys just both a quick minute to introduce yourselves. Angela, do you want to kick off?

Angela Connell-Richards: 0:31

Okay, Hi. So, as Lauren said, my name is Angela Connell-Richards. I am the owner of Vivacity Coaching and Consulting and we've been helping training organisations to get compliant, save time and save money since 2009. I've been in the training industry for over 30 years and I've seen many changes in legislation and how we've been governed over the last 30 years, so I'm very excited to be part of this project of Lauren's.

Lauren Hollows: 1:01

Awesome, Maree, over to you.

Maree Thorne: 1:04

My name is Maree, as I've been introduced, and I'm also a consultant with Compliance Assist. I've probably been in VET industry for more than 20 years. I've had my own RTO and, you know, had to walk the talk in running that. I think I've, you know, as Angela said, been in the system and seen a lot of changes, but a lot of changes that haven't really been changes. So it's a case of, you know, staying the same but, you know, with different words around it. So here we go again with a new set of changes and new words around those and looking forward to unpacking them a bit more.

Lauren Hollows: 1:49

So obviously, assessment continues and it's interesting. I've had a few conversations with RTOs who are like, no, no, there's nothing new in the standards, everything's the same. And obviously, particularly with regards to assessment, I think people would be looking at that and going well, it's just 1.8 has just gone to 1.3. It's still the you know, principles of assessment, rules of evidence. Nothing's changed is what it is. Interested to get your guys' thoughts on whether or not you feel that these are same same, whether they're, you know, same, but the language has changed. So what are your thoughts on kind of these world standards? Marie, we might kick off with you um, thanks, Lauren.

Maree Thorne: 2:32

um, I think, as I said, um previously, when we talked about a previous standard, I think some of the devil is in the detail of the, the wording um, I noticed that.

Maree Thorne: 2:43

You know, we we're really talking in these two standards, in particular around the assessment systems, as you know, and I think that we sometimes I think RTOs sometimes really focus on the assessment tools rather than the system itself, and I think, for me, when I read this, I think that that's one of the changes that I'm seeing, or shift in focus, if you like, it's looking at an assessment system as a whole, as previously, you know, it tended to be on the tools or the instruments, and I think that for me, that is one of the changes.

Maree Thorne: 3:24

Of course, 1.3, b and C in particular are fairly significant changes about the assessment tools being tested prior to use and then using that testing to actually inform changes, a continuous improvement approach, but also a you know, a test it before you use it, you know, which is obvious to most of us but obviously needed to be spelled out a little bit more in these standards. I think that in 1.4, when the you know the principles of assessment have not changed, the rules of evidence has not changed. I think what has changed is, well, it hasn't really changed. It says assessment must be conducted according to this, and that's not any different to what it was. But do we truly understand what that means? So that's my take on it.

Lauren Hollows: 4:19

Absolutely, Angela, your thoughts.

Angela Connell-Richards: 4:22

Okay. So I agree with the rules of evidence and principles of assessment and I'm glad they kept that in, because I think it is actually a really good way to be able to validate your assessment tools to ensure that they are fair and equitable and that they are meeting the compliance requirements. But yeah, it's the testing. This testing is going to be.

Angela Connell-Richards: 4:49

We did the, our team did the pilot at the end of last year and we had very interesting conversation with our team with well, yes, we can help with testing, but how are the small RTOs going to go with testing and what is testing? How are they defining what is a test? Is testing assessment validation or is testing actually testing with the trainers? And that was something that we came up with. Is that perhaps it would be the trainers and assessors that would actually put themselves through the assessment and complete the assessments themselves, and that's how they're testing it, or are they getting a pilot group and testing it with a pilot group? So I'm really interested to see when the user guide comes out. So they're going to be working on the second half this year. What does ASQA see? We'll be testing, okay.

Lauren Hollows: 5:48

Yeah, I agree. I think that the testing component is definitely going to be super interesting and I guess one of the things that I'm interested to find out, because obviously there are so many words in these new standards that kind of jump out from an audit perspective. I mean they could do, but I don't believe that they're going to come out with like a super comprehensive definitions guide of, like you know, what does testing mean? Or what does you know like each of these terms. You know, clear guidance and definition, because they don't want to be overly prescriptive. But where terms aren't defined, is it then up to RTO to go out, engage with industry, engage with consultants, um, create their own definition as to how they're going to interpret and implement that and then be able to justify that in order? Is that a valid approach for RTOs to take, especially at the moment when we don't have things like User Guides?

Lauren Hollows: 7:01

you know when, realistically, those you know we still don't have a credential policy Come on seriously, you know, like is that really the only rational approach that RTOs can take to kind of navigate their way through this when they're updating their policies, when they're thinking about how they're going to implement new processes and things like that? What are your thoughts on that?

Angela Connell-Richards: 7:26

Okay, Marie.

Maree Thorne: 7:27

I think as well, Lauren, we're talking about words and you know words can be terribly powerful and I think when I saw a previous draft of these standards come out, this, you know 1.3 in particular was basically using the word validate or pre-validate prior to use. And I think you know a lot of people, a lot of the feedback that I saw go through was you know, don't use the word validate because it has a different meaning in terms of you know what we've understood that meaning to be. And so now this word you know about, tested prior to use. You know, I wonder whether it's sort of, you know, we're getting tied up in the meaning of the word, meaning of the word. I personally believe that, oh my God, this it shouldn't have to be in the standards, but obviously you know it shouldn't have to be prescribed, but surely you know you would be actually looking at whether you use the word tested.

Maree Thorne: 8:27

But you know, the next part of it is for me to ensure that assessment can be conducted in a way. So it's less about the testing, but it is, you know, does this assessment work? Is it the right method? You know? Are we assessing our practical skills in a way that you know really, you know, with valid, with the, the principles of assessment. So it's, it's whether, whether we look at the, you know, I think a lot of people might look at this um and interpret it that we, you know you have to pilot this. I I don't know how you get on piloting with a group of people and giving them an assessment that really doesn't work and it doesn't meet all the requirements. And what happens at the end of that pilot, when you have all these learnings and discoverings and go well, I'm sorry but I can't give you, actually, you know, the piece of paper that you've done, because you know, really, this assessment didn't work. The tools didn't work. The you know they weren't in the right sequence. The recording you know tools that we were using, you know didn't have enough, all of those sorts of things. You know the benchmarks were not, you know, able to be interpreted reliably.

Maree Thorne: 9:44

So I think, when we're using the word testing them prior to use, I'm seeing that as a critical part of what you should be doing. You know, as part of, you've developed an assessment system. Make sure that it can do what you want it to do or that it needs to do. You know, rather than it being I have to trial it on a group of students, then you know what happens. If you trial it and find out that it really didn't meet the requirements. What? What happens to all those students? Um, that's, that's my initial thought yeah, um, okay.

Angela Connell-Richards: 10:22

So from my perspective, because we've got a whole other thing that's happening. We've got the qualifications reform as well. So all of the training products that we have are going to be totally different to what we're used to now. So I was interested with this being placed in here, that you know. It's very similar. It's validation, it's rules of evidence, principal assessment. How is that going to tie in with the new qualifications report and how we're going to be delivering training? So I'm actually expecting that we'll most probably have another change in the legislation once we get the new qualification report. When that happens who knows when it's going to happen? We're supposed to be in the third year of the rollout and it started with. According to their roadmap, it's supposed to be we're in the third year. The first part was micro-credentialing. So it was very interesting with how really encouraging RTOs to take up micro-credentialing and build micro-credential courses, but I haven't seen a lot of change happening with that and I haven't seen a lot of change happen with the qualifications reform as well. So testing is it just another word for validation or is it? We are validating the assessment tool and then trialling it and testing it?

Angela Connell-Richards: 11:53

I remember back when I was a trainer assessor many, many years ago and I used to write assessment tools and test them on my students all the time, because I had no choice. So it was like the RTI I worked for. That was my responsibility to write assessment tools, and I would write the assessment tool, test it. That didn't work better, change that and gather more evidence. And it was a constant to and throw with with okay, let's see if this works. If it doesn't work, oh, hold on. The students didn't understand that. Maybe I need to change the wording of the questions, and this is something I think all trainers and assessors should be doing now. They should be adjusting their training and assessment tools as they're using them. And I call this you know you've got pre-assessment validation, post-assessment validation. This is validation during assessment. We should be changing and adapting our assessment tools to meet the student needs on an ongoing basis and making sure that they work.

Lauren Hollows: 13:06

And I mean, I think this is where because I mean, all of us came up, probably through that era of like, hey, you're training on Monday, here's your class, this is what you'll be teaching. You'll now need to spend the week in creating your assessments, creating your training materials, creating your this, that and the other. It was like, oh okay, cool, all right, no worries. But now, in this era of like everything, like everything has to be so clearly evidenced trainers have been removed from that assessment. Like that assessment design process in particular, um, I think the RTOs do take on feedback from their assessors of like, this question, this question isn't working. This, you know we're going to make an adaptation in this way. Um, but I also get the fear of so many compliance, um, and you know curriculum development people that turn around and go if you change my tools, right, actually, no, sorry, you actually can't change my tools like, end of the story, you can't change my tools. If you're going to make a change, you need to submit this form, you need to go through this process, and a lot of trainers look at everything that they've got in their thing and they go it's. I'm putting that in the two hard pops.

Lauren Hollows: 14:19

So I find it interesting that this has been brought out of like 1.5, which is very specifically focused on validation, and I think you're both right in that. This is definitely meant for providing clear evidence that before you just roll it out so you don't go and buy the off-the-shelf assessment tools and roll them out directly to your students that there is some process that happens before that occurs, to make sure that what you're going to be doing is going to, as you said, like meet your assessment system. You know the interesting thing, as you were talking about before, angelique, you were talking about AI. I think one of the interesting things is, like who is going to be doing this? Does it have to be done? Because validation very explicitly says it has to be done by people who hold all of these things right. This assessment testing does not have that requirement to it as such. So I wonder if this is like one of the areas where we can look at it and make the argument and go well, look, we have a system that does this we have a.

Lauren Hollows: 15:32

You know we utilize AI to test it.

Lauren Hollows: 15:36

I test my questions with AI all the time I go. Here's the question, give me an answer. And if it gives me an answer, that's like way left of field, I'll go on to a different AI. So like, if I test it in GPT, I'll then go and test it in Gemini and go well, what do you think? And if both of them are coming up with something completely different, I'm going I'm phrasing this question in the wrong sort of way, right? So is that me testing my tools? You know you're like, but it's not a person testing my tools. You know it might be like. I created some recipes last night for a baking unit and then I ran them through ChatGPT and I went do they meet all of these requirements? Like. So I was just testing myself to kind of go, okay, have I covered everything?

Lauren Hollows: 16:20

And he was like, well, you haven't explicitly covered this, and I was like I have covered that, but you're right, I haven't explicitly covered that.

Angela Connell-Richards: 16:29

Do you know?

Lauren Hollows: 16:29

what I mean Like so is that testing? Because it's done? You know, I think this is one of the areas where we could see some of these sort of things come into play.

Angela Connell-Richards: 16:38

And I think something that's really important that we need to focus on is it's not just testing the tool, it's testing the tool in specifically for your learner cohort. So it's who is, who is your target market? Who is your target market, who is outlined on your training and assessment strategy and how are you ensuring that your assessment tools are meeting the requirements of that learner cohort? I think that's one thing that we really need to be incorporating in and it doesn't specifically say that in here, but it is part of our learner journey is that we're ensuring that we're delivering training and assessment that meets the learner cohort needs, and no one assessment tool can't meet all the different types of learner cohorts that are out there, so we need to contextualize it for our learner cohort absolutely okay.

Lauren Hollows: 17:31

Well, we might take a little break there, guys, and then we're going to go into my next question. For those that are following, I am here with Angela from Vivacity and Marie from Compliance Assist. You can find all of their contact details below. We are talking about the new standard 1.3, 1.4, previously known for some as 1.8. We're going to go into a little bit more detail when we come back on our next episode. You can follow that by following the playlist and hitting the subscribe button. My name is Lauren Hollows Aniwaya Education Services.