So I recently switched from aircooled 3080ti ftw3 ultra to a hybrid 3080ti, and trying to overclock it a bit with the extra thermal headroom.
I fully understand that there's a silicon lottery thing and overclock results are not guaranteed. But whenever I'm trying to push my vram higher than +700 (which is not too high for gddr6x, I was able to reach stable +1000 on my previous air cooled 3080ti), I get artifacts. 
Based on many detailed articles you can find online, (including this article from tpu) gddr6x has built-in error detection/correction, it will reduce the performance when you overclock too much, instead of producing artifacts.
 
So I'm not worried about my new hybrid card can't reach a certain level of vram overclocking, I understand the overclocking results vary. I'm worried about the error-correcting function on my card is not working. Should I RMA? Could use some inputs. 

arestavo
Sounds like you're VRAM thermal contacts aren't as good as with your stock cooler. Might want to make sure that all the pads are lined up. Maybe set the fan speed at 70% minimum to make sure that the card is getting good cooling as the shroud fan is still tied to GPU temp.
 
As far as ECC, it's doing its job still as it's not something that can be turned off. Not getting the same OC results is not a valid reason to RMA, as overclocking isn't guaranteed.
 
FWIW, my 3090 FTW3 Ultra air cooled to hybrid conversion didn't lose me any OC headroom. Then again, I made sure everything was lined up and I slapped on an 80mm fan onto the backplate since the 3090 has VRAM on the backside that is only passively cooled (which would have held the overall VRAM OC back).

Sorry for not being clear enough. I didn't purchase the hybrid conversion, I purchased another factory hybrid card.
So I'm worried about my new card has a faulty VRAM controller/chip so that the ECC is not working, which is why I wanna RMA. 
Not that I want to RMA because I can't reach a certain level of overclocking, I stated in my OP that I perfectly understand that overclocking results vary and are not guaranteed. 
 

zhubaohi
So I recently switched from aircooled 3080ti ftw3 ultra to a hybrid 3080ti, and trying to overclock it a bit with the extra thermal headroom.
I fully understand that there's a silicon lottery thing and overclock results are not guaranteed. But whenever I'm trying to push my vram higher than +700 (which is not too high for gddr6x, I was able to reach stable +1000 on my previous air cooled 3080ti), I get artifacts. 
Based on many detailed articles you can find online, (including this article from tpu) gddr6x has built-in error detection/correction, it will reduce the performance when you overclock too much, instead of producing artifacts.
 
So I'm not worried about my new hybrid card can't reach a certain level of vram overclocking, I understand the overclocking results vary. I'm worried about the error-correcting function on my card is not working. Should I RMA? Could use some inputs. 

That "sort of ECC" can help if somewhat unstable; however, it is not a cure all for too high of OC. -- (this is my understanding from your linked article)
 
OC testing ... if you change any major component - you need to start over from scratch
1) stable power
2) remove MB & CPU OC ... go back to stock .... does GPU OC any better ?
3) some MB will OC the front side bus ... so check your MB BIOS settings ... remove "auto OC" features
4) test with a quality (reproducible) load & make small adjustments
5) Drivers
6) OS updates
7) Room temp
8) hardware temps
 
List your hardware ... even if your using all the same stuff as with the air cooled GPU
 
what benchmark do you run ?
what is GPU-Z telling you ?
 

What are your memory junction temperature?

Once I got my memory junction temperature down to 55-60C, I can do +1650 on my memory now. But previously I was not able to.

Perhaps your memory is hot. The hybrid doesn't cool the memory down that well, it's mediocre.

The hybrid does do a poor job of cooling the VRAM. I dropped about 8C by putting good pads (Gelid Extreme) under the backplate. Although the results have been mixed on this for some reason.

And any of the cards will artifact and even spontaneously reboot if you crank the mem OC high enough. Both my 3080ti FTW3 and Zotac 3090 will do this. I lost the silicon lottery on the 3090. Even with sub 60C VRAM temps, I can only get +850 or so.

So getting temps down only gets so much. Maybe +300 to +500. But that's from dropping 20-30C.