Thanks for the input @ted_b and @jbuddha882! I appreciate your comments.
Version 6 makes the situation worse with the exact same settings. So I've moved back to 5. Sounds like that is not uncommon. I've done the list @jbuddha882 recommended before and again today with no luck. And the only thing I have attached to the M4 is a usb drive to store the music files. The USB drive and having Roon on that server might be the issue. I do not use DSP in Roon but do have some light PEQ in HQPlayer.
For me DSD256 is working great and I can not tell that big a difference between it and DSD512. However I do hear a significant difference in filters. For the DSD sources I'm using FIR2/XFI and for Modulators I toggle between ASDM7EC-super and v3. Also played around with v2 and -light. Of the ones I tried, these seem to work best for me. I like the smoothness of v3 for some things but the spaciousness of -super for others. For PCM sources I like the poly-sinc-ext2-xl (1x) and -hires-ip (Nx) for most things. Also like the poly-sinc-gauss-long and -hires-ip for others. So for now I'm optimizing for filters and modulators plus having Roon with the usb drive versus upsampling to DSD512. I also played around with PCM as the output but still prefer upsampling to DSD. One of the downsides of my DAC is it pops sometimes when the sample rate changes. This is my 2nd R30 and it still does it. So locking in DSD256 allows it just to play. Not sure if a dedicated streamer with I2S would help but am curious.
One other issue that I asked Gustard about was the hiss when upsampling to DSD512 that I do not get with upsampling to DSD256. Might be an issue with the streamer in the R30 but not sure since I do not have another streamer with NAA and I2S or USB out to compare. Gustard's response: "If the modulation settings in HQPlayer are not ideally matched to the R30's architecture, it can result in noise or instability."
So I'm set with DSD256 and version 5 for now. They sound great. One day I might add a streamer but I am a buy once consumer so when I do, I want one that matches/compliments my other gear.