Vinyl***What If***

You will now receive umpty-nine completely different opinions and suggestions.  But there's nothing wrong with that.  You could put some sort of boundary on the variety within the responses by stating your upper limit as to cost.

I'll start.  I'd buy a good restored DD turntable with a low cost MC cartridge, like a Hana SL or a high end MM or MI cartridge (prices are similar in each case).  The turntable would have its own tonearm or I'd add a decent modern tonearm that fits your budget, if you have a budget.

Rega P3 or MoFi or Technics 1200GR table
Sky’s the limit on the cart, but a good MM can be had in the $200-$500 range that will keep you happy I’m sure. I myself will likely get a Audio Technica VM540ML very soon to replace my old Shure M97xE
phono pre-amp, I like the Schiit Mani, but I’ve heard good things about the UTurn Pluto. (Would like to get a Manley Chinook if I could - tube phono - joy! But that is out of my price range)
other than the main items get what I think are MANDATORY accessories:
-Onzow Zero Dust (keeps the needle immaculate )
-Record Cleaner (I like the Record Doctor V, since it has vacuum and only $199) - this may be the most important item of all to keep your records dead quiet
-Carbon Fiber Brush

A nice to have, but probably not necessary
- Milty Zero Stat anti-static gun
-White gloves for record handling - jk, lol...

What's "inexpensive"?  What's your budget?  Are you open to used? 

It sounds like the Cayin only does MM cartridges?  If so, the suggestion for a Hana SL (and many other MC cartridges) is not a good one.  You could get by with the Hana SH (the high output version) but might want to stick to MM.  There are some very good ones out there.

I would not buy a thing at this point if you are on the fence -- do you have a dealer who would be willing to loan you a better than entry level set up for a week or so? And where are you going to get records to listen to? Buy them and get rid of them if you decide it isn't for you? I assume you have some friends who'd be willing to provide some records for you to play temporarily....
I'm not trying to be negative about this-- I've been playing LPs all my life (well, since I was a child) and am fairly well 'invested' in the medium. Ultimately, that means money in records. Not just gear. Part of this is meant to be fun- looking for better sounding pressings, finding stuff in bins, not just hitting "BUY" on the latest remaster. 
I'd also be concerned that the total cheapy stuff probably isn't going to give you any sense of the potential of the medium. Nor will some records, no matter how how heavy the vinyl weight or fresh off the press remaster. (Nothing against re-do's, some actually do sound better than early pressings). 
In other words, I'm not sure you can get a real sense of what the potential is by dipping your toe in tentatively. So, see if you can get a dealer to loan you something decent and set it up for you and get a few good records to try. In the long run, it may be cheaper to pay for a dealer's time to do this than for you to buy something mediocre, declare every one who is into records delusional and then try to sell what you bought. 

What whart said, +1. Are you anywhere around Indiana? I’d be up for bringing my older/retired vinyl rig(and a few records) over, for a demonstration.