Problem with Turntable leveling?

I leveled my turntable before I started using it, but:

when I lift the tone arm by means of the lever the arm tends to swing out towards the edge of the record a bit. If it were truly level, would it not go nearly straight up (and down)?
Check your anti skate setting.
Yeah, antiskate setting is moving the arm as it falls.
Not a lot to do about it.

Though perhaps wiping the lift bar (if you have one) off with alcohol.
And flipping the drop lever more slowly, instead of fast.
The arm now is free floating down, if you hold the lift and release it slower, the arm will stay resting on the lift bar and not move sideways as much.
There is also sometimes a natural pull, since there are magnets involved. It may not be enough to actually have a detrimental effect on your sound, it just depends - too many variables to make a guess.
If a good level says that your turntable platter is (level), the problem lies elsewhere. I once had a Rega arm that did this. I've also had tables where the plinth was not perfectly level when I leveled the platter and THAT affected the arm. I couldn't detect any audible problem, though.
Even if the platter is level and the arm is level it's possible the arm rest upon which your arm is lifted is a bit off level. This certainly could not impact playback at all
The cue bar that lifts the armtube may not be parallel to the record and might cause the arm to move outward.

The cue bar might be rotating slightly when the piston raises it. They have anti-rotation locks, but can move a bit back and forth.

The cue bar might be loose.

The internal wiring, going through the pivot, might be dragging a bit and pulling the arm outward.

Antiskate may be high as previously mentioned.

I've seen all of the the above.
If you can cue up the record at a song you want to hear without the cartridge shifting as you lower it to the record you should be fine. Mine moves at times but everything else is correct with the alignment so I do not worry about it.