The RCA to 3.5mm would be the preferred way to go, have you verified that audacity is recording any input at all with an iPod or some other device? If you can get that test to work, you know audacity is recording and it is not an audacity/macbook problem. From there you could try a different RCA to 3.5mm cable, and if it still is not working, make sure the output of the mixer is sending a signal.
Were you going to try an RCA/USB cable to USB/3.5mm cable? I am not sure if audacity can read anything other than from the Macbook audio in cable – plugging sound input to the USB connector might not work.
In any case, the straighter the line, the better the signal quality will be.
