On the XDJ-RX you have to press Menu -> Mixer Mode -> Midi , then it’s ready for usb connection with rekordbox.
Have you got your Rekordbox DJ (Performance mode) lisence? It was attached to the RX in the last month…
Still don’t understand why you want to use this as an controller, the whole point with the RX is the standalone player. 😉