Pin voltage of X4 and RPi is 3.3V, so yes - you can connect GPIO pins directly for UART communication.
Check this post to configure X4 pins for UART: Which serial ports can I safely map to GPIO on the X4? - #2 by Arnas
But in my opinion it is better to use USB port in this case. A few advantages:
- More sturdy connection (with USB cable)
- Leaves free GPIO pins for other uses
- You can remotly flash X4 firmware using esptool.py