Often you end up with a program that works with some hardware but doesn't do exactly what you want or runs on windows when you are trying to use Linux. Sometimes the serial protocol is poorly documented or worse, it has errors in it.
First start socat in a terminal with some reasonable options.
$ socat -x /dev/ttyUSB0,raw,echo=0,nonblock,min=0,b9600 \
You can figure out exactly what options you need by running stty on the serial port when it is running normally with the program you are trying to debug.
$ stty -F /dev/ttyUSB0
speed 38400 baud; line = 0;
min = 1; time = 0;
ignbrk -brkint -icrnl -imaxbel
-isig -icanon -iexten -echo -echoe -echok -echoctl -echoke
Configure a virtual machine to use /tmp/ttyV0 as com1, then load the program you are trying to get working in the VM.
You should now see a hex dump of the serial data in the terminal.
Another option to try is something like this which should write the data out to two separate files.
$ socat /dev/ttyUSB0,raw,echo=0 SYSTEM:'tee input.txt | socat - "PTY,link=/tmp/ttyV0,raw,echo=0,waitslave" | tee output.txt'