My recent assignment requires working with PHP script listening for connections from network-enabled devices using proprietary protocol.
When new device came in for testing noone knew why it’s data didn’t arrive into the system even though it used the same protocol as older devices. After confirming some details and checking every possible logs I’ve found I decided I need to go deeper – I’ve resorted to network monitoring.
I knew only few things about the device I needed to debug – the most important was it’s serial number which was hidden in each packet and information about it’s IP address and port (which I found out were changing quite often). My first thought was to dump packets originating from current IP address of the device:
tcpdump -i bond0 -vvv -X -s0 port 1025 and host xx.xx.xx.xx
Selected options stand for:
-i – interface to listen on
-vvv – maximum verbosity mode
-X – ASCII and hex content of a packet
-s0 – make sure to display full packet content (max 65535 bytes)
port 1025 and host xx.xx.xx.xx – filter only packets with specified src / dst addresses
The problem was I did not get any packets to work with – at the time I have set up actual IP address/port it has already changed. I could dump all packets arriving at PHP service port but with such a high rate of new packets I would need to dump hundreds of megabytes of data to get anything useful from it.
But this was about to change.
Ngrep is an interesting tool which you can use almost as easily as grep but in network context. Instead of dumping whole lot of data to a file and then search through it looking for my pattern I was able to do this on the fly.
It was that easy. I could simply grep through all incoming packets to look for the device ID I was looking for. It didn’t matter if the IP address / port has changed, I could just run the tool and few minutes later analyze the results.
ngrep -X '0x00000932' -q -x
Options I have used:
-X – treat the pattern as hex value instead of regular expression
-q – display only packets I’m interested in
-x – display hex data (similarly to standard hex viewers)
After little fiddling I was able to retrieve few packets I could analyze, the results were similar to this packet:
T XX.XX.XX.XX:4186 -> XX.XX.XX.XX:12300 [AP] 00 00 00 00 00 00 00 3e 08 01 00 00 01 49 51 d2 .......>.....IQ. 97 5e 00 0d 1f 08 30 1d 9e 8a 20 01 32 00 97 0b .^....0... .2... 00 0f 00 0a 05 01 01 45 01 f0 01 15 03 c8 00 03 .......E........ b5 00 0d b6 00 08 42 35 3d 02 c7 00 00 00 09 f1 ......B5=....... 00 00 65 93 00 01 00 00 e1 a8 ..e.......
Now, using protocol documentation I could go byte by byte and look for anything strange. Aha! There it was – one strange byte value pointing at undocumented feature which caused packet to be treated as invalid. Now I was sure that it wasn’t an issue on the system side (any protocol changes must first be applied on the system side) but on the device firmware side and after passing this information to technicians not so long later I have received updated protocol specification including this new byte value.