Parsing "watch" protocol AMR sound file

marko89kv8 years ago

A device which I use (3G Electronics, watch protocol) has a function where it sends sound files in AMR format to the server. In traccar's log I see that the file is broken multiple packets:

http://pastebin.com/raw/ArMWekwS

So, it starts like this:

[3G61051784260CAA*TK,#!AMR

It's in the TK range of messages, so I have three questions, or better, nudges in the right direction:

  • How would I detected when the packet with the complete message ends? Since the first packet contains a string like that one from the top and the last packet contains ] ?

  • With which method from the PatternBuilder could I parse it for saving in the database? Probably with binary() ?

  • Would it be 'safe' to save the blob as an attribute of Events model?

Thank you.

Anton Tananaev8 years ago

Message header include the length, so you can use that to detect where message ends.

You can't use regex patter or even convert it to string. You have to handle it as binary buffer.

It should be fine to save it in attributes as long as it fits into the column.

marko89kv8 years ago

So, the fact that logger breaks it into 3, 4 or however many different logs means that it is broken into different packets or is it one big one?

As for the binary buffer - do you mean something like this?
http://stackoverflow.com/questions/140131/convert-a-string-representation-of-a-hex-dump-to-a-byte-array-using-java

Anton Tananaev8 years ago

If it's broken into different logs, yes, it's different network packets, but it doesn't really matter.

You have to convert this protocol into a binary one (like Meiligao for example), and deal with it as a binary data, not text.

marko89kv8 years ago

The whole protocol or just the part where the watch sends an AMR file?
I've found in the source how custom frame decoders are added - pipeline.addLast("frameDecoder", new MeiligaoFrameDecoder());

Since my documentation about the protocol is same as yours from the website, should I just try my luck with trying out different already existing frame decoders from other protocols?

Anton Tananaev8 years ago

Of course you can continue using text for the rest of the protocol, but you can't use string decoder.

You should implement a frame decoder. There are no existing ones that would work.

marko89kv8 years ago

This is what I have for now, not implemented in traccar, just standalone, for testing:

public static void main(String[] args) {
        String amrSound = "2321414D520A7D049EC07C00FF01FCB9A32F508B262FB3F58360AF527D049EC07C00FF01FCB9A32F508B262FB3F58360AF527D049EC07C00FF01FCB9A32F508B262FB3F58360AF527D04B545B206E00996604BC234E3287D0301F433073A267D04B458B21B6004C944B3A3DAC794777AB0AC5313187D0493DDC81E000BF6FE104EA87D02739D7AF933B903627D04B51CC01E0004C4B82D71B624F213DCC40A8B56287D045C00241E0001661ABE83EA0BDB61EAF61F273FD47D04B4184E1E00B14D87EFCD1D764FE34B80355228CA7D04CE328C3CC227DE67493770ECE52795941A6CD9847D04E31871065109291620B4297BCEFBB6B229609D9C7D049789541F8011E061247763CD314C70541852431A7D04C46EC61F800384A96B3B91BA11D37D05E9AE40D53A7D04951E469788886B605C02CFFDD3198FA0BE8A43247D04B06F281F4046FBC0412B0EA8521A6AB1A9D394CC7D049C1E8E1E00140365D26650F4E5419BC05A9D48247D049C009016811005D806DA945C62AD95434250D3487D04B47D04160B401F9E97A6FF3CC825CBA7D345C145DA7D047B04C014A0A4B9087D03F6494E019EE44374965A887D04A0A39012C04AF6BE177BEC541724CFA6C0CA32CC7D04978532068101C559DACA5EB9075C67F5C33EBB067D0465E6820B40A09B43465635B4347ADCB68DE6FD447D043FD37D01A5D9C60E7B845385BCD941B05F1909E9727D046C8DE6D3210013B34400D4B65C3264205E2BA2A87D043397E6D32982CA1E0D0EF9E8BC32497D01138F360C7D0455FA1DA45047FE0CD28AA76D91E19B5C16CA2D4A7D0457171C7D02B005F61A4A06CE6CB1107C96C8E47D054A7D0457EDDA9763006400B105EB174615533D8624F0607D043F97041EC0600A96CB6BCEEF4B5A9D588062CA707D04658C1C1FE89E9E42AFD37B7A9F58F392F19E66827D046D68CA7D0220EB720AFACC21376F6CA53A38C878367D04951CCA1FE001DA636AD7E4B5BE0BD2D932BD74A27D04B812E21F8151A4177D019EAF538A92345961EC3AF07D04B903CA1FA01AE5FDA2E5E20D71DD14CF6B0832B27D04AFCAFE1F6020F0B8E3B700DA2E05F8AA9DC5C1C07D049E3F8E17C0014ECB6AE6A3119091ED90242BA4747D04A0A3BE1A4052BC869E88B1364108BE38DAB4A63C7D04D0166A1A4081B954E987A5EC8E20FA2F4D7F8A047D04A618B21C201FE1D2EADA43B2DFEE6D05A9EDAF007D0457321812E01578B570C7AEF59A39CF2E43C233807D046B6FFDAD19A80FD48E80C1B270858D8F5049C75A7D046C0DF50EE02E0B0B4C86C74E708B5134B90BED247D045672FC1EA00B74B11E6332ED26CCBD36867FB3AE7D045E00941F20154449D88376DE08222D01D26085B67D049C7D021C1E000BD5A760B3BBFF7D057D01D15150D902AA7D04AF53901E0006D356DFE5B6F24C3370B2C787ED3C7D049C6EBC16C0030A89D289F9F72F16E4A6230B14847D04A1066A16A0157498D18D885A3DCB3D5E74B656527D047AA39814E03B9F4A1965D38C9DE0B46DBB0567787D04A5536A1E00117D04A1C98ABF53F6187227CA7504DE7D04544920DE68801905491596B1BBD359DC6EBEBBB47D046B047C79C0258168A48740C52D934B7C7D048B7D04727D045AB4ACD34C801812700BB7CE1991C73B737F81B87D047D0204DB69A1FAEE9731B1591B7A179982FE27EBA27D0436D5F4B5F1AC7E240D35F3774E6635D47D0538DE5A7D04500C1CB5E199A4F03313FAED065127C1213B4D767D04552BE6B54599EE0924F4204992C9AA8539C09D3E7D046F411E3DC0DE89FF754059AE2FF6CCB6C25390B67D04A42555A441208807B746C53838D31826DFF4DA867D047A7289C22984C4FFBB20650AF76B4F513D11E2407D04A824E7C320313DCF083D54B13E65683610BBA2FC7D04B386A187610189EF63C1C766E1D34AB17D0395D07C7D04B6008E3DC00CE43D8A8FDEF20F5AB59F2110CD2E7D04DFA9541FC014394666AA8368847D014CDA2ECF317D047D04ACCC1C1D2101056B9EA4B0A4976230A1B68876267D0497451C1E80481830B0588910F8F6EF7D02E888185E7D04B5103A1E0002E3790560F0B2723CF0FEED95DB047D04A40AB81A6147299BB89934DDE7465285444A147D047D041F99541E827D05BB23D63B4EE7E84F99BCE2DFF7A07D046FBDF53DD502A70054A3CFA27E22D79DC25299CE7D046F7D037B0EC0089E4356C85AB34806C354189D350A7D0495DD2892401FB947EA2E37DC72F7CCBAB47D0254527D04AF06901E001317967625FE1459272D4F61AB41907D04BA25E61E0095D4F1E63D8F6ACB867070823691867D04B4189A1E818D81A125A8D9378939A9CE2F422DD87D04A7866A1C201D434B98BAED988677B12607F8F81E7D04BAA44E1C200532A2C669FB8C927F22E02FFA60BE7D04A4256A1A401D13F480DD4BBBEB9F60AE918D0D4E7D04A7B5901A400ABE75BE47E27D057D029115B5E011B5F07D0456A1D83C024FBC7D04AC45AD38785596AD040BA7507D0464247F9E79820A54C08055F63E82797D029079B1627D043498E7688214DE4905703E9172DA593E7D038778F87D043D17F5A4FEED8C82643051E4F942CC6026F89A607D043E151CF1AF9989E1831714AABC5386981098850C7D0437ED3DE000F9A43332897D010408B5AAC530FCCA747D043C91DB4AA1B9BB0DC1EC5A1EB24C06C25870EB9E7D04569F057D04C2050BC5BB74EDB50F6D1583737D02CE807D0479ED7D01C224A99C2504561B50CACD81F172B4EE4A7D04A92BFFA4C81743D42E63E29CFFEA21D31B49D1F87D04A586FFE004D04415127683F5FF018E9A94A013647D04A818E74AA100264F69F2822916C1786CEB156EB27D04B59EFC1FE0151D355EEE0DB88D6C3A77403487FE7D04D514CC1F8009468E324B60D4E3499582280BC2127D04AF1CCE1E000723F8519817EC868B8030A616AAB67D04A624741E0010F4141E1CE10E9C0846FCE19B7D04E27D04C514061E0008B270A6B4EA1B75B293B67ECF85647D043DA33816A02181E9AC5ABB58C4CB9ADF497BBC1A7D0432EB149DA98516956DD486F1D736347D016B01112E7D04DA45ACD7466AEBFF8CD8FF7E3B9834DFF6801DEA7D04B6B4549A611FAE0E8A8208A5AC49AE93C8FC99C47D04E9285A7D02A015D12090D0A20A304A3F47C2498BE87D04DF1FF9C4E549AA433B9C526929B8F4964E99188C7D04B381901EA001DA7D04BE5C319BBF6E7C0EABEF2B767D04D4EA741E406C648538688B61B7CF345365F81CFE7D046C1E1696A8E863447E26D784DE6426761F8D71587C";
        String result;
        result = hexToBinary(amrSound);
        System.out.print(result);
    }
    
    public static String hexToBinary(String hex) {
        return new BigInteger(hex, 16).toString(2);
    }

Am I on the right track? Sorry for all the questions that are not directly traccar related, Java is not my primary language, primarily I'm a web-dev :))
I forgot to write on my first post that if I take the hex string of the AMR file (put it together manually, by copying from the logs), put it in a hex editor and save as AMR, if I try to play it in VLC for example, it's just random noise, not at all the original sound recording.

Anton Tananaev8 years ago

That's definitely not correct. You are converting it to some string, not binary.

dimonb7 years ago

I have extracted file from packet. But have no any idea how to play it :(
Here is a piece of python code.

import binascii, struct

data = '5b33472a363130353632333930362a324333372a.....'
data = ''.join(d.strip() for d in data.split('\n')) #multiline string support

data = binascii.unhexlify(data)

company, device_id, length, message = data.split(b'*', 3)
length = struct.unpack('>H', binascii.unhexlify(length))[0]

assert message[length:] == b']'
message = message[:length]
assert company[:1] == b'['
company = company[1:]

command, params = message.split(b',')

print (company, device_id, length, command)

if command == b'TK':
    with open('out.amr', 'wb') as f:
        f.write(params)

i have recorded a single sound tone 1440Hz, but when amr file played I listen only scratches https://www.dropbox.com/s/3f8s0i1fh0ugzce/out.amr?dl=0 — here is file

Have you any ideas how to decode this file?

Anton Tananaev7 years ago

Your file looks fine. Have you tried to compare the data in the file with the HEX from Traccar log file?

dimonb7 years ago

This file was parsed from tcpdump and wireshark. It seams correct except it does not played ;(

Anton Tananaev7 years ago

I guess there is some problem with the device then.

dimonb7 years ago

No, it is not :(
Message played successfully from native application (seTracker).
I configured nginx to pass all traffic to their server and dump traffic from watches.
Message is played fine in application (but it comes to it in MP3 from server)

Anton Tananaev7 years ago

Maybe device encodes some special characters?

dimonb7 years ago

:( is there any way to decode this file?