data_processor module



class data_processor.MessageParser(data)

Bases: object

For memory saving, Protobuf3 defines that one field is inplicit (won’t be displayed, saved or serialized) when its value is set to default 0.

Therefore a related change was made in data_processor.py that every single value of messages is incremented by 1 additionaly, except for msgType and prfType.

currentAttributeMsg(value, idx, flag=1)

Pack messages of CurrentAttribute type

metaDataMsg(value, idx, flag=0)

Pack messages of MetaData type

positionMsg(value, idx, flag=0)

Pack messages of Position type

profLongMsg(value, idx, flag=1, prfType=None)

Pack messages of ProfileLong type

profShortMsg(value, idx, flag=1, prfType=None)

Pack messages of ProfileShort type

segmentMsg(value, idx, flag=0)

Pack messages of Segment type

stubMsg(value, idx, flag=0)

Pack messages of Stub type



data_processor.batch_processing()

Batch processing messages

This function calls the processing() function to process message in batches.

One can simply add processing() to th end of the function to process newcome messages.



data_processor.data_parsing(Msg, value, num, flag, msgType, prfType=None)

Based on the input parameters, the corresponding function is called.

Parameters

  • Msg – A instance of MessageParser class

  • value – The value of message of a specific msgType

  • num – The length of value

  • msgType – The type of message

  • prfType – The type of profile

Raises

ValueError – In case msgType is None



data_processor.dump_all_msg()

Dump all .pickle files in outputDir into outputFile. And the messages is ordered by time.

Note: only .pickle files of message you want to dump are allowed to be under outputDir directory, otherwise, an exception will be thrown.

Raises

AttributeError – In case some unexpected files or directories are under unputDir directory



data_processor.get_args_from_terminal()

Get arguments from terminal

Parameters

  • inputfile – The path of raw data file (.mat file)

  • messageType – adasis_v2 message type, for example, Position

  • profileType – adasis_v2 profile type, for example, Longitude

  • outputName – The prefix of output filename you want to define

  • outputType – The extention type of output filename, e.g., .pickle

  • verbose – If you want to increase output verbosity, set it to True

Returns

All arguments you get from terminal



data_processor.get_keys_and_values(data, msgType, prfType=None)

Return flag, keys of message, value of message corresponding to each key.

Parameters

  • msgType – Type of message, int or str

  • prfType – Type of profile, int or str

  • flag – Denotes if Dict matData has the value of key msgType of specific message(e.g. STUB). If not, flag = 0 otherwise flag = 1(default)

Returns

A tuple contains flag defined in get_keys_and_values(), msgKeys which is a list of keys of message that is type specified and value get from data through msgKeys

Raises

ValueError – In case msgType is None



data_processor.main()

To get arguments from terminal

Call processing() function to get arguments from terminal and process messages of A specific msgType.

Examples:

  • Segment message:

$ python data_processor.py  SEGMENT Segment  --outputType pickle  --verbose

  • ProfileLong_NaviPredicted message:

$ python data_processor.py  PROFILE_LONG  --profileType NaviPredicted  ProfileLong_NaviPredicted  --outputType bin  --verbose

To process in batches

If batch processing the messages is needed, comment out the processing() function and uncomment the function batch_processing() below.

To sort message

Call dump_all_msg() to merge all messages into a .pickle or .bin file, and order them by time.

To parse message

If you want to parse messages from .pickle or .bin file, parse_from_file() is a go-to method you can call.



data_processor.parse_from_file(fname)

Message Parser

  • .pickle and .bin file are supported.

Parameters

fname – The path of file

Returns

A <class adasis_v2_protocol_pb2.Message> type object

Examples

>>> msg = parse_from_file('./data/pickle/bin/CuA_Segment.bin')
>>> msg
message {
    Header {
        time: 738724.4364457406
        msgType: SEGMENT
        ...
    }
    Segment {
        functionalRoadClass: 2
        ...
    }
}
    ...


data_processor.print_log(inputfile, msgType, prfType, outputName, outputType)

Displays log messages

If verbose=True, this function will be called and log information will be displayed, else stay silent.



data_processor.print_status(fname)

Checks if fname path exist and if this file is empty, if both are True, returns SUCCEED, else returns ERROR.



data_processor.processing(inputfile, msgType, prfType=None, outputName='UNKNOWN_MESSAGE', outputType='pickle', verbose=False)

The core of message processor

Parameters

  • inputfile – File path of raw data (.mat file)

  • msgType – Type of message, int or str

  • prfType – Type of profile, int or str

  • outputName – The prefix of the name of output file

  • outputType – The suffix of the name of output file (extention of file) pickle and bin are available by now.

Examples

>>> processing(msgType='POSITION', outputName='Position')
>>> processing(msgType='PROFILE_LONG', prfType='ExtendedLane', outputName='ProfileLong_ExtendedLane')