History

OML Overview » Internal Development Documents »

Description of Text Protocol

The text protocol was defined to simplify the sourcing of measurement streams from applications written in languages which are not supported by the OML library or where the OML library is considered too heavy. We primarily envision this protocol to be used for low-volume streams which do not require additional client side filtering. The OML client package includes a Ruby class which implements this protocol and provides convenient meta programming extensions to define measurement points easily. However, implementing the protocol from scratch in your language of choice should be very straight forward.

Protocol

The protocol is loosely modeled after HTTP. The client opens a TCP connection to an OML server and then sends a header followed by a sequence of measurement tuples. The header consists of a sequence of key, value pairs terminated by a new line. The end of the header section is identified by an empty line. Each measurement tuple consists of a new-line-terminated string which contains TAB-separated text-based serialisations of each tuple element. Finally, the client ends the session by simply closing the TCP connection.

Header

The header contains the following keys. All of them have to appear exactly once, except for the 'schema' field which needs to appear once for every measurement stream carried by the connection.

  • protocol: Has to be "1"
  • experiment-id: String identifying the session and with it the database the measurements will end up in
  • start_time: Local UNIX time in seconds taken at the time the header is being sent
  • sender-id: A string identifying the source
  • app-name: A string identifying the application producing the measurements
  • content: Encoding of tuples, needs to be "text"
  • schema: Describes the schema of each measurement stream. See details below

Because of an implementation limitation in the server, which we plan to remedy in a future version, the sender-id and start_time headers must come after the experiment-id header.

Schema Description

The description of the schema used in each measurement stream is a space-delimited concatenation of the following elements:

  • local schema id
  • name of the schema
  • a sequence of name, type pairs, one for each element. The name and type in each pair are separated by a ':'

Each client should number its measurement streams contiguously starting from 1.

Example for schema headers are as follows:

schema: 1 generator_sin label:string phase:double value:double 
schema: 2 generator_lin label:string counter:long

Valid types in OML v2.3.x are:

  • string
  • long
  • double

NOTE: the upcoming OML release v2.4.0 will add several new types and the long type will be deprecated.

The name of the schema must consist of only alpha-numeric characters and underscores, i.e. [A-Za-z0-9_], and must start with a letter or an underscore. The same rule applies to the names of the elements of the schema.

Measurement Tuple Serialization

Each tuple is serialized into a new-line terminated string with all elements separated by a TAB. In addition, three new elements are
inserted before the measurements themselves. These three elements are defined as follows:

  • time_stamp: A time stamp in seconds relative to the header's 'start_time'
  • stream_id: This is the same number as used in the 'schema' header definition
  • seq_no: A sequence number in the context of the specific measurement stream.

The sequence numbers of each measurement stream is independent of the others. That is, the first measurement in a given stream should have seq_no=0, and all subsequent measurements in the stream should increment the sequence number by 1.

Example

protocol: 1
experiment-id: ex1
start_time: 1281591603
sender-id: sender1
app-name: generator
schema: 1 generator_sin label:string phase:double value:double 
schema: 2 generator_lin label:string counter:long 
content: text

0.903816    2    0    sample-1    1
0.903904    1    0    sample-1    0.000000    0.000000
1.903944    2    1    sample-2    2
1.903961    1    1    sample-2    0.628319    0.587785
2.460049    2    3    sample-3    3
2.460557    1    3    sample-3    1.256637    0.951057
3.461064    2    4    sample-4    4
3.461103    1    4    sample-4    1.884956    0.951056