Feature #541

Rethinking OML's config file

Added by Max Ott about 1 year ago. Updated about 1 month ago.

Status:New Start date:08/05/2011
Priority:High Due date:
Assignee:Olivier Mehani % Done:

0%
Category:client
Target version:2.8.0

Description

The client config file was initially designed in haste with little reflection and is therefore quite confusing. Especially the relationship between measurement points and measurement streams collides here. The config file is essentially defining a dataflow, from the MPs inside the client app to what goes out over the net or into a local file. Another way to think about it, is as a set of new measurement streams which are created from a set of filters on elements of other measurement streams.

The following is an attempt to capture that. What is different to the current format is the renaming of 'collect' into 'bundle' which is rather cosmetic, and the notion that a filter can have more than one input. The latter would require substantial re-architecting of the filter framework.

I also included the source stream as part of the input parameter's name. This automatically treats MPs in the app as streams.

The last stanza defines the elements of a stream. While this is not necessary for configuring the filter arrangements, it could be included in a default config file the client library could create (using a new flag) which includes all the available streams to start from.

This more generic view would require that filters take their input from a stream and create a elements for a new stream, which then can be fed into another set of filters. Right now filters write into file, or net streams which are somewhat different from measurement streams.

<omlc>
  <bundle url="file:-" encoding="text">
    <stream name="video"/>  <!-- forward stream with identical name -->

    <stream name="video" interval="1" />  <!-- create stream with default filters and identical name -->

    <stream name="video_overview" interval="1"> 
      <f name="default" in="video.id" />  <!-- 'video.id' is the 'id' parameter from the 'video' stream (which may be a MP) -->
      <f pname="video.psnr" />  <!-- use 'default' for default filter name -->
    </stream>

    <stream name="video_2" interval="1"> 
      <f name="delta" in="video.id"  out_prefix="delta-frame-d" />
      <f in="video.psnr" />   <!-- 'psnr' would be default 'out_prefix' -->
      <f name="stddev" in="video.psnr" out_prefix="psnr-stddev"/>
    </stream>
  </bundle>

  <bundle url="tcp:oml.acme.com" encoding="binary">
    <stream name="video_2"/>  <!-- include above defined stream -->

    <stream name="video_3">
      <f name="combiner" in="video_2.psnr_max, video_2.psnr_min"/> <!-- multiple input paramaters -->
      <f name="combiner"> <!-- multiple input paramaters (more XML like) -->
    <in name="video_2.psnr_max" />
    <in name="video_2.psnr_min"/>
      </f>
    </stream>
  </bundle>

  <stream name="remote"> <!-- defines all the elements of a stream -->
                         <!-- Could be used as def. config file listing all available streams -->
    <e name="psnr.avg" type="xsd:float"/>
    <e name="psnr.min" type="xsd:float"/>
    <e name="psnr.max" type="xsd:float"/>
  </stream>
</omlc>

Related issues

related to Feature #435: Allow users to disambiguate measurement streams from diff... Closed 23/09/2010
related to Feature #699: Allow OML initialisation from something else than command... New 21/03/2012

History

#1
Updated by Olivier Mehani about 1 year ago

That looks similar to what Jolyon and I have been discussing at coffee breaks (:

A couple of comments.
  • I don't like one letter elements. XML is not meant to be concise. It may be better (i.e. clearer to the occasional human reader) to spell their name completely. Also, I'm not sure if there is a semantic difference between f and e elements; I have the feeling they are the same logical concept which can be defined in two ways (filter of another field, or type), and each of those two elements implement one specific way.
  • I'm not sure about using strings namespaced in plain text e.g. , name="video_2.psnr_max". Ultimately, this adds complexity to the parsing as it is necessary to first parse the XML tree, then parse the attribute. For that previous example, I'd think stream='video_2' name='psnr_max' would be more logical. The same stands for URIs. In this case, they don't really refer to another XML document, as HTML href or RDF resource do, but contain data which needs to be understood by the application parsing the XML file. URI are just a human-friendly way to summarise this information; it's fine on the command line, but not for an XML document. A more semantic separation of the components would be better, to be readily accesible from the XML tree: server='oml.acme.com' port='3003' transport='tcp' or file='-' (Though, how is the difference made between a file named - and stdout? Maybe there is a need for a pipe attribute for the latter case. URI syntax is not necessarily in the scope of the last suggestion.)
  • For the same reason as above, I don't like multiple input fields in a single attribute, as in <f name="combiner" in="video_2.psnr_max, video_2.psnr_min"/>, as it requires additional parsing that is not XML-based for all the semantics of the document to be obtained. I much prefer the second option. Actually, I would go as far as saying that this syntax (the second one) should be the default, even for a single input. I'm not saying that we should drop the in attribute for f (for lack of a clearer name (;) elements, but we should keep in mind that this is just syntactic sugar, when there is only one input, for the more verbose, in-child-element-based syntax.
  • Though I don't disagree with it, the concept of out_prefix should be clarified. I suspect there are times when it's not needed.

I think that's about all.

#2
Updated by Thierry Rakotoarivelo about 1 year ago

Hi Max and all,

I would like to also add the support in the config file of user-defined metadata, such as the unit of each element in a stream. As you may remember, we had that discussion a while ago about the current lack of metadata about the measurement in the resulting OML database.

The unit of a measured variable is for me a must-have one, as we should be able to share data with our peers without needing to sent a separate email saying "by the way, this collected measurement is in Kbit.s-1". Also on the same subject, I would suggest that OML automatically compute the unit of integrated variables (i.e. the result of a filter) if that unit is not specified explicitly in the filter definition. So if the filter designer explicitly set the unit(s) of the output(s) then take it, if not then compute it from the different units of the inputs. For example: [bits] / [sec] -> [bits*sec-1] ,or another example: [sec] / [sec] -> [unitless]

#3
Updated by Thierry Rakotoarivelo 7 months ago

  • Parent task set to #592

#4
Updated by Thierry Rakotoarivelo 7 months ago

  • Target version set to 2.7.0
  • Parent task deleted (#592)

#5
Updated by Thierry Rakotoarivelo 7 months ago

  • Assignee set to Olivier Mehani
  • Priority changed from Normal to High

#6
Updated by Thierry Rakotoarivelo about 1 month ago

  • Target version changed from 2.7.0 to 2.8.0

Moved to 2.8 as agreed with Olivier.

Also available in: Atom PDF