Integration Pack for Data Manipulation released on Codeplex RRS feed

  • General discussion

  • There has been quite a bit of discussion in various places around manipulating data that is captured and processed through Opalis Integration Server.  While Opalis Integration Server comes with many objects out of the box that make data management a breeze, there are opportunities for better data manipulation capabilities for the information on the data bus.

    This is where the Integration Pack for Data Manipulation comes in.  A series of objects that can be added into your Opalis Integration Server environment to provide extended data management capabilities.

    Integration Pack for Data Manipulation

    This Integration Pack has no official support but it can solve many of the data manipulation needs that have been identified.

    Object listing:

    • Aggregate Data – Free form population of data that can have advanced in-line data manipulation applied to create correlated published data sets.
    • Compare Multiple Values – Provides the ability to perform multiple compare operations in one object and then resolve to true based on conditions of one or more comparison results (e.g. resolve to true only if comparison 1 and comparison 2 are both true).
    • Expand Fields – Allows multiple delimited text streams to be injected and produce correlated data sets.
    • Join Fields – Allows multiple fields to be joined together by a defined delimiter (useful for putting together strings that will be referenced multiple times by other objects in the policy).
    • Match Pattern – Search for a regular expression pattern within text and return a specific match or all matches.
    • Replace Text – Search for a specific string or regular expression and then replace it with defined text.
    • Split Fields – Split a string into multiple published data instances based on a specific string or regular expression delimiter.
    • Transform Data – Allows the use of a XML based configuration file to define input fields and published data fields and apply one or more transformations to the text that is provided at runtime (e.g. format dates, pad strings, replace text etc.).
    Wednesday, June 9, 2010 3:16 PM

All replies

  • Hi,

    This IP looks very promising. Especially the Expand/Split Fields & Replace Text functionality.

    However, In noticed there's a change in calling these "Inline Functions" :-(
    I was used to call inline functions like "Len" in the following way: [Len(.....)]
    From the documentation belong to this IP, I understood it's supposed to be <@Len(....)>

    Is this a correct interpretation?

    In what way will this interfere with normal usage of the regular inline functions?

    Or is this change in calling on purpose to distinguish regular inline functions with IP-inline functions?

    Rob van Eerd

    Friday, June 18, 2010 9:37 AM
  • Hello Rob,

    The change in format for calling the inline functions has two purposes one of which you nailed.

    1. This IP is not an official component of the Opalis Integration Server system and these inline functions are provided as replacements for the built in functions.  By leveraging a different syntax, users can use either the data manipulation functions built into OIS or the functions built into the IP with a clear recognition of which functions will be resolved in which circumstances.
    2. The format for calling functions in Opalis Integration Server is flawed in that it makes it exceedingly difficult to recognize nested functions because the format actually changes when nesting.  For example:  [Len(...)] becomes [Len(Sum(...))] when nesting functions making the resolution of the Len and Sum functions behave differently.  This can be mitigated by a certain degree of human logic when nesting only 1 or 2 levels deep, but provides no consistent recursion capabilities as there may be an unknown number of nested functions intermixed with text that may or may not be a function component.  The IP requires that every function call use the same syntax of <@function()> which allows for predictable recursion.




    Thursday, June 24, 2010 1:22 AM
  • Hi Jeff,



    I do see your point about nesting / recursion. However, to me, personally, this makes it a bit too complex as I'm already used to counting round brackets and now I have to locate both round brackets AND lt/gt-signs (but I'll get used to that) ;-)



    Wednesday, June 30, 2010 1:23 PM
  • Has anyone gotten the inline functions to work?

    I added the following:


    into the 'Input String 01' input box of a Aggregate Data Properties IP.  I then saved the workflow and ran a test and OIS server didn't convert the syntax into a value of '5' and so the published data in the next object of the policy only saw the literal text.

    I also tried a similar test utilizing the old functions syntax with brackets '[]' and the conversion of text work as expected.  However, I was really looking forward to some of the new functions.

    Any ideas?  Thanks in advance.

    I'm running OIS 6.2.2 with the latest integration pack for data manipulation loaded and deployed to my client.


    Monday, October 18, 2010 2:27 PM
  • Hi akniheu,

    I just tested your example withhout your issues.

    If I check what's published in 'Output String 01', it's the expected "5" (no quotes).
    I even tried putting single quotes around the input, and after executing the policy, the value is still the same but that surrounded by the single quotes as well.

    Have you enabled logging? What is shown in the log-info?
    Are you sure the Data Manipulation IP is successfully deployed to (all) the Action Server(s) involved as well?


    Regards, Rob van Eerd
    Tuesday, October 19, 2010 6:24 AM
  • Your ideas pointed me in the right direction and were very helpful.

    My mistake was in the deployment of the ip to our action servers (we have 2 of them) and they did not properly get deployed to all of them.

    I corrected that and retested and everything is working as expected.


    Tuesday, October 19, 2010 4:27 PM
  • Hi,

    Ive been using this integ pack alot and it does wonders for me. However I recently noticed that I cannot use this integration pack when I deploy this to an action server outside of the integration server. The integ pack was deployed successfully to the action server but whenever I run the object specific published data are nowhere to be found. I tried checking the pre-requisites in the documentation but Ive already installed them all except the opalis integ server 6.2.2

    Does this integration pack have any more pre-requisites for it to work?

    Wednesday, March 27, 2013 8:49 AM