none
What's Function.From for?

    Question

  • I see that the November release of Power BI Desktop adds a new library function "Function.From"? What's that for?
    Thursday, November 09, 2017 2:55 PM
    Owner

Answers

  • This is to enable clever tricks where you can write functions to operate on other functions. This is largely expected to be useful only when writing a custom data connector. 

    Some usage examples:

    Function.Compose = (innerFunction as function, outerFunction as function) as function =>
        Function.From(Value.Type(innerFunction), (list) => outerFunction(Function.Invoke(innerFunction, list)))

    Function.Compose allows two functions to be composed together into a new function with the same signature as the inner function but which calls the outer function before returning. This might be useful in some error-handling or generic logging functions.

    Function.PartialApply = (function as function, arg, optional position) =>
      let
        functionType = Value.Type(function),
        parameters = Type.FunctionParameters(functionType),
        minCount = Type.FunctionRequiredParameters(functionType),
        pos = if position = null then 0 else position
      in
        Function.From(
          Type.ForFunction([
              Parameters=Record.RemoveFields(parameters, Record.FieldNames(parameters){pos}),
              ReturnType=Type.FunctionReturn(functionType)],
            if pos < minCount then minCount-1 else minCount),
          (list) => Function.Invoke(function, List.InsertRange(list, pos, {arg})))

    Function.PartialApply takes a function and produces a new function with one fewer parameter where that parameter has been replaced by a constant value.



    Thursday, November 09, 2017 2:59 PM
    Owner

All replies

  • This is to enable clever tricks where you can write functions to operate on other functions. This is largely expected to be useful only when writing a custom data connector. 

    Some usage examples:

    Function.Compose = (innerFunction as function, outerFunction as function) as function =>
        Function.From(Value.Type(innerFunction), (list) => outerFunction(Function.Invoke(innerFunction, list)))

    Function.Compose allows two functions to be composed together into a new function with the same signature as the inner function but which calls the outer function before returning. This might be useful in some error-handling or generic logging functions.

    Function.PartialApply = (function as function, arg, optional position) =>
      let
        functionType = Value.Type(function),
        parameters = Type.FunctionParameters(functionType),
        minCount = Type.FunctionRequiredParameters(functionType),
        pos = if position = null then 0 else position
      in
        Function.From(
          Type.ForFunction([
              Parameters=Record.RemoveFields(parameters, Record.FieldNames(parameters){pos}),
              ReturnType=Type.FunctionReturn(functionType)],
            if pos < minCount then minCount-1 else minCount),
          (list) => Function.Invoke(function, List.InsertRange(list, pos, {arg})))

    Function.PartialApply takes a function and produces a new function with one fewer parameter where that parameter has been replaced by a constant value.



    Thursday, November 09, 2017 2:59 PM
    Owner
  • Thanks Curt. I guess you expected someone like me to raise this question and didn't want to wait for that?

    Anyhow, I bet people like me will start exploring these new functions.
    Just mention "clever tricks" and you have us challenged...

    To be continued.

    Thursday, November 09, 2017 3:18 PM
  • This is a rare instance where the built-in description for a new function makes sense and has reasonably good examples! (for Function.From, that is). I assume that Compose and PartialApply exist in the custom data connector library only, since they don't appear in the Power BI Desktop list of functions.

    Curt, given that there's now Function.PartialApply, does it mean that we're likely to see a "Function.Curry" function in the future?


    Thank you very much for the details, by the way.
    Thursday, November 09, 2017 4:40 PM
  • Silly me. On re-reading Curt's post, I just realized that Function.Compose and Function.PartialApply are custom functions. I got tricked because other functional languages have similarly named functions, and my brain just extrapolated these names into new M functions. :D

    The one thing I don't understand though is that, according the built-in documentation and examples, the first parameter in Function.From should be a function type signature, which is not the case in Curt's examples.

    Friday, November 10, 2017 1:37 PM
  • Doh! I used examples from our internal library review without checking that they work with the final version -- so yes, the parameters are in the wrong order and I will fix them by editing my comment.

    EDIT: Wow, I really botched these. That's what I get for going against my notes without actually testing before posting :(.

    Friday, November 10, 2017 2:10 PM
    Owner
  • Quick tests:

    Function.Compose(Number.Mod, Number.Sqrt)(11,3) = 1.4142135623730951

    Function.PartialApply(Number.Mod, 11)(3) = 2

    Brilliant! So the function type parameter in Function.From doesn't have to be a custom type as implied in the built-in function examples. In Function.Compose, Function.From accepts a primitive function type, and derives a list from the supplied arguments.

    Friday, November 10, 2017 3:52 PM
  • That's cool!

    But I must admit that I got used to my function "IChain" already. I find the function arguments are in a better place there, especially when reading the function:

    IChain = (x as any, operations) => List.Accumulate(operations, x, (state, current) => Function.Invoke(current{0}, {state}&List.Skip(current)))

    To be used like this:

    IChain(11, {{Number.Mod,3}, {Number.Sqrt}})

    So the first argument x is the starting value and the second argument "operations" is a list of the functions with (optional) parameters.

    Wonder what you guys think of this?


    Imke Feldmann - MVP Data Platform (PowerBI) - http://www.TheBIccountant.com 

    Please use the 'Mark as answer' link to mark a post that answers your question. If you find a reply helpful, please remember to vote it as helpful - Thanks!


    Wednesday, November 22, 2017 4:35 PM
    Moderator
  • Hi Imke,

    What you're done is very different to the intent of function composition, or partially applying a function, which are both common concepts in virtually every commercial functional language.

    Function Composition

    The intent of function composition is simply to create a new function from other functions. So in my example, the new function is Function.Compose(Number.Mod, Number.Sqrt). If you were to enter this function in the query editor, it will display a window indicating what parameters are expected, just like any other function. You might use this composed function in a custom column, where it takes one or more of its arguments from other columns. I think that composition is useful when you want to create an ad-hoc function that is not sufficiently general to warrant breaking out into a general-purpose custom function. "Modding" and then squaring a function is certainly not general-purpose in nature. Curt also mentions a couple of use cases.

    There are some limitations however. If the outer function takes more than one parameter, you have to modify the compose function to handle the additional parameters, and things start looking a bit messy:

    Function.Compose = (innerFunction as function, outerFunction as function, optional additionalParms as list) as function =>
    if additionalParms = null then
        Function.From(Value.Type(innerFunction), (list) => outerFunction(Function.Invoke(innerFunction, list)))
    else
        Function.From(Value.Type(innerFunction), (list) => Function.Invoke(outerFunction, {Function.Invoke(innerFunction, list)} & additionalParms))

    So, if instead of Number.Sqrt, we wanted the outer function to be Number.Power to square the result of the mod, we can then use our modified Function.Compose as follows:

    Function.Compose(Number.Mod, Number.Power, {2})

    and then supply parameters for inner function as required, e.g. Function.Compose(Number.Mod, Number.Power, {2})(11, 3).

    It would be a design decision to determine if the composed function should take only unary outer functions, or whether it is worth the additional complexity to support outer functions with multiple parameters.

    Partially Applied Functions

    In virtually all commercial functional languages, functions are curried, meaning that functions with multiple parameters are treated internally as a series of one-parameter functions. This allows you to invoke functions without having to pass all of the parameters, which can be useful in certain scenarios. M contains specific functions that can be partially applied - All Splitter & Combiner functions are partially applied, meaning that they return functions (known as closures) that take a parameter. M has built-in functions that use splitter and combiner functions, but let's look at a general purpose scenario where partial application can be useful.

    In M, we have built-in functions to split text only by a delimiter, or a list of delimiters (Text.Split and Text.SplitAny). However, what if we wanted to split text by positions, lengths, ranges, etc.? We could create separate functions to split text however we wish, using an appropriate splitter function. So we can have Text.SplitByPositions, Text.SplitByLengths, Text.SplitByRanges, etc., or we can have one function - Text.SplitBy, which uses a splitter function as a parameter:

    Text.SplitBy  = (text as text, splitter as function) => splitter(text)

    and if we wanted to split by say, lengths, we would then do:

    Text.SplitBy("ImkeFeldmanColinBanfield", Splitter.SplitTextByLengths({4, 7, 5, 8}), which returns the list "Imke", "Feldman", "Colin", "Banfield"

    The point here is that Function.PartialApply could potentially provide similar benefits to splitter and comparer functions by creating our own partially applied functions.

    Function Pipelining

    Another common feature in functional languages is function pipelining, and this is exactly what your function does!!! So intentionally or not, you've demonstrated how to perform function pipelining in M :) Function pipelining is a useful alternative to nesting functions. I would name your function "Function.Pipe" or "Function.Pipeline". You've done a great job on this one.

    Friday, November 24, 2017 5:54 PM
  • Hi Colin,

    thank you very much for taking your time to write such a good description of what these concepts are, this was extremely helpful for me! 

    I think you've answered one of my remaining mysteries in the M-language: How does for example the record come into the 3rd parameter of Table.AddColumns? There a function is expected, but the way the function is written means that the record (current row) must have been passed into the function as the first parameter already. Is that what Partially Applied does? (Or does M have a row-context like DAX has ;) ? )

    BTW: Credits to IChain go to Kim Burgess. I just spent half a day or so to shorten the syntax :) 


    Imke Feldmann - MVP Data Platform (PowerBI) - http://www.TheBIccountant.com 

    Please use the 'Mark as answer' link to mark a post that answers your question. If you find a reply helpful, please remember to vote it as helpful - Thanks!

    Friday, November 24, 2017 7:17 PM
    Moderator
  • "Is that what Partially Applied does? (Or does M have a row-context like DAX has ;) ? )"

    This is a row context. Consider that we can simply use something like each [Qty] * [Unit Price]. Calculated columns in Excel tables also use row contexts. I suppose that we don't need to explicitly use the term "row context" in Excel or M because, unlike DAX, we don't have to make a distinction between that and the "filter context."

    Remember that when we write: each [Qty] * [Unit Price], this is syntax sugar for: each _[Qty] * _[Unit Price], where the underscore refers to the current record. Also, it's syntax sugar for: (rec) => rec[Qty] * rec[Unit Price], where rec refers to the current record.


    Friday, November 24, 2017 8:17 PM
  • Really?

    How about Table.Group then for example? There the function syntax (in the 3rd arguments) imply that the table-partition is passed in as an argument. Is that a "table context" then?


    Imke Feldmann - MVP Data Platform (PowerBI) - http://www.TheBIccountant.com 

    Please use the 'Mark as answer' link to mark a post that answers your question. If you find a reply helpful, please remember to vote it as helpful - Thanks!

    Friday, November 24, 2017 8:29 PM
    Moderator
  • "How about Table.Group then for example? There the function syntax (in the 3rd arguments) imply that the table-partition is passed in as an argument. Is that a "table context" then?"

    Well, I've never thought of that. If we examine a step like:

    Table.Group(Source, {"CustomerID"}, {"Total", each List.Sum([Price]), type number})

    we know that the table is filtered by the current CustomerID value. So the filtered table must be implicitly passed to the List.Sum function, since List.Sum needs a list in the form tableName[Price]. When we use the form: each List.Sum(_[Price]), or (i) => List.Sum(i[Price]), the "_" or "i" must represent a table. 

    I would consider this scenario to be a "Grouped" row context. The reason being that if we didn't group, but created a new column that aggregated the whole Price column, we get the same total in each row of the new column. In other words, for each row, we are calculating the same total in a row context. Also consider that in DAX, if we use an expression like SUMX(RELATEDTABLE(Sales), Sales[SalesAmount]) in a dimension table calculated column, the sales table will be filtered by the current value in the dimension table, the SalesAmount column in the filtered table is summed, and the result placed in the calculated column. This calculation is occurring in a row context, even though each row is being passed a table, similar to what's happening in Table.Group.

    Friday, November 24, 2017 10:39 PM
  • Hi Colin,

    thanks for explanation. I was clearly overcomplicating things here.

    Also think that introducing a term like "row-context" here doesn't make sense.

    My mental model looks like this now:

    "These arguments are function expressions for functions who will only hold one parameter. That parameter will be passed to the function when the "main function" is invoked. The content of the parameter is “coded” in the function itself and it is the current row for Table.AddColum for example but can be sth else in other functions. It is basically the "relevant item", so could be an item of a list as well. 

    Because we cannot influence the content of the (single) parameter of those functions, there is no need to have an identifier for them, hence using syntax sugar is possible: "each" allows us to write expressions where a explicit reference to that parameter in the function expression can be omitted for  specific expressions. But if one uses “each” and needs to use a formula where an explicit reference to that parameter has to be made, the underscore _ will act as the identifier. If we have to nest those statements, we can define our own identifiers to distinguish between them, like this (Inner) =>.”


    Imke Feldmann - MVP Data Platform (PowerBI) - http://www.TheBIccountant.com 

    Please use the 'Mark as answer' link to mark a post that answers your question. If you find a reply helpful, please remember to vote it as helpful - Thanks!

    Saturday, November 25, 2017 8:51 AM
    Moderator
  • Hi Imke,

    Now you are oversimplifying things :-).

    The inner function may have multiple parameters, e.g. when Comparer functions are involved.

    This would be an example of sorting a column case insensitive:

    Table.Sort(MyTable, (x,y) => Comparer.OrdinalIgnoreCase(x[SortColumn],y[SortColumn]))

    Otherwise, Table.Sort accepts various formats of the second parameter, comparisonCriteria:

    = Table.Sort(MyTable,{{"SortColumn", Order.Ascending}})
    
    = Table.Sort(MyTable,{"SortColumn"})
    
    = Table.Sort(MyTable,each [SortColumn])

    I do have a theory how sorting would actually work in the background, i.e. all values are compared with all other values, with results -1 if smaller, 0 if equal, 1 if larger. These values are summed and then the table is sorted on those values.

    This theory is illustrated in the following query:

    let
        // The Comparer function to be used:
        Comparer = Comparer.OrdinalIgnoreCase,
        // Table with 1 column TextList with examples texts:
        Tabled = let
            Texts = {"Book", "Encyclopaedia", "encyclopedia", "encyclopædia", "wikipedia"},
            TextList = List.Transform(Texts, Text.Lower)&List.Transform(Texts, Text.Upper),
            Tabled = Table.FromColumns({TextList},type table[TextList = text])
        in
            Tabled,
    
        // Create lists with each text item paired with all items:
        AddedEachTextWithAllTexts = Table.AddColumn(Tabled, "EachTextWithAllTexts",
                      (This) => List.Transform(Tabled[TextList], each {This[TextList], _})),
    
        // Use the Comparer function to score all pairs and sum the scores:
        AddedScore = Table.AddColumn(AddedEachTextWithAllTexts, "Score",
           each List.Sum(List.Transform([EachTextWithAllTexts], each Comparer(_{0},_{1})))),
    
        // Sort on score:
        SortedOnScore = Table.Sort(AddedScore,{{"Score", Order.Ascending}}),
    
        // Remove temporary columns:
        #"Removed Columns" = Table.RemoveColumns(SortedOnScore,{"EachTextWithAllTexts", "Score"})
    in
        #"Removed Columns"
    Again, this is just a theory. Maybe an insider can either confirm or nuance it.

    Saturday, November 25, 2017 10:02 AM
  • Hi Marcel,

    yes I might have oversimplified here, but in my eyes, your arguments don't prove this ;-)

    Lets take this expression:

        AddedScore = Table.AddColumn(AddedEachTextWithAllTexts, "Score",
           each List.Sum(List.Transform([EachTextWithAllTexts], each Comparer(_{0},_{1}))))

    How many functions does it contain? 4 or 6?

    4 functions whose arguments you've specified (Table.AddColumn, List.Sum, List.Transform, and Comparer) and 2 functions ("each") whose (single) argument you cannot specify: Its each row/record for the first "each"-function and each list item for the second "each"-function. So still: These inner-functions only contain one argument. In the Comparer-function (which needs 2 arguments) the argument is referenced twice, but that doesn't make it 2 (different) arguments.

    Or did I miss sth?

    Interesting theory and very nice implementation of the sort-algo!

    It would be great if you could show me an implementation of this:

    Table.Sort(MyTable, (x,y) => Comparer.OrdinalIgnoreCase(x[SortColumn],y[SortColumn]))

    Neither do I understand what 2 different records/tables (?) x and y shall represent, nor did I manage to directly use "Comparer.OrdinalIgnorCase" in a Table.Sort-Expression at all (what you did in your example was to use it as a stand-alone function within a Table.Transform-expression, creating a helper-column that was then used for the sort-reference).

    Thanks & cheers :)


    Imke Feldmann - MVP Data Platform (PowerBI) - http://www.TheBIccountant.com 

    Please use the 'Mark as answer' link to mark a post that answers your question. If you find a reply helpful, please remember to vote it as helpful - Thanks!

    Saturday, November 25, 2017 12:08 PM
    Moderator
  • Maybe this is a better description:

    We have 4 expressions where a function is invoked (the library-functions) and 2 expressions where a function is written/defined (the each-functions).

    The invocation of these (each-) functions happens when their outer/main functions are evaluated. The parameter that will be passed in that process is out of our control, but coded in the main functions already (as is the 11 in Colins "Function.PartialApply(Number.Mod, 11)(3) btw, but I see that this is different to what we have here, as there is no request for 2nd "real" function parameter (like the 3) ).

    Does that make sense?


    Imke Feldmann - MVP Data Platform (PowerBI) - http://www.TheBIccountant.com 

    Please use the 'Mark as answer' link to mark a post that answers your question. If you find a reply helpful, please remember to vote it as helpful - Thanks!

    Saturday, November 25, 2017 2:59 PM
    Moderator
  • Basically my statement is that the outer function must supply 2 parameters to the inner function, in case the inner function is a comparer function. The proof is that you get an error message, stating that 2 arguments are expected, if you invoke a Comparer function with 1 argument, e.g. =Comparer.Ordinal("a").

    Otherwise it looks like both parameters are supplied with the same value, i.e. the record in case of Table.Sort.
    Indeed it is not very clear what would be the first, and what would be the second parameter, but I think the first parameter is the actual sort value and the second parameter is the reference value: see the examples below with custom function in which x and y values are compared in the comparisonCriteria argument of List.Sort.

    A complicating factor is that sort functions accept their comparisonCriteria parameter in various formats, I already provided some examples with Table.Sort in my previous post; examples with List.Sort include:

    - no parameter (with List.Sort, the comparisonCriteria parameter is optional)
    - a value (Order.Ascending is just 0)
    - a function with 1 parameter (each _, or (x) => x)
    - a function with 2 parameters

    let
        Source = {"a".."z","A".."Z"},
    
    //  5 Examples of regular sort (ascending):    
        Sorted1 = List.Sort(Source),
        Sorted2 = List.Sort(Source, Order.Ascending),
        Sorted3 = List.Sort(Source, each _),
        Sorted4 = List.Sort(Source, (x) => x),
        Sorted5 = List.Sort(Source, (x,y) => if x < y then -1 else if x > y then 1 else 0),
    
    //  Sort descending:
        Sorted6 = List.Sort(Source, (x,y) => if x < y then 1 else if x > y then -1 else 0),
    
    //  Sort case insensitive, first ascending then descending:
        Sorted7 = List.Sort(Source, Comparer.OrdinalIgnoreCase),
        Sorted8 = List.Sort(Source, (x,y) => -1 * Comparer.OrdinalIgnoreCase(x,y)),
    
    // The next line throws an error: we cannot convert the value "a" to type Number.
        Sorted9 = List.Sort(Source,(x,y) => x)
    in
        Sorted9

    An interesting implementation is sorting a table case insensitive and descending, this can be done by multiplying the result from Comparer.OrdinalIgnoreCase by -1 (btw I didn't use this in a Table.Transform expression; it was a different and standalone example):

    let
        // Table with 1 column TextList with examples texts:
        MyTable = let
            Texts = {"Book", "Encyclopaedia", "encyclopedia", "encyclopædia", "wikipedia"},
            TextList = List.Transform(Texts, Text.Lower)&List.Transform(Texts, Text.Upper),
            Tabled = Table.FromColumns({TextList},type table[TextList = text])
        in
            Tabled,
    
        SortedOnTextList = Table.Sort(MyTable, (x,y) => -1 * Comparer.OrdinalIgnoreCase(x[TextList],y[TextList]))
    in
        SortedOnTextList

    Another different example is a complex sort on 2 columns: first on TextList descending, using culture en-US, ignore case, and next on Fraction descending, using high precision:

    = Table.Sort(MyTable, (x,y) =>
                let
                    Sort1 = -1 * Comparer.FromCulture("en-US",true)(x[TextList],y[TextList]),
                    Sort2 = if Sort1 <> 0 then Sort1
                                          else -1 * Value.Compare(x[Fraction],y[Fraction],1)
                in
                    Sort2)
    

    Saturday, November 25, 2017 6:42 PM
  • Hats off Marcel!

    This is truly awesome discovery.

    So would you agree to the following statement then?:

    "These arguments are function (-definition) expressions for functions who will be invoked in the background. The function parameters will be passed to the function when the "main function" is invoked. The content of the parameters is “coded” in the function itself and it is the current row for Table.AddColum for example but can be sth else in other functions. It is basically the "relevant item", so could be an item of a list as well (or 2 items when used with a comparer function or List.Accumulate for example)

     

    As long as there is no need to have an identifier for them, using syntax sugar is possible: "each" allows us to write expressions where an explicit reference to that parameter in the function expression can be omitted for certain expressions. But if one uses “each” and needs to use a formula where an explicit reference to a parameter has to be made, the underscore _ will act as the identifier (this will only work if there is just one parameter). If we have to nest those statements or there are multiple parameters passed into the function, we can define our own identifiers to distinguish between them, like this (Inner) => or (x,y) =>. "


    Imke Feldmann - MVP Data Platform (PowerBI) - http://www.TheBIccountant.com 

    Please use the 'Mark as answer' link to mark a post that answers your question. If you find a reply helpful, please remember to vote it as helpful - Thanks!




    Saturday, November 25, 2017 8:10 PM
    Moderator
  • Hi Imke,

    Looks fine to me, except the reference to Comparer.OrdinalIgnorCase, which I would replace by the more generic a comparer function (note: apart from the Compare.xxx functions,, also Value.Compare is a comparer function).

    Sunday, November 26, 2017 1:19 PM
  • Thanks Marcel,

    edited the thread above :)


    Imke Feldmann - MVP Data Platform (PowerBI) - http://www.TheBIccountant.com 

    Please use the 'Mark as answer' link to mark a post that answers your question. If you find a reply helpful, please remember to vote it as helpful - Thanks!

    Sunday, November 26, 2017 9:58 PM
    Moderator
  • This returns the number 55. How does this work or better why does this work?

    = Function.From (
        type function(A as table, B as record) as text,
        List.Sum) ( 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 )

    Monday, December 11, 2017 3:07 PM