A very good explanation about how it works by Ben Gribaudo:
Power Query M Primer (Part 5): Paradigm
Streaming Semantics
Directly or indirectly, a M mashup is built by chaining methods. Call a function that fetches data from the source, pass its output to a function that applies a transformation, take the result of that function and pass it to another function that computes
an aggregate and so forth.
However, the just-described way of passing data can be inefficient. Implemented literally, it requires that the full output of the first method be assembled before being passed to the next method in the chain. Then, that method processes its entire input,
doing whatever it does until it has produced a full set of output. This is then passed to the next method, and so forth. Each step of the chain can involve iterating through the entire data set, once per step.
Why iterate through all the data once for each transform? Instead, why not iterate it once, feeding each item, as it’s encountered, through all the transform steps? Why not take the first item from the source, apply each transformation step to it in turn
and then collect it as part of what eventually will be output by the expression as a whole, then repeat the process for the second item, and so forth?
For lists and tables, M supports something like this automatically, without you having to do anything!
Streaming semantics allows M to pull items from lists and
tables through the chain of transforms. You simply code up M like you always have—by building the chain of methods that define the desired mashup—and M’s interpreter, when appropriate, streams items through that chain instead of executing each transformation
step in its entirety on the entire data set before executing the next step.
A moment ago, I mentioned pulling items through the chain of transforms. Why pull? Pulling allows the steps that needs data to control the amount of data that’s processed.
let
Numbers = List.Generate(()=>1, each _ <= 100000000, each _ + 1),
Filtered = List.Transform(Numbers, each _ * 2),
Result = List.FirstN(Filtered, 3)
in
Result
|
|
Above, only the first three items are needed to produce the requested output. Since data is pulled as needed by the methods that needs it,
List.FirstN causes only the first three items to flow through the method chain. Since only three items are pulled,
List.Generate only generates the first three items. This is true even though
List.Generate is coded to produce the sequence of numbers from 1 to 100,000,000. Whew! It would be wasteful to produce a 100,000,000 item list to have all but three of those items thrown away. Thankfully, with M’s streaming semantics, this didn’t
happen.
You don’t need to do anything to turn on streaming behavior. You simply assemble the processing steps in the order you want them. M takes care of streaming, when appropriate.
M allows you to code up what should be done (the logic of the computations) without needing to define specifically how that
what should be done (the flow of control). M, like other functional languages, is
declarative, not imperative. You focus on declaring the intent; M takes care of choosing the technical steps to achieve that intent.
