none
receiving streaming data in ssis RRS feed

  • Question

  • Hi I run developer 2017.  short of finding a way (if its even possible) to plumb power bi to a new streaming data source, what are my options in ssis for targeting a local or azure sql db to receive streamed data directly?  or alternatively how can a pkg receive streamed data at least into the buffer?  my understanding from Arthur z thus far is that we are in a program to program communication when we talk about streaming, outside the realm of normal packet (td?) transmissions.
    Friday, June 14, 2019 3:22 PM

Answers

  • db042190:

    SSIS is not a good choice for streaming data pipelines authoring because it is designed to operate in a batch mode whereas the streaming apps remain alive at all time with added resilience in form of redundancy (nodes in a cluster typically), although SSIS can be used to publish some data to a streaming destination (spout pattern).

    If you want to be me more versed in the streaming data processing domain take a look at the offerings in say Azure https://docs.microsoft.com/en-us/azure/architecture/data-guide/technology-choices/stream-processing

    Mind, and the following are my personal opinions:

    1. streaming data processing is much harder to get right,
    2. most companies do not need it, or do it just for hype, the traditional ETL is still good, and here to stay
    3. Kafka, and alike, does not solve many of the business cases, limited, and often un-necessary

    Arthur

    MyBlog


    Twitter

    • Proposed as answer by Yitzhak Khabinsky Friday, June 21, 2019 7:35 PM
    • Marked as answer by db042190 Friday, June 21, 2019 9:06 PM
    Friday, June 21, 2019 6:27 PM
    Moderator

All replies

  • Hi db042190,

    The following links will be helpful.

    Data Streaming Destination

    SSIS Data Streaming Destination

    Best Regards,

    Mona


    MSDN Community Support
    Please remember to click "Mark as Answer" the responses that resolved your issue, and to click "Unmark as Answer" if not. This can be beneficial to other community members reading this thread. If you have any compliments or complaints to MSDN Support, feel free to contact MSDNFSF@microsoft.com

    • Proposed as answer by ArthurZModerator Monday, June 17, 2019 7:27 PM
    • Unproposed as answer by db042190 Friday, June 21, 2019 6:14 PM
    Monday, June 17, 2019 9:54 AM
  • thx Mona, I think my question is different.  The component you showed uses an ssis pkg as a data source, at least from what I read and what I can understand.  I'm asking about publicly available data streams like stock prices etc.  I think such things exist.  Maybe like RSS feeds which I think are streamed.
    Thursday, June 20, 2019 7:32 PM
  • Hi db042190,

    Please let us know what you want to achieve with SSIS.

    Besides, PowerBI can connect to many streaming sources on its own.

    For example, I recently delivered a number or PowerBI reports that read from Azure Log Analytics. You can stream into any data, but you cannot consume the entire universe.

    Stock data for free 100% does not exist. You can search online and find some offerings that throttle you or give stale data.


    Arthur

    MyBlog


    Twitter

    Thursday, June 20, 2019 8:09 PM
    Moderator
  • thx Arthur, i'm trying to understand the underlying mechanics of consuming streamed data.   If that is possible in ssis then I think I may be able to understand it even if c# must enter the picture.  pbi seems more of a black box with only a known list of possible stream sources.  pbi didn't seem extensible in that respect but I may have missed that part.

    so any source would serve as a good example.  I assume there is a time element with some sort of web interface.

    the idea would be to capture streamed data in ssis and land the sample taken in a sql db.

         
    Friday, June 21, 2019 6:10 PM
  • db042190:

    SSIS is not a good choice for streaming data pipelines authoring because it is designed to operate in a batch mode whereas the streaming apps remain alive at all time with added resilience in form of redundancy (nodes in a cluster typically), although SSIS can be used to publish some data to a streaming destination (spout pattern).

    If you want to be me more versed in the streaming data processing domain take a look at the offerings in say Azure https://docs.microsoft.com/en-us/azure/architecture/data-guide/technology-choices/stream-processing

    Mind, and the following are my personal opinions:

    1. streaming data processing is much harder to get right,
    2. most companies do not need it, or do it just for hype, the traditional ETL is still good, and here to stay
    3. Kafka, and alike, does not solve many of the business cases, limited, and often un-necessary

    Arthur

    MyBlog


    Twitter

    • Proposed as answer by Yitzhak Khabinsky Friday, June 21, 2019 7:35 PM
    • Marked as answer by db042190 Friday, June 21, 2019 9:06 PM
    Friday, June 21, 2019 6:27 PM
    Moderator
  • thx, I don't see a spout pattern in ssis.
    Friday, June 21, 2019 9:15 PM