none
Python in FS4SP RRS feed

  • Question

  • Hi Experts,

    With some good experience in ESP I'm so much addicted to Python custom stages. Were as in FS4SP we are recommended to use C#..

    My questions is, what are the negative points if I one choose Python for their custom stage over C#..? Your suggestions plz..

    In the coming versions of FS4SP, will the native pipeline be in C# instead of Python? 

    Much Thanks!!

    PS: I'm sure python custom stages will work in the current version of FS4SP as I tested them myself!!


    Freddie Maize ..A story with Glory is History. Doesn’t matter whether Glory rest in the world of Demon or God. Lets create History..
    Tuesday, August 16, 2011 4:27 PM

Answers

  • Hello freddie,

    The #1 negative point is that it is unsupported.

    #2 for me is maintainability. For most SharePoint devs Python will be a language they are not familiar with, and bringing in new developers to work with Python can be an obstacle. I'm no Python expert myself, and don't know how I could easily set up unit tests for the stages I created in Python in a build environment, whereas this is very easy (for me) in Visual Studio.

    #3 Any edits you do to pipelineconfig.xml can be overwritten by a service pack or cumulative update

    #4 You have to store your custom stages in the same folders as FAST, while I like to keep my custom modules in a separate folder (I have the same grief with ESP on that one)

    I'm sure there are more ,) and there are also some benefits to using Python, but I won't go into those as that is not part of your question :)

    As for new versions, my educated guess is that the pipeline will be removed all together as-is, and be replaced with a new version of CTS, which is available for FSIS today. And CTS uses .Net modules.

    To sum it up: If you are happy with maintaining an unsupported solution, go for Python stages. If you want to work with the flow and prepare for whatever comes next, go with the pipeline extensibility instead. And remember, you are free to write pipeline extensibility modules in Python as well, not just .Net, as it calls an executable. I sometimes use PowerShell for prototyping custom stages.

    Regards,
    Mikael Svenson 


    Search Enthusiast - SharePoint MVP/WCF4/ASP.Net4
    http://techmikael.blogspot.com/
    • Marked as answer by freddieMaize Wednesday, September 12, 2012 10:06 AM
    Wednesday, August 17, 2011 9:44 AM

All replies

  • Hello freddie,

    The #1 negative point is that it is unsupported.

    #2 for me is maintainability. For most SharePoint devs Python will be a language they are not familiar with, and bringing in new developers to work with Python can be an obstacle. I'm no Python expert myself, and don't know how I could easily set up unit tests for the stages I created in Python in a build environment, whereas this is very easy (for me) in Visual Studio.

    #3 Any edits you do to pipelineconfig.xml can be overwritten by a service pack or cumulative update

    #4 You have to store your custom stages in the same folders as FAST, while I like to keep my custom modules in a separate folder (I have the same grief with ESP on that one)

    I'm sure there are more ,) and there are also some benefits to using Python, but I won't go into those as that is not part of your question :)

    As for new versions, my educated guess is that the pipeline will be removed all together as-is, and be replaced with a new version of CTS, which is available for FSIS today. And CTS uses .Net modules.

    To sum it up: If you are happy with maintaining an unsupported solution, go for Python stages. If you want to work with the flow and prepare for whatever comes next, go with the pipeline extensibility instead. And remember, you are free to write pipeline extensibility modules in Python as well, not just .Net, as it calls an executable. I sometimes use PowerShell for prototyping custom stages.

    Regards,
    Mikael Svenson 


    Search Enthusiast - SharePoint MVP/WCF4/ASP.Net4
    http://techmikael.blogspot.com/
    • Marked as answer by freddieMaize Wednesday, September 12, 2012 10:06 AM
    Wednesday, August 17, 2011 9:44 AM
  • Hi Mikael,

    Thanks for the reply!!

    >>As for new versions, my educated guess is that the pipeline will be removed all together as-is, and be replaced with a new version of CTS, which is available for FSIS today. And CTS uses .Net modules.

    I too have the same guess..

    Except for this, everything else is Okay for me.. Like I mentioned in Sub Pipeline on a conditional basis? I will have to have multiple pipelines each for one of my ContentSource. I could not think of anyother solution so I managed a python script for my FS4SP and make it work. Now I'm able to call multiple subpiplelines from my main one on conditional basis.

    With all these things in place, if the native pipeline is migrated to C# then I'm having big time problem.. 

    >>#3 Any edits you do to pipelineconfig.xml can be overwritten by a service pack or cumulative update

    I actually did not see thing happening for some reason. We recently installed SP1.

    >>The #1 negative point is that it is unsupported.

    One last thing, what do MS exactly means by unsupported? Do they say that they will not help us to build a module using Python even on a billing basis or something else, like, Don't you use it Mr. ?

    :) Much Thanks!!

    Can some MS employee shed some light on this? Especially on "replaced with a new version of CTS, which is available for FSIS today. And CTS uses .Net modules."? Much Thanks!!!!


    Freddie Maize ..A story with Glory is History. Doesn’t matter whether Glory rest in the world of Demon or God. Lets create History..
    Wednesday, August 17, 2011 10:18 AM
  • Hello Freddie,

    The pipelineconfig.xml is not overwritten, only patched, but that is not to say that they might overwrite it some time in the future. But probably not.

    As for "not supported", it's a very vague term. I suspect it means that if you have issues with your deployment you might not get help from support if you have edited files you were not supposed to.

    There is a KB article which lists "use native stages" in scenarios where you have particular needs for indexing speed, so you might be allowed to use Python if you are a huge customer with feeding speed issues... in a supported manner.

    Regards,
    Mikael Svenson 


    Search Enthusiast - SharePoint MVP/WCF4/ASP.Net4
    http://techmikael.blogspot.com/
    Wednesday, August 17, 2011 10:45 AM
  • The pipelineconfig.xml is not overwritten, only patched, but that is not to say that they might overwrite it some time in the future. But probably not.

    Hi Mikael,

    Thanks for the response!!

    Sure. I completely agree that taking a backup for pipelineconfig.xml   is A must.

    Yeah, 'Not supported' is a very vague term.. If the MS team helps in letting us know what exactly they mean by that then it will be great..

    Posting it again,

    Can some MS employee shed some light on this? Especially on "replaced with a new version of CTS, which is available for FSIS today. And CTS uses .Net modules."? Much Thanks!!!!

    Will the Python module be replaced by C#?

    Much Thanks to Mike and others!!

     


    Freddie Maize ..A story with Glory is History. Doesn’t matter whether Glory rest in the world of Demon or God. Lets create History..
    Wednesday, August 17, 2011 12:09 PM
  • Hi MS folks,

    I would like to hear from you guys.. on this.. So plz :)...


    Freddie Maize ..A story with Glory is History. Doesn’t matter whether Glory rest in the world of Demon or God. Lets create History..
    Friday, August 19, 2011 6:57 AM
  • Good thread so far; no mistakes yet that I can see. I will point out that the FS4SP pipeline stages run in a 'sandbox' which, I think, is intended to protect FS4S (and SP itself) from the errors we strive so hard to include in our pipeline code:) In the FS4SP sandbox, your code has to complete in 10 seconds, after which FS4SP returns a fail code. I can't imagine anyone would ever intentionally include a stage that takes anywhere close to 10 seconds per document.. but just sayin’..

    One approach that *might* provide some protection would be to put your custom code (Python or any language) might be to create a web services that a conventional 'supported' stage can call. You still need to complete within 10 seconds, but at least your pipeline stages are all 'supported' and you can write your web services code in Python or anything you want. As a plus, you don't have any overhead in opening data sources and other start up tasks..

     

    Miles

    Monday, February 20, 2012 8:13 PM
  • Hi,

    I know MSC created something called PEWS which is a custom stage calling out to a webservice which executes all modules you add to it. It's not public available and shouldn't be too hard to create.

    I'll also update on the term "not supported" which I managed to figure out while writing the upcoming book on FS4SP (shameless PR). It means that if you have edited a file you were not supposed to and end up calling MS support for some reason or another, they can deny support as you have changed files you were not supposed to. If your support issue is unrelated to modifying pipelineconfig.xml you could easily revert during support, or they might even help you out anyways :)

    So, doing pipelineconfig.xml modifications is not that bad imo if it solves an important issue and the solution is more maintainable compared to one following the supported route.

    You can also change the 10 second timeout for custom stages if needed... but as you say Miles, who would want or can accept 10 seconds per item :) On that note, want to improve indexing speed? Comment out any pipeline stage not needed, for example the company and location stages if you're not using them. There are others as well which can be commeted out... worth looking into if you want to improve indexing speeds.

    Regards,
    Mikael Svenson


    Search Enthusiast - SharePoint MVP/WCF4/ASP.Net4
    http://techmikael.blogspot.com/

    Monday, February 20, 2012 8:30 PM
  • Just wanted to make a comment on this thread that Mikael is completely on-point with his answers(including future releases of the product and maintenability).

    As far as support, yes, it is not supported to make many of the modifications to the config files that were routinely done with our legacy ESP product.  The list below gives an idea of which files can be changed and are still supported, while modifications to all other files is considered to be unsupported:

    http://technet.microsoft.com/en-us/library/ff354943.aspx

    Pipelineextensibility is typically the preferred/supported way for making pipeline modifications.  But what Mikael is saying is true.  Let's say you have modified some files you were not supposed to or created some custom stages, etc.  If you run into some issues, MSFT support will typically ask you to reproduce the same issue without any unsupported customizations, so it's important to keep that in mind and maybe have a stand-by  system with just default settings that could be easily accessed.  Also, if it is deemed that something got corrupted or system was put in a truly unstable state because of modifications, you may be asked to do something drastic like refeed from scratch or even reinstall if there is a suspicion of custom files being one of the contributing factors.  So you really have to remember this when going with an unsupported configuruation.


    Igor Veytskin




    Tuesday, February 21, 2012 4:25 PM
    Moderator