locked
Custom MP with multiple file import: where to put the logic? Connector vs Workflow vs Services RRS feed

  • Question

  • Hello!

    Working on my first MP, still with 2012 R2, I still have comprehension issues about the whole architecture and I hope I could get some recommendations here.

    1. I created my own objects and relationship,
    2. Now I need to import multiple files (typically CSV files),
    3. Work the data between those files, preferably with C# code (I already have my algo/object model behind)
    4. and create and update the appropriate object from step 1.
     

    I am using both VSAE in Visual Studio 2015 and the Service Manager Authoring Tool (AT) to create my MP and then the Service Manager Console (SMC). 
    At the first look, the Authoring Tool and the Manager Console seems to be restricted to a limited amount of activities and connectors, but I suppose it is all about creating your own connector type and/or custom activities.

    The default CSV import connector support only ONE file as input, it is no use for me.

    I am following this tutorial to create a custom connector:
    https://blogs.technet.microsoft.com/servicemanager/2009/12/30/how-to-write-a-custom-connector-csv-connector-example/

    Are they more recent connector tutorials around?

    I also see this guidelines about custom activities: https://technet.microsoft.com/en-us/library/hh495648(v=sc.12).aspx

    So the main basis question: where should I put my import and business logic?

    1. in an custom activity within my ManagementPack?
    2. in a custom Connector outside?
    3. a bit of both?
    4. or something else? E.g. I see the default Orchestrator Web Service connector, but seems even more complicated...

    Thanks for your recommendations!

    Eric

    Wednesday, October 19, 2016 5:19 PM

Answers

  • It looks like you may be hung up on "connectors", "activities", etc. Connectors, in Service Manager parlance, are nothing more than "a means to import or export data"..(typically import). So connectors can be a console task (like the CSV connector) or it can be a workflow (like the AD connector). It looks like you're more interested in the console task approach, so don't worry about custom activities for this particular project since those are workflow related.

    When you ask if you should keep your code in your MP, I'll assume you're talking about a custom powershell workflow or console task (which can be defined in an MP).

    I've never done it..I always roll my own C# DLLs that I either copy to the management server (in the case of workflows) or bundle with the MP (in the case of console tasks, forms, etc).

    There are some advantages and disadvantages to both approaches.

    The MP approach makes it much easier to tweak your code (tweak, re-import, re-run). It also makes it much easier to screw it up by importing the wrong MP, getting your versions out of sync, etc.

    Testing DLLs can be a bit tedious. In the case of workflows, the Microsoft Monitoring Agent keeps a lock on your DLL after it runs your workflow..so to replace it, you have to restart that service. In the case of console/form DLLs, you have to restart the console after every time you update your MP to ensure your console gets the latest version of your code.

    The DLL approach is more flexible than Powershell since you can use the SDK. (I know, you can use the SDK directly in Powershell, but have you ever tried it? It's a nightmare :) ).

    Long story short: Business logic in compiled DLLs, definitions and calling business-logic in your MP.

    • Marked as answer by EricBDev Thursday, October 20, 2016 9:49 AM
    Wednesday, October 19, 2016 6:29 PM

All replies

  • It looks like you may be hung up on "connectors", "activities", etc. Connectors, in Service Manager parlance, are nothing more than "a means to import or export data"..(typically import). So connectors can be a console task (like the CSV connector) or it can be a workflow (like the AD connector). It looks like you're more interested in the console task approach, so don't worry about custom activities for this particular project since those are workflow related.

    When you ask if you should keep your code in your MP, I'll assume you're talking about a custom powershell workflow or console task (which can be defined in an MP).

    I've never done it..I always roll my own C# DLLs that I either copy to the management server (in the case of workflows) or bundle with the MP (in the case of console tasks, forms, etc).

    There are some advantages and disadvantages to both approaches.

    The MP approach makes it much easier to tweak your code (tweak, re-import, re-run). It also makes it much easier to screw it up by importing the wrong MP, getting your versions out of sync, etc.

    Testing DLLs can be a bit tedious. In the case of workflows, the Microsoft Monitoring Agent keeps a lock on your DLL after it runs your workflow..so to replace it, you have to restart that service. In the case of console/form DLLs, you have to restart the console after every time you update your MP to ensure your console gets the latest version of your code.

    The DLL approach is more flexible than Powershell since you can use the SDK. (I know, you can use the SDK directly in Powershell, but have you ever tried it? It's a nightmare :) ).

    Long story short: Business logic in compiled DLLs, definitions and calling business-logic in your MP.

    • Marked as answer by EricBDev Thursday, October 20, 2016 9:49 AM
    Wednesday, October 19, 2016 6:29 PM
  • Thanks for your input! And great: BL in compiled DLLs is what I intend to do as well and indeed don't want to fight with .NET code within Powershell!

    When I said "inside an activity in the MP", I was more thinking about coding a custom activity in C#, even if I don't understand yet how it is called from the workflow!

    My requirement is indeed a multiple CSV import, but to be done once a day, so that I update my objects from my new input data.

    But I can also configure a connector to be run once a day, right?

    Next step is now how to deal with the custom objects of my MP so that I can instanciate them within the C# DLL.

    And curious about testing as well when I read your comment. 
    I come from multi tier application development with DAL over Entity Framework and Test projects in Visual Studio and run automatically with TFS. Wonder if/how I can get the same approach with SCSM development!

    By the way, I expect to see the custom objects created in my own MP translated into SQL tables: where are they stored? In ServiceManager database or in an extra one?

    But at the first place, I need to find where to download the Service Manager SDK!!
    Microsoft list only the SDK Documentation

    And the only download link I found so far is for the Operation Manager SDK: https://msdn.microsoft.com/en-us/library/hh329086.aspx

    I ve VSAE and AuthoringTool installed on my dev VM, but when trying the demo custom connector solution, I am still missing some references, like Microsoft.EnterpriseManagement.UI.xxx and Microsoft.EnterpriseManagement.ConsoleFramework for the ConsoleCommand base class. I suppose these are provided by the SDK.


    • Edited by EricBDev Thursday, October 20, 2016 12:31 PM
    Thursday, October 20, 2016 12:23 PM
  • The SDK is not a separate download, it's part of the SCSM install. It's really just the DLLs included when you install the management server or the console.

    However, depending on what classes/namespaces you want to re-use, you may have to grab the DLLs from your console's cache directory. (c:\users\<me>\appdata\local\microsoft\microsoft system center). For the most part, however, all the DLLs you'll need are in the installation directory of a management server..so just copy them to your development environment and reference them.

    Custom workflows are called from management packs via the <Rule> node. Rules contain a trigger <Criteria> and an action <WriteAction>. So, you define your trigger..like "run this when my service request changes from new to in progress". The SCSM framework manages and monitors triggers for you. When the framework sees the trigger, it executes whatever is defined in the <WriteAction>..in this case, it can be your custom connector workflow.

    If you're planning on building a workflow connector, then there's a good chance you're going to need a "console task" portion of your connector as well..this will let you run your connector at will, as well as configure it if necessary. An example is the AD connector..you can create, edit, and "synchronize" an AD connector via console tasks. But the actual work is done by a workflow.

    When you say "custom objects of your MP", what are you referring to?

    As for testing, yes, you can configure TFS to deploy and release your code (and probably even run automated tests, but I've never done that with my SCSM environment). Personally, I just use post-build events in Visual Studio to kick off powershell that seals, bundles, and imports my MPs. If you do it that way, here's a tip..when you're importing an MP bundle, the import routine maintains a lock on your file, so you can't re-bundle the MP until you close powershell or the console or whatever you used to import it. And so, I have a "batch" file that spins up a separate powershell session from within powershell to import MPs.

    Most everything you define in an MP is stored in the ServiceManager database in some way or another. Classes create new tables and views. (My.Custom.Class -> MT_My$Custom$Class table in SQL). Object Templates are stored in the ObjectTemplate table. (Get used to using convert(xml,...) in your SQL queries so you can see the XML ;) ). Rules are stored in the Rules table (weird, I know). "Resources" (console based DLLs) are stored in the Resource table. Workflow DLLs, however, are only stored on management servers, not in the database. Only the Rules and WriteActions are stored in the database.


    Thursday, October 20, 2016 1:31 PM
  • Thanks for your detailed answer!

    Yeah, I figured out myself the DLL I was looking for are on my productive machine where Service Manager is installed, but not on my development machine. So I simply copied/pasted those dll from my prod to my dev env.

    Thursday, October 20, 2016 2:40 PM
  • With "custom objects of your MP", I mean the classes I've defined in my MP which get eventually stored in the SQL tables. Let's call them "XML classes".

    They are defined in xml in the MP. Can I get the .NET variant of them (as if they were C# classes)? Can I can instance them in my connector with Intellisense support in Visual Studio, or should I go with plain string?

    In Visual Studio, I have:

    1. one ManagementPack project to define the XML classes
    2. one ClassLibrary project to define the connector. 

    Nothing directly related though, but what is the recommanded way to refer to xml classes in the connector?
    Friday, October 21, 2016 9:21 AM
  • 2nd part: SQL and relationships: (answer found below - irrelevant)
    As expected, I found my XML classes in ServiceManager database as  MT_MyCustomMP$Class but I only see the plain tables, the relationships are not translated in SQL.

    I go with an exmple: first the xml from my MP (as in the xml output file from my ManagementPack Project in VisualStudio), and below the generated tables.

            <ClassType ID="AbacusManagementPack.Company" Base="System!System.ConfigItem" Accessibility="Public" Abstract="false" Hosted="false" Singleton="false">
              <Property ID="CompanyId" Type="string" AutoIncrement="false" Key="true" CaseSensitive="false" MaxLength="256" MinLength="0" Required="false" Scale="0" />
              <Property ID="OU" Type="string" AutoIncrement="false" Key="false" CaseSensitive="false" MaxLength="256" MinLength="0" Required="false" Scale="0" />
              <Property ID="BUKR" Type="int" AutoIncrement="false" Key="false" CaseSensitive="false" MaxLength="256" MinLength="0" Required="false" Scale="0" />
            </ClassType>
            <ClassType ID="AbacusManagementPack.CompanySite" Base="System!System.ConfigItem" Accessibility="Public" Abstract="false" Hosted="false" Singleton="false">
              <Property ID="CompanySiteId" Type="int" AutoIncrement="false" Key="true" CaseSensitive="false" MaxLength="256" MinLength="0" Required="false" Scale="0" />
              <Property ID="Description" Type="string" AutoIncrement="false" Key="false" CaseSensitive="false" MaxLength="256" MinLength="0" Required="false" Scale="0" />
              <Property ID="Lagernummer" Type="int" AutoIncrement="false" Key="false" CaseSensitive="false" MaxLength="256" MinLength="0" Required="false" Scale="0" />
              <Property ID="PostalAddress" Type="string" AutoIncrement="false" Key="false" CaseSensitive="false" MaxLength="256" MinLength="0" Required="false" Scale="0" />
              <Property ID="PostalCode" Type="string" AutoIncrement="false" Key="false" CaseSensitive="false" MaxLength="10" MinLength="0" Required="false" Scale="0" />
              <Property ID="LocalityName" Type="string" AutoIncrement="false" Key="false" CaseSensitive="false" MaxLength="256" MinLength="0" Required="false" Scale="0" />
            </ClassType>
    
            <RelationshipType ID="RelationshipCompanySiteCompany" Accessibility="Public" Abstract="false" Base="System!System.Reference">
              <Source ID="AbacusManagementPack.CompanySite.CompanyId" MinCardinality="0" MaxCardinality="2147483647" Type="AbacusManagementPack.CompanySite" />
              <Target ID="AbacusManagementPack.Company.CompanyId" MinCardinality="0" MaxCardinality="1" Type="AbacusManagementPack.Company" />
            </RelationshipType>
    
            <DisplayString ElementID="RelationshipCompanySiteCompany">
              <Name>CompanyId</Name>
            </DisplayString>
            <DisplayString ElementID="RelationshipCompanySiteCompany" SubElementID="AbacusManagementPack.CompanySite.CompanyId">
              <Name>CompanyId</Name>
            </DisplayString>
            <DisplayString ElementID="RelationshipCompanySiteCompany" SubElementID="AbacusManagementPack.Company.CompanyId">
              <Name>CompanyId</Name>
            </DisplayString>   

    Generated SQL Tables

    Why there is no relationship in SQL?

    Edit: ok, for that part, I got it: Relationships between MP classes are not translated in SQL but are stored in the RelationShip table of ServiceManager as described in this blog:

    https://blogs.technet.microsoft.com/servicemanager/2009/12/31/service-manager-database-tour-useful-queries/

    However, 2 questions remains:

    Here, I explicitly defined CompanyId in CompanySite table: good idea or useless?

    How can I define an unique key (but not the primary key) based on multiple parameters?

    • Edited by EricBDev Friday, October 21, 2016 1:06 PM
    Friday, October 21, 2016 9:22 AM
  • Keep in mind that when working with the SDK, you should ignore the SQL database. The SCSM framework handles all the translation for you. BUT..the SQL database is an excellent reference :)

    Now that you've created SCSM classes (in your MP file), you can get them using the SDK. For example:

    EnterpriseManagementGroup emg = new EnterpriseManagementGroup("myMG");
    ManagementPack AbacusMP = emg.ManagementPacks.GetManagementPack("MyMP.Name","<keytoken or null>", new Version("<MP version>"));
    ManagementPackClass AbacusCompany = mp.GetClass("AbacusManagementPack.Company");

    There are shortcuts to this, of course, but in my opinion it's best to learn the long way around first until you're comfortable with the schema :)


    Friday, October 21, 2016 1:08 PM
  • I see, it is what I feared: working with string, as with WMI stuff, is still far from the convenience and robustness of full object oriented programming where the risk of typing error is minimized and the whole is checked during the compilation.
    Here, it's again something than might do "boom" during running time instead of during "compiling time".
    Coming from stuff like design patterns, best practices, PRISM, etc, I have the impression to go back to the stone age...

    So to minimize these risks, I will create the same classes in C# that I created in XML, to be able to work fully with objects and put my whole validation logic there. I wanted something even more robust with strong data validation on the SQL side, as unique key on more attributes outside primary key but it won't be possible here I think.
    I will translate my OO classed into my MP "xml classes" via the connector one-2-one as late as possible. Not very convenient since I will need to read the full data before updating it. Without the whole SCSM stuff required from my management, I would have work with lazy loading from EntityFramework!

    Going coding now.
    Friday, October 21, 2016 2:53 PM
  • Hey Aaron,

    I always enjoy reading the replies you post as they are always very detailed and well-structured. Please, keep doing this. Thank You! 

    Regards,


    Stoyan (Please take a moment "Vote as Helpful" and/or "Mark as Answer" where applicable. This helps the community, keeps the forums tidy, and recognizes useful contributions. Thanks!)


    Friday, October 21, 2016 3:03 PM
  • Two weeks later, I have now a clear architecture with many projects within my Visual Studio 2015 solution:

    1. Import class library (dll) which deals with the full import of data and the related business logic and transformed to my target data-model (DM) with a bunch of C# classes.
    2. MP is the project to define the ManagementPack, where I also defined my DM within mpx/MP fragments files: same properties of my C# classes defined in XML.
    3. ConnectorBL references Import and transform the DM from C# objects to MP objects.
    4. ConnectorUI references ConnectorBL and define the Xaml file of the connector related to MP.
    5. DAL is only a test project independently from 2-4 directly. It references Import and transform the DM objects to EntityFramework Code-First objects and save the whole to an SQL Server database. I indeed first defined my DM in SQL directly.

    Now, 1., 2. are done. 5. is also done and I can see how my data appear in a SQL database. 4. is ongoing but first I concentrate now on 3.

    As we decided here to go directly with SCSM 2016, I have my whole 1-5 projects targeting .NET 4.5.1 (it would have to be 3.5 for SCSM2012). I am still waiting to get our Windows Server/SQL/SCSM all-2016 machine ready. I managed to get the Microsoft.EnterpriseManagement .NET 4 dllls of SCSM 2016 into my development machine (with Visual Studio 2015) as I described here: https://blogs.technet.microsoft.com/servicemanager/2016/08/03/scsm-2016-upgrade-steps-for-custom-development/ (install of ServiceConsole + manual extract of xxx.Core.dll)

    As said, I am now writing the BusinessLogic for my connector, e.g. implementing 3.
    3. shall be similar to 5., reading existing data, and create new one if not yet present.

    E.g. for a table called ‘source’ having a single field ‘description’ supposed to be unique, I have the following code in 5.:

    foreach (var source in modelContainer.Sources)
    {
        DAL.PO.Source srcDal = null;
        if (sourceExists)
            srcDal = context.Source.SingleOrDefault(s => s.Description == source.Description);
        if (srcDal == null)
        {
            srcDal = new DAL.PO.Source { Description = source.Description };
            srcDal = context.Source.Add(srcDal);
            entitiesAdded++;
        }
        sourceDict.Add(source, srcDal);
    } 
    if (entitiesAdded > 0)
       context.SaveChanges();

    Where:
    •context is my DbContext object. Goal is to create DAL object into it.
    •modelContainer.Sources my list of objects in my data-model
    •sourceDict a dictionary to map DAL objects to DM objects.

    (continue below, had to split my answer, seems this forum does not like multiple code blocks)
    • Edited by EricBDev Friday, November 4, 2016 3:51 PM
    Friday, November 4, 2016 3:44 PM
  • What is the recommended equivalent with SCSM SDK for my transform class in 3.? Is the code below correct and the equivalent of the one above?
    Instead of context, I deal with ManagementPack managementPack and EnterpriseManagementGroup emg objects.

    ManagementPackClass sourceClass = managementPack.GetClass("AbacusManagementPack.Source");     

    foreach (var source in modelContainer.Sources) { var genericCr = new EnterpriseManagementObjectGenericCriteria($"Description = {source.Description}"); var reader = emg.EntityObjects.GetObjectReader<EnterpriseManagementObject>(genericCr, ObjectQueryOptions.Default); EnterpriseManagementObject cemo = null; if (reader != null && reader.Count == 1) { cemo = reader.First(); } else { cemo = new CreatableEnterpriseManagementObject(emg, sourceClass); cemo[sourceClass, "Description"].Value = source.Description; cemo.Commit(); entitiesAdded++; } sourceDict.Add(source, cemo); } if (entitiesAdded > 0) managementPack.AcceptChanges();

    This is for the entities, but what about the relationships? We've seen the relationships between MP entitites are defined in the RelationShip data table, but how to store the content, i.e that entity xy is linked to entity yz?

    EDIT: relationship creation is documented on MSDN

    Thanks!

    Eric




    • Edited by EricBDev Tuesday, November 22, 2016 11:33 AM fix link
    Friday, November 4, 2016 3:47 PM