Sunday, August 25, 2019

Overview of the Commerce 9.1 SIF Installation Process

The Sitecore Commerce installation really puts SIF through its paces, and a number of things can go wrong along the way. Having a clear sense of what the script is doing, how it is structured, what the bumps are, and how to pick thing up after an error can make a big difference, so I thought I'd share my experiences. These notes are based on the Commerce 9.1 installation, though I touch on what has changed since 9.0. I hope to look at the recently released 9.2 installation process in a future post.

Commerce SIF Structure and Function

The Commerce download contains a number of sub zip directories:



These zips fall into three categories: classic packages to be installed on Sitecore, external site roots for the commerce engine and the Business Tools (a.k.a. "bizfx") and "SIF.Sitecore.Commerce.x.x.x.zip", the SIF installation resources required for the install.  The rest of this post reviews the contents of that folder, analyzes the process and shares a few things that can go wrong and how to resolve them.

Note: The 9.0.x installation process also contained a zip of an early version of the Sitecore Identity server, but since 9.1 that is now part of the main installation.

Another note: SIF is tricky but very powerful.  It is pretty much the de facto documentation of the installation process, where settings live, what cert thumbprint needs to be set where. You need to learn to read it. I found this video series (eight short 2 to 3 minute videos) extremely helpful.

Inside the SIF folder

There are three levels of PowerShell scripts in the SIF folder.   The Deploy-Sitecore-Commerce.ps1 file in the top level:
This is used to configure parameters (SQL Sever connection, Sitecore password, URL names, etc.) and contains almost no logic. (There's a if/then for whether you are using Solr or Azure search. That's it.)  It in turn calls Configuration/Master_SingleServer.json which does the actual orchestration of the installation, visible in the "Tasks" section:




Here's a list of the actual tasks, along with a brief description of what they do:
  • SetIdentityServerCommerceConfiguration 
    • Set up the identity server to expect to support authentication from the commerce engine and bizfx
  • InstallSolrCores
    • Setup Solr cores required by the commerce engine
  • DeployCommerceEngine
    • Creates four instances of the Commerce engine site (for authoring, shops, minions, and DevOps)
  • DeploySitecoreBizFx
    • Deploys the authoring "Business Tools" from end for the Commerce Engine.
  • PreconfigureStorefrontInstance
    • This one does a lot: creates a signed cert for the storefront web site, changes the binding to that (removing the existing one), creates a host file entry, and disables indexing.
  • InstallPowershellExtensions
  • InstallSXAFrameworkModule
  • PublishExtensions
  • InstallHabitatImagesModule
  • InstallAdventureWorksImagesModule
  • InstallCommerceConnectModule
    • Install Commerce Connect, generalized connection layer to wire Sitecore front end and xDB to a third-party commerce provider
  • InstallCommercexProfilesModule
    • Extensions to the Experience Profile
  • InstallCommercexAnalyticsModule
    • Extensions to Experience Analytics
  • CopyConnectModels
    • Deploy model configuration to xConnect
  • InstallCommerceMAModule
    • Marketing automation configuration
  • InstallCommerceEngineConnectModule
    • Customizations to Commerce Connect for Sitecore Commerce
  • InstallSXAStorefrontModule
  • InitializeCommerce
    • Configures Sitecore XP to point to the Commerce Engine
  • InitializeCommerceEngine
    • This "bootstraps" the commerce engine, which causes a number of JSON files to be loaded to the commerce "Global" database.
  • InitializeCommerceEngineUsingHost
    • Same as above, but using a full URL, not a localhost port.
  • EnableCEConnectDataProvider
    • Enables the Sitecore config file for the Commerce data provider.
  • GenerateCatalogTemplates
    • Run Sitecore Commerce Engine Connection functionality to dynamically generate templates. 
  • CreateDefaultTenantAndSite
    • Create the "Sitecore" SXA tenant and the "Storefront" site, and associate the sample Habitat catalog to this site.  This is driven by powershell configuration items loaded with  the SXAStorefrontModule packages.
  • PublishCommerce
  • PostconfigureStorefrontInstance
    • Reenables search
  • EnableCEConnectIndexing
    • Switches on some configs
  • Reindex
  • RemoveSiteUtilityFolder
    • Removes a folder with ASPX utility files for publishing, loading packages, and indexing.

Small Powershell side note: I got that list with these command:

$config = (Invoke-ReadJsonConfigFunction   "SIF.Sitecore.Commerce.2.0.19\Configuration\Commerce\Master_SingleServer.json" )
$config.Tasks.GetMembers() |% {"<li>$($_.Name)</li>"} | clip


Summarizing, the installation configures a new cert bound site binding, creates the engine and business tool websites, installs packages to Sitecore, loads commerce engine configuration,  creates the default tenant and site, reindexes and does some cleanup.

Tracking Down What's Happening

I mentioned above that the script logic has three layers. First, in Deploy-Sitecore-Commerce.ps1, there is the invocation of  the command "Install-SitecoreCommerce" with a whole bunch of parameters, including "Path", which points to "Master_SingleServer.json".  Next, Master_SingleServer.json has tasks for another layer of SIF files. For example, the first task, SetIdentityServerCommerceConfiguration, has this definition in Master_SingleServer.json:
 "SetIdentityServerCommerceConfiguration": {
      "Type": "InstallSitecoreConfiguration",
      "Params": {
        "Path": "[concat(parameter('BaseConfigurationFolder'), '\\Commerce\\IdentityServer\\sitecore-identity-config.json')]",
        "CommerceInstallRoot": "[parameter('CommerceInstallRoot')]",
        "SitecoreIdentityServerApplicationName": "[parameter('SitecoreIdentityServerApplicationName')]",
        "CommerceServicesHostPostfix": "[parameter('CommerceServicesHostPostfix')]"
      }
    }

Almost all the Master-SingleServer tasks are of type "InstallSitecoreConfiguration", which calls the script in the "Path" parameter, passing along the other parameters to "sitecore-identity-config.json".  Going to this file show the following tasks:

    "UpdateIdentityServerCommerceConfiguration": {
      "Skip": "[not(parameter('CommerceServicesHostPostfix'))]",
      "Type": "SetXml",
      "Params": {
        "FilePath": ".\\IdentityServer\\Sitecore.Commerce.IdentityServer.Host.xml",
        "XPath": "//Settings/Sitecore/IdentityServer/Clients/CommerceClient/AllowedCorsOrigins",
        "Element": "AllowedCorsOriginsGroup2",
        "Value": "[variable('Group2')]"
      }
    },
    "CopyIdentityServerCommerceConfiguration": {
      "Type": "Copy",
      "Params": {
        "Source": ".\\IdentityServer\\Sitecore.Commerce.IdentityServer.Host.xml",
        "Destination": "[joinpath(parameter('CommerceInstallRoot'),  concat(parameter('SitecoreIdentityServerApplicationName'), '\\Config\\production\\Sitecore.Commerce.IdentityServer.Host.xml'))]"
      }

Now, note that these tasks do not have "Type" set to "InstallSitecoreConfiguration", so they won't be hitting yet another json SIF file.  The meaning of "SetXml" is pretty clear, but what does it actually do? Well, if you have SIF installed and you run:

PS C:\WINDOWS\system32> Get-SitecoreInstallExtension |? {$_.Name -eq "SetXml"}

Name   Type Command           Module
----   ---- -------           ------
SetXml Task Invoke-SetXmlTask SitecoreInstallFramework

You can see the name in the "Type" parameter maps to a PowerShell command, "Invoke-SetXmlTask". (This "Invoke-<name>Task" is followed for all Sitecore written SIF commands, but a few mappings of standard PowerShell functions do not follow this convention. Run an unfiltered "Get-SitecoreInstallExtension" to see the complete list of mappings.)

So how can we see the implementation of Invoke-SetXmlTask? One way is to run 

>Get-Command Invoke-SetXmlTask -ShowCommandInfo 

which "decompiles" the command. (You can even run this on "Import-SitecoreConfiguration" to see exactly how SIF itself is implemented. Blog it for extra credit.)  Here's a bit of the implementation of Invoke-SetXmlTask:


                    function HasMember($x, $Name) {
                        return $x.psobject.Members.Where({($_.MemberType -eq 'Property') -and ($_.Name -eq $Name) })
                    }

                    [xml]$xml = Get-Content $FilePath
                    WriteTaskInfo -Tag Update -MessageData "$FilePath"

                    if($found = Select-XML -Xml $xml -XPath $XPath) {
                        $found.ForEach({
                            $contextNode = $_.Node;
                            if($PSCmdlet.ParameterSetName -eq 'Add'){
                                $ns = $xml.DocumentElement.NamespaceURI
                                $newNode = $xml.CreateElement($Element, $ns)
                                $newNode.InnerText = $Value
                                if(($contextNode.ChildNodes.Count -eq 1) -and ($contextNode.ChildNodes[0].Name -eq
                '#text')) {
                                    $contextNode.InnerText = ''
                                }

This works for the main SIF defined commands, but the commerce specific scripts make use of commands defined in PowerShell modules. For example, the script CommerceEngine.Deploy.json registers the following modules:

  "Modules": [
    "ManageCommerceService",
    "DeployCommerceDatabase",
    "DeployCommerceContent",
    "WindowsLocal",
    "SitecoreUtilityTasks"
  ]

So the task DeployCommerceDatabase:

  "DeployCommerceDatabase": {
      "Type": "DeployCommerceDatabase",
      "Params": {
        "CommerceServicesDbServer": "[parameter('CommerceServicesDbServer')]",
        "CommerceServicesDbName": "[parameter('CommerceServicesDbName')]",
        "CommerceServicesGlobalDbName": "[parameter('CommerceServicesGlobalDbName')]",
        "CommerceEngineDacPac": "[parameter('CommerceEngineDacPac')]",
        "UserName": "[concat(parameter('UserDomain'), concat('\\', parameter('UserName')))]"
      }
    }

makes use of a type, "DeployCommerceDatabase", that is not part of OOTB SIF, but is instead defined in DeployCommerceDatabase.pms1, which contains the actual implementation:

   #Drop the CommerceServices databases if they exist
    Write-Host "Deleting existing CommerceServices databases...";
    Add-SQLPSSnapin;
    DropSQLDatabase $CommerceServicesDbName
    DropSQLDatabase $CommerceServicesGlobalDbName

    Write-Host "Creating CommerceServices databases...";
    $connectionString = "Server=" + $CommerceServicesDbServer + ";Trusted_Connection=Yes;"

    #deploy using the dacpac
    try {
        deploydacpac $CommerceEngineDacPac $connectionString $CommerceServicesGlobalDbName
        deploydacpac $CommerceEngineDacPac $connectionString $CommerceServicesDbName
        write-host "adding roles to commerceservices databases...";
        AddSqlUserToRole $CommerceServicesDbServer $CommerceServicesGlobalDbName $UserName "db_owner"
        AddSqlUserToRole $CommerceServicesDbServer $CommerceServicesDbName $UserName "db_owner"
    }
    catch {
        Write-Host $_.Exception
        Write-Error $_ -ErrorAction Continue
        $dacpacError = $TRUE
    }
}

So you can always see exactly what PowerShell commands are getting used to implement each step in the process. This can come in mighty handy when you need to do something like replace a certificate, and you want to see exactly where it is used. You can always use the SIF scripts as documentation on how to build a working installation.

Starting in the Middle

Something will go wrong. You can read from the SIF output exactly which task failed, you can configure Deploy-Sitecore-Commerce.ps1 to run from a certain point by adding a "From" parameter, and giving it the name of the first task in Master_SingleServer.ps1 that you wish to run:

From = "DeployCommerceEngine"

Usually, you will not want to run the whole subtask again, so you can define where you want the subtask to start by adding a "From" parameter to the definition in Master_SingleServer.json:

"From": "PreconfigureCommerceEngineOpsInstance"

In short, you need to modify the top level Deploy-Sitecore-Commerce.ps1, and the second level Master_SingleServer.json.  You need to read the third level configuration file to get the names of its tasks, but in my experience you don't need to modify the third level files.

Sometimes you just want to run a single task. That is easily done. In Deploy-Sitecore-Commerce.ps1, instead of a "From" parameter, use Tasks, and set it to an array:
Tasks = ("DeployComerceEngine")

You do the same thing at the next level, but with JSON instead of PowerShell syntax:

"Tasks": ["PreconfigureCommerceEngineOpsInstance"]

Of course, you can list several tasks in the Tasks array. You can also use "To" if you know where you want your script to stop.

A Few Pain Points

1.  The DeployCommerceDatabases fails if SQL is not locally installed, so if using this script to deploy to a machine without SQL server, you will need to bypass this step using the "To" and "From" parameters described above. The error message, "SQL Server Provider for Windows PowerShell is not installed" is a little misleading. The SQL installation is pretty straightforward to do by hand in SSMS using the packaged DACPAC files. This post describes how to do that.

2. The DeployCommerceDatabases also attempts to clear the existing databases in the default SQL Server instance on the machine. If you have two instances running, this might fail. Again, easy to do by hand.

3. The installation of the commerce engines depends on the names of sites to drive logic:

{($_ -match "CommerceShops") -or ($_ -match "CommerceAuthoring") -or ($_ -match "CommerceMinions")}

So if you rename CommerceAuthoring to something else, say "XCAuthors", then the installation will be bypassed. (This should be an additional type parameter, so that implementers have full control over the site names. Of course you can modify the PowerShell module code, but I don't think the name of a site should be driving implementation logic.)

4. If an invalid cert gets stored in the Trusted Root Store of your machine, certificate validation can get broken.  I moved the wrong cert to the wrong spot while troubleshooting, and I went half mad trying to figure out what was going on, until a more cert-savvy colleague dug me out of my jam. (Thank you, Sean Dailey!)

5. A lot happens with the stock tenant install. If you later on try to create your own catalog instead of the out of the box "Habitat_Master", you will need to configure template overrides (defined in fields on the catalog itself) or each product and category will throw a null reference exception. How to do this is documented hereAgain, looking at the powershell to figure out what is going is often key to troubleshooting. Here, it was the Sitecore PowerShell extensions scripts in /sitecore/system/modules that contained the secret.  (I'll come back to that in a later post.)

6.  The scripts install SXA but don't add the SXA indexes, so you will have to do that at the end. Which I tried to do half way through the script process, only to find that indexing was shut off, which led to one of my sillier Sitecore Support tickets. I closed it once I found:

"DisableIndexUpdate": {
      // Speed up deployment
      "Type": "SetXml",
      "Params": [
        {
          "FilePath": "[variable('ContentSearchConfig'))]",
          "Xpath": "[variable('ContentSearchEnabledXpath')]",
          "Attributes": {
            "value": "false"
          }
        }
      ]

7. There are a few utility pages that get used during the installation. One of them, InstallModules.aspx, contains this line:

        <%@ Import Namespace="log4net" %>
        [...]
        var log = LogManager.GetLogger("LogFileAppender");

Unfortunately, log4net.LogManager exists in both the standard log4net DLL and Sitecore.Diagnostics, so if you use this script in a solution that has log4net in the bin directory, the compiler won't know which one to use. I wasn't able to find a way to scope references in an ASCX. and the first time I encountered this, I actually created a DLL that referenced the Sitecore DLL and added my own namespace to solve the problem. I may have written some powershell steps to automate changing the line of code, but I won't admit that here.  Because when I took a second look at this a few moths later, I noted that "log" variable is never used! So adding  "// " before the "var log = ..."  took care of the issue.  "Slash slash win" has become a minor personal mantra for looking carefully at a problem, and then doing the absolute minimum (economy of motion) to solve it.

Okay, that's my brain dump on this. Having SIFting!

No comments:

Post a Comment