Migrating to v23.5.x

Algolia B2C cartridge version 23.5.x introduces “v2” indexing jobs with a multitude of improvements over the “v1” jobs.

Overview of the v2 jobs

For an overview of the v2 job system, see Indexing overview.

Before you begin

Backward compatibility

Migrating to the latest cartridge version won’t remove the old v1 jobs from your cartridge, but they are now considered deprecated and will be removed from the cartridge at a later time.

Since the v1 jobs rely on an internal middleware (Stream) to process the records sent by the jobs, these jobs will stop working when the middleware is fully retired. The sunset timelines of the middleware will be communicated shortly with a dedicated deprecation notice.

Algolia highly recommends upgrading to the v2 jobs which instead send records directly to Algolia’s Search API at your earliest convenience due to the many benefits outlined below.

Delta exports

The v2 delta export job, AlgoliaProductDeltaIndex_v2, requires the Delta Export feature and its prerequisite, the Change Log feature, to be enabled on your instance. Create a support ticket with Salesforce to activate the feature.

For more information, see Ongoing indexing.

Estimated upgrade duration

The duration of the upgrade depends on the extent of customization you have applied to the jobs and whether you plan to continue using the v1 jobs.

If you plan to fully transition to the v2 jobs without any extra customizations, the upgrade should take about 30 minutes to complete.

If you have made customizations or extensions to the v1 jobs and want to apply them to the v2 jobs as well, the upgrade duration depends on the extent of these.

Before attempting any such customizations, consult your Solutions Engineer, Customer Success Manager/Engineer, or submit a Support ticket to discuss recommended integration approaches.

Comparison of the jobs

The old jobs (v1)

• Relied on the old B2C job framework (no steptypes.json)
• Were sequential (no parallelization)
• Relied on intermediary XML files to store data temporarily
• Relied on a middleware to split the data into multiple indices by locale
• Delta calculation was slow with large datasets due to calculating the delta by comparing the current full catalog export to a previous one (snapshot) and writing the diff into a file
• Couldn’t be used for multi-instance indexing (where part of the product data is sent from Staging, while another part from Production)
• Use a less error-resilient and scalable end-to-end indexing approach and have limited monitoring and logging capabilities

List of v1 jobs

• AlgoliaCategoriesIndex - for indexing the category tree
• AlgoliaProductsIndex - for indexing product data - both full catalog indexing and product delta updates
• AlgoliaProductsDeltaExport (v23.4.x only) - an earlier, sequential version of the v2 delta job
• AlgoliaProductPricesExport (v23.4.x only) - an earlier, sequential version of the v2 price job
• AlgoliaProductInventoryExport (v23.4.x only) - an earlier, sequential version of the v2 inventory job

The new jobs (v2)

• Rely on the newer B2C job framework (steptypes.json)
• Are chunk-based and parallelized (chunks are executed in parallel). Full (atomic) reindexing provides a zero-downtime full catalog reindexing option, useful for removing stale records
• Use no intermediary XML files. Data is processed in chunks and sent immediately at the end of each chunk
• Send data directly to Algolia
  • Data is split into multiple indices by the B2C job internally
  • There’s less possibility for error due to the end-to-end chain being shorter
• Delta calculation is fast since it uses the platform’s built-in delta export mechanism
  • Temporary data isn’t written to a file before sending
  • B2C delta exports provide a fast and efficient way to determine which products have changed since the last export
• Can be used for multi-instance indexing (part of the product data can be sent from staging, while another part, such as inventory or price data, from production)
  • Partial updates allow for more frequent, lightweight updating of specific attributes rather than the entire record (for example price or inventory data only)
• Text-based logs are more verbose, they cover more edge cases and error messages contain more information
• Have a more extensive job reporting and monitoring system where reports are stored as custom objects instead of an XML file and are cleaned up automatically after a predetermined amount of time

List of v2 jobs

• AlgoliaCategoryIndex_v2 - for indexing the category tree
• AlgoliaProductIndex_v2 - for full catalog indexing (incl. full reindexing)
• AlgoliaProductDeltaIndex_v2 - for product delta updates
• AlgoliaProductInventoryIndex_v2 - for indexing inventory data only
• AlgoliaProductPriceIndex_v2 - for indexing price data only

For more details about v2 jobs, see the Indexing overview page.

Changes from previous versions

The list of base (non-configurable) attributes has changed, as well as the recommended starting list of Additional Product Attributes (configurable), which was previously called “Custom Fields”.

Base attributes

• Changed from: id, primary_category_id, in_stock, price, categories
• To: name, primary_category_id, in_stock, price, categories, image_groups, url

id is always sent (as the objectID field).

Additional Product Attributes recommended starting list

• Changed from: name, short_description, long_description, price, brand, color, size, image_groups
• To: short_description, long_description

You can extend this list with any other attributes you like to index, see Indexing attributes.

Items added or updated in the latest version

Jobs

• AlgoliaCategoryIndex_v2
• AlgoliaProductIndex_v2
• AlgoliaProductDeltaIndex_v2
• AlgoliaProductInventoryIndex_v2
• AlgoliaProductPriceIndex_v2

Services

• algolia.http.search.write - used by the v2 jobs

Site Preferences

• Algolia_AdditionalAttributes (from Algolia_CustomFields)

Deprecated items to be removed

These items can be removed if you don’t plan on using the v1 jobs anymore (see more detailed instructions below).

They are now considered deprecated and will be removed in a future cartridge version with mode="delete".

Jobs

• AlgoliaCategoriesIndex
• AlgoliaProductsIndex
• AlgoliaProductsDeltaExport
• AlgoliaProductPricesExport
• AlgoliaProductInventoryExport

Services

• algolia.http.api (versions before v23.3.0) and related credentials and profile
• algolia.http.export (versions after v23.3.0) and related credentials and profile
• algolia.http.ingestion and related credentials and profile

Site preferences

• Algolia_HostBase
• Algolia_OCAPIClientID
• Algolia_OCAPIClientPassword
• Algolia_CustomFields (renamed to Algolia_AdditionalAttributes). You can remove Algolia_CustomFields if you’re not planning on using the v1 jobs in the future

Migration steps

Take care when upgrading customized files so that you don’t overwrite any customizations.

For any customizations and extensions you’ve made to the cartridge, consider whether they’re still applicable as they may now be supported by the cartridge natively:

• Partial updates (including for price and inventory data updates)
• Multi-instance indexing
• Facet bucketing
• Server-side rendering of category landing page results (first page only, can be extended) for improved SEO

For more details, contact your Solutions Engineer, Customer Success Manager/Engineer, or submit a Support ticket.

To ensure backward compatibility, upgrading the cartridge version won’t remove the old jobs, but new installations will no longer create the old jobs on your instance. They are now considered deprecated and will be removed at a later time with mode="delete" upon import.

v2 jobs have new names and a “_v2” suffix to differentiate them from the old job system, so they won’t overwrite any existing jobs.

Update the code

1. Download cartridge version 23.5.x from Algolia B2C Cartridge GitHub repository.
2. Update the code
   1. If you want to transition to the v2 jobs and don’t plan on using the v1 jobs anymore, remove the previous cartridge version from your repository and add the v2 version to ensure that no outdated files are retained in your repository.
   2. If you made customizations to the v1 jobs and plan to keep using them, selectively update the cartridge files, taking care that any customized files/scripts are not overwritten entirely. You can use GitHub’s release comparison feature to create and apply a diff manually.

Code changes highlights

To help you migrate your customizations, this section highlights the main logic changes between the two cartridges.

Generated data

Before version 23.5.x (v1), jobs built the records data with the AlgoliaProduct model (algoliaProduct.js). For each product, this model generates a JSON object, where each attribute is an object containing values for each enabled site locale. The final object looks like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
{
  "name": {
    "en_US": "Draped Cowl Neck",
    "fr_FR": "Top à col drapé"
  },
  "color": {
    "en_US": "Ivory"
    "fr_FR": "Ivoire",
  },
  "price": {
    "USD": 61.99,
    "EUR": 44.63
  },
  "in_stock": true,
  "primary_category_id": "womens-clothing-tops",
  "categories": [
    [
      {
        "id": "womens-clothing-tops",
        "name": {
          "en_US": "Tops",
          "fr_FR": "Tops"
        }
      },
      {
        "id": "womens-clothing",
        "name": {
          "en_US": "Clothing",
          "fr_FR": "Vêtements"
        }
      },
      {
        "id": "womens",
        "name": {
          "en_US": "Womens",
          "fr_FR": "Femme"
        }
      }
    ]
  ]
}

In version 23.5.x and greater (v2), the cartridge directly builds the final Algolia records. For each product, the job generates one separate AlgoliaLocalizedProduct object (algoliaLocalizedProduct.js) for each desired locale. A typical record from this model looks like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
{
  "name": "Draped Cowl Neck",
  "color": "Ivory",
  "price": {
    "USD": 61.99,
    "EUR": 44.63
  },
  "in_stock": true,
  "primary_category_id": "womens-clothing-tops",
  "categories": [
    [
      {
        "id": "womens-clothing-tops",
        "name": "Tops"
      },
      {
        "id": "womens-clothing",
        "name": "Clothing"
      },
      {
        "id": "womens",
        "name": "Womens"
      }
    ]
  ]
}

Jobs logic

The v1 job code:

1. Generates models with the getNextProductModel() helper
2. Wraps the models into an UpdateProductModel object. This object is sent to the indexing middleware.
3. The middleware splits the provided object to create one final record per locale. This is then sent to the corresponding Algolia index.

In v2, the cartridge doesn’t use the indexing middleware. It’s the job itself that loops on enabled locales and directly creates the final Algolia records. You can see this logic in the algoliaProductIndex.js file.

Thus, one of the main differences between v1 and v2 is the location of the loop handling the locales:

For v1 jobs, the loop is inside the AlgoliaProduct model, when fetching a product attribute value:

1
2
3
4
5
6
7
8
9
  // algoliaProduct.js
  value = {};
  for (var l = 0; l < siteLocalesSize; l += 1) {
      var localeName = siteLocales[l];
      request.setLocale(localeName);
      value[localeName] = aggregatedValueHandlers[attributeName]
          ? aggregatedValueHandlers[attributeName](product)
          : ObjectHelper.getAttributeValue(product, config.attribute);
  }

For v2 jobs, the local handling loop is processed before generating the record:

1
2
3
4
5
6
7
  // algoliaProductIndex.js
  for (var l = 0; l < siteLocales.size(); ++l) {
      var locale = siteLocales[l];
      var indexName = algoliaData.calculateIndexName('products', locale);
      var localizedProduct = new AlgoliaLocalizedProduct({ product: product, locale: locale, attributeList: attributesToSend });
      // ...
  }

Update metadata, jobs, and services

1. Import metadata files from the metadata/algolia/meta folder in the repository: system-objecttype-extensions.xml and custom-objecttype-definitions.xml
2. Import job definitions: metadata/algolia/jobs.xml
3. Import services: metadata/algolia.services.xml

For detailed instructions on importing files in Business Manager, see the Set up the Algolia cartridge page.

Business Manager configuration updates

1. Go to the Merchant Tools > Algolia > Algolia in Business Manager
2. Fill in your Additional Product Attributes according to your use case.
   You can start with short_description, long_description.

Job configuration

1. Go to Administration > Operations > Jobs in Business Manager
2. For each of the v2 jobs, set the execution site by clicking on the job, then go to Job Steps. Click Scope and then select the sites you want to run the jobs for.
3. Click AlgoliaProductDeltaIndex_v2, click Job Steps, then *Job Parameters
4. Click the catalogIDs parameter and add the catalogIDs assigned to your sites.

For more information about setting up the jobs, see Start indexing your data.

If you don’t plan on using the old jobs

If you intend to fully transition to the v2 jobs, you can remove the old jobs and some of the related metadata and configurations.

Jobs

Go to Administration > Operations > Jobs and remove the v1 jobs:

• AlgoliaCategoriesIndex
• AlgoliaProductsIndex

Services

Go to Administration > Operations > Services and remove the following services:

• algolia.http.api and related credentials and profile
• algolia.http.export and related credentials and profile
• algolia.http.ingestion and related credentials and profile

Site Preferences

Go to Administration > Site Development > System Object Types > SitePreferences > Attribute Definitions and filter for preferences starting with “Algolia_” by searching for “Algolia_*”. Remove the following site preferences:

• Algolia_HostBase
• Algolia_OCAPIClientID
• Algolia_OCAPIClientPassword
• Algolia_CustomFields (renamed to Algolia_AdditionalAttributes)
  • you can safely remove Algolia_CustomFields if you’re not planning on using the v1 jobs in the future

Old reporting data

The v2 cartridge version comes with a new job reporting section in the Algolia Business Manager module called v2 Job Reports, but the old reports are still displayed on the page (v1 Job Report (Deprecated)).

To remove the v1 Job Reports section in the Algolia Business Manager module, go to Administration > Site Development > Development Setup > Folder Browser > Impex > src > Algolia and delete all files containing “lastUpdateLog” there.

Confirm the success of the upgrade

Your Application ID, Search API key, and Admin API key should be configured already from your previous installation (go to the Algolia Business Manager module and set them up if not).

After performing the preceding steps to upgrade to the v2 jobs, you should have the five new Algolia jobs in Business Manager under Administration > Operations > Jobs.

Run all the jobs, one after the other, and verify whether they finish with an OK status.

Go to your Algolia dashboard and confirm that your indices were updated.