Oracle Service Cloud Bulk Data Import – Best Practices

This blog is part of the series of blogs the A-Team has been running on Oracle Service Cloud (OSvC).

In the previous blog I went through an introduction to the Bulk APIs in OSvC and briefly touched upon throughput considerations(If you haven’t, please read that first). In this blog I build on the previous post and delve deeper into various operational factors to consider when implementing bulk data load into OSvC.

For the purpose of this blog, I define bulk import(or bulk load) as scenarios where records of the order of millions need to be imported into Service Cloud. For smaller scale data load OSvC has an import utility, described here.

Following is an overview of this blog post:

1. 1.   I first discuss the pros and cons of bulk-loading data into OSvC.
2. 2.   Then I present a high-level architecture of any bulk-import implementation.
3. 3.   After that I present the results of a series of performance tests that demonstrate how well the OSvC APIs scale.
4. 4.   Finally, based on the performance tests and other findings I provide a few recommendations that will ensure high throughput during the data import.

1. Should I Bulk-Load ?

The first rule of OSvC bulk data load is, don’t unless you absolutely have to. In an OSvC implementation an initial bulk-load may seem like an obvious requirement, but it’s a lot of stress on the Service Cloud database and other resources (especially considering that OSvC is multi-tenant), and it can potentially degrade the overall performance. Bulk import may be avoided by designing use-cases so that OSvC data is created on demand, only when needed.

However, sometimes it may be imperative to perform an initial bulk-load. For example, any lead time needed to fetch and create data on-demand may be undesirable, especially when the Service Agent is on call with the customer.

For the rest of this blog it is assumed that bulk-import is indeed a key requirement.

2. High Level Architecture

Any bulk-load implementation mainly requires two things: An API that allows bulk-import, and an Integration layer where data is prepped and massaged before importing.

API

OSvC’s Connect Web Services API enables bulk data import via the batch operation(explained in the previous blog), which lets multiple heterogenous operations and objects be batched into one request.

Integration Layer

The Integration layer is used to validate, massage, transform and orchestrate data before importing into Service Cloud. It is depicted below:

The integration layer can be SOA Cloud Service, SOA on-premise or any other integration tool. For this blog I assume an on-premise SOA environment.

3. Performance Testing

Given the architecture above, for high-throughput data import it is important that the integration layer as well as the Batch API performs well. Tuning the integration layer is specific to the product chosen, and there may be plenty of ways to do that.

However, there isn’t much documentation available on the CWSS APIs’ performance under stress. So, I conducted a performance-testing exercise to evaluate how well the CWSS APIs (batch operation) scale under various conditions. I will now present the results of those tests.

Test Conditions

Following the integration framework above, I used a two-node Oracle SOA 12.1.3 Cluster on a VM with 24-core 3GHz processor and 140GB RAM.
After that I developed a BPEL process that acted as the bulk-import process. The SOA Composite is shown below:

The first BPELProcess1 takes in a parameter called ‘NumOfThreads’ and creates that many BPELProces2 instances in a loop, without waiting for a response. The BPELProcess2 in turn invokes the OSvC Batch API. This is how we simulated ‘n’ concurrent threads. We can also simulate this by using an inbound DB Adapter’s NumberOfThreads parameter.

Varying Batch Sizes and record sizes were simulated using XSLTs that generated an OSvC Batch request payload of the desired size, commit points etc.

The following four areas were tested:

1. 1)How well the API scales with increasing number of client threads.
2. 2)How much overhead is added with custom OSvC attributes.
3. 3)Effect of varying batch sizes.
4. 4)Performance over a period of time.

Test Results

The graphs below depict the result. The values on the y-axis are depicted relatively (x, 10x, 20x, etc). Please note that these graphs and any numbers in them are only indicative of the observed trend, and no other conclusion should be derived from these.

Increasing Number of Threads

Implication:

• Increasing the number of concurrent request threads increases the overall throughput

Standard vs Custom Attributes

25 OSvC custom attributes (for the Contact object) were created to test the custom attributes scenario. Also, the batch size was set to 100.

Implication:

• Batch API throughput(Contacts created per second) reduces with increasing number of attributes

Batch Size Analysis

Implication:

• Throughput increases with increasing batch sizes. Batch size of 1 is almost always a BAD idea. Also, even though the results seem to be best with batch size 100, other test/environment conditions might result in a different number being optimal.

Long Running Tests

Implication:

• Contacts were created at a fairly constant rate over a period of time (1 hour)

4. Operational Recommendations

I will now list a few recommendations which will ensure that the data import process is consistently high-performing without affecting overall Service Cloud performance. These recommendations are based on the performance tests above and other factors.

Record Size

In Service Cloud custom attributes and custom fields can be defined for standard data objects. Each record to be inserted can contain standard as well as custom fields’ data, thus increasing the record size. As we saw, larger record sizes lead to higher API response times.
This should be kept in mind when designing the Service Cloud data model such that we don’t create unnecessary custom attributes. Also, if possible, the initial data load can be done with minimum required data so that auxiliary data can be added later.

Batching Records

A key component of the CWSS Batch operation is the batch size, i.e. the number of records being sent in the Batch request, to be committed in a single transaction. As the results aboved showed, committing one record per transaction is a BAD idea. It adds a lot of unnecessary and repetitive I/O on the database. Instead, larger batch sizes should be used. Some testing may be necessary to find the ideal batch size.

Parallel API Invocations

As the tests demonstrated, CWSS scales well with multiple parallel invocations. This should be leveraged when designing any bulk-load interface. The degree of parallelism will depend on your actual use-case, how well the client scales, and other factors.
At the same time, this does not imply we create large number of concurrent client threads and pump as much data as possible. As the number of concurrent API requests increase, more and more Service Cloud database threads get busy with data insertion, and that may interfere with the Agent (Customer Service Reps using the Agent Desktop) experience.
Hence, the number of concurrent client threads inserting data should be configurable, so that we can increase the number when agent activity is less and vice-versa. For example, concurrent requests can be kept low during the Service Cloud Agents’ working hours, and it can be increased when the data load runs overnight.

Disabling Triggers within Service Cloud

OSvC allows Custom Process Models (also called Custom Processes or Events) and Business Rules to be configured for a given object. Following is a brief description:
Custom Process Models (CPMs): CPMs , also known as Custom Processes are used to execute custom logic after an object is created/modified/deleted. They can be attached to any standard or custom object, and they are written in PHP.
CPMs can be synchronous or asynchronous. Synchronous CPMs, as the name suggests, are executed immediately, i.e. if an Update Contact API is invoked and a corresponding Sync CPM exists, then the API won’t return until the CPM is executed. Asynchronous CPMs’ execution is decoupled from the API execution, so the API response time is slightly better.
Business Rules: Business Rules offer the ability to automate tasks such as incident or queue assignments, escalations etc. From a technical perspective these rules are executed in the same transaction as the API execution, hence directly add to the API response time.
For bulk-load scenarios Rules and CPMs may not be needed, and suppressing them significantly improves the overall performance. This can be done by setting Processing Options appropriately in the input payload.

Cleansing the Data Before Insertion

When CWSS insert is invoked, OSvC performs a number of validations on the data, such as if the email address is valid (using regex), uniqueness constraints, upper and lower limits of the data, etc. OSvC rejects the payload whenever such a validation fails. In a batch request payload, even a small percentage of data errors can lead to the entire batch being rejected. This can potentially bring the bulk-load interface to a crawl.
Validation errors would be minimal if we could sanitize the data with OSvC validations before trying to insert. This can be done by using the getMetadata API. This API exposes all the metadata associated with standard and custom objects such as data types, nullability, uniqueness, regex patterns, etc. The metadata is returned in the form of a ‘MetaDataClass’ for each object.
Below is a sample response payload from getMetadata :

<n0:MetaDataClass>
	<n1:Attributes>
	   <n1:MetaDataAttributeList>
		  <n1:DataType>STRING</n1:DataType>
		  <n1:IsDeprecated>false</n1:IsDeprecated>
		  <n1:Description>Authentication user name</n1:Description>
		  <n1:MaxLength>80</n1:MaxLength>
		  <n1:Name>Login</n1:Name>
		  <n1:Nullable>true</n1:Nullable>
		  <n1:Pattern>[^ \t\n<>"]*</n1:Pattern>
		  <n1:UsageOnCreate>ALLOWED</n1:UsageOnCreate>
		  <n1:UsageOnDestroy>IGNORED</n1:UsageOnDestroy>
		  <n1:UsageOnGet>ALLOWED</n1:UsageOnGet>
		  <n1:UsageOnUpdate>ALLOWED</n1:UsageOnUpdate>
		  <n1:UsedAsName>false</n1:UsedAsName>
	   </n1:MetaDataAttributeList>
	   <n1:MetaDataAttributeList>
		  <n1:DataType>STRING</n1:DataType>
		  <n1:IsDeprecated>false</n1:IsDeprecated>
		  <n1:Description>Social or professional title (e.g. Mrs. or Dr.)</n1:Description>
		  <n1:MaxLength>80</n1:MaxLength>
		  <n1:Name>Title</n1:Name>
		  <n1:Nullable>true</n1:Nullable>
		  <n1:Pattern>[^\n]*</n1:Pattern>
		  <n1:UsageOnCreate>ALLOWED</n1:UsageOnCreate>
		  <n1:UsageOnDestroy>IGNORED</n1:UsageOnDestroy>
		  <n1:UsageOnGet>ALLOWED</n1:UsageOnGet>
		  <n1:UsageOnUpdate>ALLOWED</n1:UsageOnUpdate>
		  <n1:UsedAsName>false</n1:UsedAsName>
	   </n1:MetaDataAttributeList>
	</n1:Attributes>
	<n1:CanCreate>true</n1:CanCreate>
	<n1:CanDestroy>true</n1:CanDestroy>
	<n1:CanGet>true</n1:CanGet>
	<n1:CanUpdate>true</n1:CanUpdate>
	<n1:DerivedFrom>
	   <n2:Namespace>urn:base.ws.rightnow.com/v1_2</n2:Namespace>
	   <n2:TypeName>RNObject</n2:TypeName>
	</n1:DerivedFrom>
	<n1:MetaDataLink>Contact</n1:MetaDataLink>
	<n1:Name>
	   <n2:Namespace>urn:objects.ws.rightnow.com/v1_2</n2:Namespace>
	   <n2:TypeName>Contact</n2:TypeName>
	</n1:Name>
</n0:MetaDataClass>

For example, the response payload above suggests that the ‘Login’ field in the Contact object is nullable, and has a maximum length of 80. Knowing this, we can ensure that the bulk-load process rejects data where Login field has more than 80 characters.
In order to use this API, we can either manually analyze the getMetadata response and keep the validation checks updated, or we can create a process that periodically fetches these validations rules and applies them on the data before inserting.
There are four flavors of this API, GetMetadata, GetMetadataForClass, GetMetadataForOperation, and GetMetadataSinceLastDateTime. They are described here

Network Latency

There is network latency in CWSS payload travelling from the integration layer to OSvC and back. Not only is latency dependant on the distance the network packets travel, another key factors is whether the packets travel over the public internet or VPN.
Thorough testing and network analysis may be needed to ensure network latency isn’t high. Also, it makes sense to keep the integration layer physically close to OSvC.

Conclusion

In this blog I discussed whether bulk-import is always necessary. I also demonstrated that Service Cloud APIs perform well under stress conditions. Finally, I discussed a few recommendations when running bulk import into Service Cloud.
Following those recommendations will go a long way in ensuring the data import runs smoothly.

All content listed on this page is the property of Oracle Corp. Redistribution not allowed without written permission