Optimizing Large-Scale Data Updates in Dynamics 365 with Asynchronous Plugins and Flexible FetchXML
Introduction
In Dynamics 365, it's not uncommon to encounter scenarios where vast quantities of records require updates or complex manipulations. Naively processing large volumes in a single operation can strain performance and lead to timeouts. This article presents a strategy to tackle such challenges, emphasizing asynchronous processing, plugin chaining, and flexible FetchXML-driven initiation.
The Problem
Directly updating thousands (or even hundreds of thousands) of records in Dynamics 365 within a single synchronous operation often leads to:
- Timeouts: Platform limits on execution time can cause failures.
- User Experience Degradation: Long-running synchronous processes can lock up the UI for users.
- Scalability Issues: Performance bottlenecks become evident as data volumes grow.
Solution: Asynchronous Plugin Chaining with Data Passing
Divide and Conquer: Break the large task into smaller, manageable chunks that can be processed independently by separate asynchronous plugins.
Plugin A: The Coordinator
- FetchXML as Input: Design Plugin A to accept a FetchXML query as an input parameter. This empowers users or external systems to precisely define the target record set.
- Paging: Implement paging logic to retrieve records in batches. For super large datasets, consider techniques like cursor-based paging.
- Triggering Plugin B: For each batch, trigger Plugin B asynchronously (via an action or within the same pipeline stage, if feasible) and pass essential data.
Plugin B: The Workhorse
- Receive Data: Plugin B receives the batched data (e.g., Contact IDs and values to update).
- Focused Updates: Perform the necessary updates to the multiple-select column based on the single-choice column's values.
- Robust Error Handling: Log errors gracefully. Potentially retry updates or use customized error reporting mechanisms back to Plugin A.
Why Asynchronous Matters
Registering both plugins in Post-Operation Asynchronous mode offers:
- Parallelism: Dynamics 365 can execute multiple Plugin B instances concurrently, enhancing throughput.
- No UI Blocking: Users continue working while updates happen in the background.
- Scalability: The system handles load better, minimizing bottlenecks.
Testing & Observations
Thorough testing with a representative dataset is crucial. In a real-world example, this approach successfully processed 112,000+ records within 5 minutes. Key observations:
- Performance: Batching and asynchronous execution provide significant speed boosts.
- Flexibility: FetchXML lets processes be initiated with targeted criteria.
Beyond the Basics
- Error Handling: Consider strategies tailored to your requirements (retries, notifying users, custom logging).
- Concurrency: If severe data conflicts are likely, explore locking, but cautiously due to deadlock risks in Dynamics 365.
- Batch Tuning: Experiment with batch sizes to balance throughput and resource usage.
Conclusion
This technique empowers you to handle large-scale Dynamics 365 updates efficiently. It promotes speed, flexibility, and a better user experience.
Code Example (Placeholders)
- [Plugin A Code ]
using Microsoft.Xrm.Sdk;
using Microsoft.Xrm.Sdk.Query;
using Newtonsoft.Json;
using System;
using System.Collections.Generic;
namespace UpdateContactType.Plugin
{
public class ContactFetchAndTrigger : IPlugin
{
public void Execute(IServiceProvider serviceProvider)
{
ITracingService tracingService = (ITracingService)serviceProvider.GetService(typeof(ITracingService));
IPluginExecutionContext context = (IPluginExecutionContext)serviceProvider.GetService(typeof(IPluginExecutionContext));
if (context.MessageName.ToLower() != "new_contactfetchandtrigger")
return;
IOrganizationServiceFactory serviceFactory = (IOrganizationServiceFactory)serviceProvider.GetService(typeof(IOrganizationServiceFactory));
IOrganizationService service = serviceFactory.CreateOrganizationService(context.UserId);
// FetchXML, paging logic (mostly copied from your original code)
string fetchXml = context.InputParameters.Contains("FetchXMLQuery") && context.InputParameters["FetchXMLQuery"] != null
? (string)context.InputParameters["FetchXMLQuery"]
: @"<fetch version='1.0' output-format='xml-platform' mapping='logical' distinct='false'>
<entity name='contact'>
<attribute name='contactid' />
<attribute name='new_contacttype' />
</entity>
</fetch>";
bool moreRecords = true;
int page = 1;
int batchSize = 1000;
int totalCount = 0;
while (moreRecords)
{
string pagedFetchXml = FetchXmlAddPaging(fetchXml, page, batchSize);
EntityCollection results = service.RetrieveMultiple(new FetchExpression(pagedFetchXml));
var dataToPass = new ContactData();
foreach (var contact in results.Entities)
{
dataToPass.ContactIds.Add(contact.Id);
if (contact.Contains("new_contacttype"))
{
dataToPass.ContactValuesToUpdate.Add(contact.Id, (OptionSetValue)contact["new_contacttype"]);
totalCount++; // Increment total count for each record
}
}
TriggerPluginUpdateContactType(context, service, dataToPass);
moreRecords = results.MoreRecords;
page++;
if (moreRecords && results.PagingCookie != null)
{
fetchXml = FetchXmlSetPagingCookie(fetchXml, results.PagingCookie);
}
}
// Trace logs at the end
tracingService.Trace($"Total Contact Records Processed: {totalCount}");
}
private void TriggerPluginUpdateContactType(IPluginExecutionContext context, IOrganizationService service, ContactData data)
{
var actionRequest = new OrganizationRequest("new_contactUpdateContactType");
actionRequest["ContactData"] = JsonConvert.SerializeObject(data);
service.Execute(actionRequest);
/* The sub plugin is running asyncronously, so action cannot capture the response. But still keep below part for potential use in the future.
var response = service.Execute(actionRequest);
// Access OutputParameters and check for keys
if (response.Results.Contains("SuccessCount"))
successCount += (int)response.Results["SuccessCount"];
if (response.Results.Contains("FailureCount"))
failureCount += (int)response.Results["FailureCount"];
*/
}
private string FetchXmlAddPaging(string fetchXml, int page, int count)
{
var doc = new System.Xml.XmlDocument();
doc.LoadXml(fetchXml);
var fetchNode = doc.SelectSingleNode("//fetch");
if (fetchNode.Attributes["page"] == null)
{
var pageAttr = doc.CreateAttribute("page");
pageAttr.Value = page.ToString();
fetchNode.Attributes.Append(pageAttr);
}
else
{
fetchNode.Attributes["page"].Value = page.ToString();
}
if (fetchNode.Attributes["count"] == null)
{
var countAttr = doc.CreateAttribute("count");
countAttr.Value = count.ToString();
fetchNode.Attributes.Append(countAttr);
}
else
{
fetchNode.Attributes["count"].Value = count.ToString();
}
return doc.OuterXml;
}
private string FetchXmlSetPagingCookie(string fetchXml, string pagingCookie)
{
// Directly use the paging cookie without additional escaping
var doc = new System.Xml.XmlDocument();
doc.LoadXml(fetchXml);
var fetchNode = doc.SelectSingleNode("//fetch");
if (fetchNode.Attributes["paging-cookie"] != null)
{
fetchNode.Attributes["paging-cookie"].Value = pagingCookie;
}
else
{
var cookieAttr = doc.CreateAttribute("paging-cookie");
cookieAttr.Value = pagingCookie;
fetchNode.Attributes.Append(cookieAttr);
}
return doc.OuterXml;
}
}
public class ContactData
{
public List<Guid> ContactIds { get; set; } = new List<Guid>();
public Dictionary<Guid, OptionSetValue> ContactValuesToUpdate { get; set; } = new Dictionary<Guid, OptionSetValue>();
}
}
- [Plugin B Code]
using Microsoft.Xrm.Sdk;
using System;
using Newtonsoft.Json;
namespace UpdateContactType.Plugin
{
public class UpdateContactType : IPlugin
{
public void Execute(IServiceProvider serviceProvider)
{
ITracingService tracingService = (ITracingService)serviceProvider.GetService(typeof(ITracingService));
IPluginExecutionContext context = (IPluginExecutionContext)serviceProvider.GetService(typeof(IPluginExecutionContext));
if (context.MessageName.ToLower() != "new_contactupdatecontacttype")
return;
IOrganizationServiceFactory serviceFactory = (IOrganizationServiceFactory)serviceProvider.GetService(typeof(IOrganizationServiceFactory));
IOrganizationService service = serviceFactory.CreateOrganizationService(context.UserId);
var serializedData = context.InputParameters.Contains("ContactData") ? context.InputParameters["ContactData"] as string : null;
if (serializedData == null)
{
tracingService.Trace("Error: ContactData not found in input parameters.");
return;
}
var data = JsonConvert.DeserializeObject<ContactData>(serializedData);
int successCount = 0;
int failureCount = 0;
foreach (var contactId in data.ContactIds)
{
var contactToUpdate = new Entity("contact", contactId);
if (data.ContactValuesToUpdate.ContainsKey(contactId))
{
try
{
contactToUpdate["new_contacttypemulti"] = new OptionSetValueCollection(new[] { data.ContactValuesToUpdate[contactId] });
service.Update(contactToUpdate);
successCount++;
}
catch (Exception ex)
{
failureCount++;
tracingService.Trace($"Error updating contact {contactId}: {ex.Message}");
}
}
}
// Set Output Parameters
context.OutputParameters["SuccessCount"] = successCount;
context.OutputParameters["FailureCount"] = failureCount;
}
}
}
No comments:
Post a Comment