Fine grained partial failure in the BulkMutateJobService

Since its introduction the BulkMutateJobService has provided support for partial failures, allowing some operations to succeed even though others have failed. Over the next few weeks we'll be rolling out improvements to this functionality, allowing for more granular failures when possible.

Bulk mutate jobs process operations in batches, and a failed operation will cause the entire batch to fail, but not the rest of the job. The results for a failed batch start with a FailureResult that holds all the errors generated by the operations in that batch. It is followed by a series of BatchFailureResults which act as placeholders for the rest of the operations in the batch. More information about how these failures are reported can be found in a previous blog post we did on the topic.

In v201008 we enabled partial failure support for the synchronous AdGroupCriterionService. Unlike the behavior in the BulkMutateJobService, this version of partial failure allows individual operations to fail without affecting successful ones. This method of partial failure is preferred as it cuts down on the number of retries needed.

While we can't update the BulkMutateJobService to process all operations this way, we are able to enable it for operations that use an underlying service that does support granular partial failures. Starting August 1st, operations of type AdGroupCriterionOperation and AdGroupAdOperation processed by the BulkMutateJobService may start failing granularly instead of in batches.

Let's see how this will change the results returned.

Existing partial failure behavior

Operation	Result
AdGroupCriterionOperation	FailureResult cause.errors[0].fieldPath = "operations[1]..." cause.errors[1].fieldPath = "operations[2]..."
AdGroupCriterionOperation (Invalid)	BatchFailureResult operationIndexInBatch = 1
AdGroupCriterionOperation (Invalid)	BatchFailureResult operationIndexInBatch = 2
AdGroupCriterionOperation	BatchFailureResult operationIndexInBatch = 3

Before the change the invalid operations will cause other valid operations to fail. The FailureResult is the first result in the batch, even if the operation at the position was valid. The field path of the errors indicates the index of the invalid operations relative to the start of the batch.

New granular partial failure behavior

Operation	Result
AdGroupCriterionOperation	ReturnValueResult
AdGroupCriterionOperation (Invalid)	FailureResult cause.errors[0].fieldPath = "operations[0]..."
AdGroupCriterionOperation (Invalid)	FailureResult cause.errors[0].fieldPath = "operations[0]..."
AdGroupCriterionOperation	ReturnValueResult

After the change invalid operations will not affect other valid operations in the same batch. Each invalid operation will have a corresponding FailureResult. There may still be cases where BatchFailureResults are returned for these operations, such as if there is a general error with the account that cannot be attributed to any specific operation.

So how does this affect code you've already written to process BulkMutateJob results? The good news is that if you've written your code to allow for a variable batch size (no hard coded batch size) then no changes are needed! Invalid operations are essentially processed as batches of one, and the same logic that you use to process the results today should work perfectly with these new granular failures. If your application is currently expecting a batch of a specific size then you should update it to detect the start and end of batches dynamically.

If you have any questions about how these new granular failures work, or feedback on how we can further improve the error handling in the API, add a post to the AdWords API forum.

Update 2011-08-25
This change has been fully deployed and all BulkMutateJobs will be processed with granular partial failures enabled.

- Eric Koleda, AdWords API Team

Discover v201008: Remove incomplete BulkMutateJobs

The v201008 release of the AdWords API added a new feature to the BulkMutateJobService: the ability to remove incomplete jobs. This allows you to clean out your job queue after an application crash or other failure. In this post we’ll demonstrate how to use the feature and discuss some of its limitations.

When a BulkMutateJob is created the field numRequestParts is set, which determines how many BulkMutateRequest objects will be added to the job. Once the final request part has been added, the job enters into the account’s job queue, and when it reaches the front of the queue it begins processing. If the application crashes while the parts are being uploaded, the job will never start and remain stranded in the account.

Jobs that are incomplete and those that are complete but waiting in the queue behind a processing job will both have a PENDING status. An account can only have 10 pending jobs at a time, so stranded jobs can begin to fill up the account, until eventually no new jobs can be added. Prior to v201008 the only way to remove stranded jobs was to add the missing request parts so that the job would start. If the job had been stranded for a while though, the operations that were previously uploaded may make undesired changes to the account.

In v201008 the REMOVE operator can be used to delete incomplete jobs. The following code does this using the PHP client library.

$bulkMutateJobId = (float) 'INSERT_BULK_MUTATE_JOB_ID_HERE';

// Create BulkMutateJob.
$bulkMutateJob = new BulkMutateJob();
$bulkMutateJob->id = $bulkMutateJobId;

// Create operation.
$operation = new JobOperation();
$operation->operand = $bulkMutateJob;
$operation->operator = 'REMOVE';

// Delete bulk mutate job.
$bulkMutateJob = $bulkMutateJobService->mutate($operation);

After being removed, the job will be in the FAILED state and the failureReason field will contain JobError.USER_CANCELED_JOB.

A limitation of the REMOVE operator is that it only works on jobs that are in the PENDING state and that are incomplete (numRequestParts > numRequestPartsReceived). Jobs that are complete and have already entered the job queue, whether they be in the PENDING or PROCESSING state, cannot be deleted.

Code examples that show how to delete jobs are included in all of the client libraries. If you have any questions about this functionality or the BulkMutateJobService, ask us on the forum.

- Eric Koleda, AdWords API Team

Discover v2009: Error handling in BulkMutateJobService.

In our previous blog post, we discussed how v2009 error handling works in synchronous APIs. In this blog post, we cover asynchronous error handling in BulkMutateJobService. If you are new to BulkMutateJobService, check out our introductory blog post and code.google.com article.

Due to its asynchronous nature, error handling in BulkMutateJobService is slightly different than in other services. Instead of throwing an ApiError object, BulkMutateJobService returns FailureResult or BatchFailureResult objects as its operation results. These objects in turn contain the relevant ApiErrors. BulkMutateJobService processes the operations within an operation stream in multiple atomic batches. If there is even one faulty operation in a batch, it causes the other non-faulty operations in the batch to fail as well. When a batch of n operations fails, you will get one FailureResult and n - 1 BatchFailureResults as results for the failed batch of operations. The FailureResult object will always be first in the batch, and will contain one ApiException object. The ApiException contains all the errors for the entire batch of operations, just like for other services. This will be followed by n - 1 BatchFailureResults, which can be viewed as placeholders, one for each operation in the batch.

For instance, assume that you are trying to create the following 9 text ads in a single OperationStream. For the purposes of this example, assume that the system processes operations in batches of 3. The errors are highlighted in red.

Index	Ad	Destination Url	Error
0	New York Budget Hotel Clean and close to subway. Students save 20%! www.example.com/NewYork	http://www.example.com/NewYork
1	Book New York Hotel Deals Research your New York Stay. Deals up to 70% from example.com. www.example.com/NewYork	http://www.example.com/NewYork
2	Hotels In New York Book your hotel room online at a lower price via example.com!! www.example.com/NewYork	http://www.example.com/NewYork	Excessive punctuation in description2
3	Manhattan Best Hotel Experience the Best of Manhattan For Rates As Low As $239/Night. www.example.com/NewYork	www.example.com/NewYork	Missing schema in destination url.
4	New York City Hotels Discount Discounted Rooms in New York City. Get Lowest Rates Available Today. www.example.com/NewYork	http://www.example.com/NewYork
5	New York City Hotels Stay Smart in New York NY. Free internet & breakfast bar. www.example.com/NewYork	http://www.example.com/NewYork
6	70% Off New York Hostels Why Pay More? From $10 We Guarantee Lowest Prices - From 1 Night www.example.com/NewYork	http://www.example.com/NewYork
7	Stylish New Hotel in NYC Enjoy Four-Star Amenities & Service Chelsea/Fashion District. Book Now! www.example.com/NewYork	http://www.example.com/NewYork
8	Budget Lodging in NYC Comfortable, Clean Modern Economy Hotel.Great Downtown Location! www.example.com/NewYork	http://www.example.com/NewYork

When BulkMutateJobService processes these operations, it will return the following results:

Result[0]: FailureResult.
 ApiException.errors has one ApiError
   PolicyViolationError @ operations[2].operand.ad.description2
Result[1]: BatchFailureResult, operationIndexInBatch = 1
Result[2]: BatchFailureResult, operationIndexInBatch = 2

Result[3]: FailureResult.
 ApiException.errors have 1 ApiError
   AdError.URL_NO_SCHEME @ operations[0].operand.ad.url
Result[4]: BatchFailureResult, operationIndexInBatch = 1
Result[5]: BatchFailureResult, operationIndexInBatch = 2

Result[6]: ReturnValueResult, successfully created Ad.
Result[7]: ReturnValueResult, successfully created Ad.
Result[8]: ReturnValueResult, successfully created Ad.

As shown in the results above, there is a 1:1 correspondence between the number of operations and number of results. You can get the successful operations by scanning the return value for results which are not FailureResult or BatchFailureResult (operations 6, 7 and 8 in this case). Then, scan through the results to locate FailureResult objects (which always appear at the beginning of the failed batch) to get the operation that actually failed (operations 2 and 3 in this case). You can now pick the failed operations that do not correspond to the ones obtained from FailureResult, and make a new job for them (0, 1, 4, 5). Note that the OGNL path in FailureResult is relative to the batch and not the operationstream, so you have to offset it by the index of FailureResult to get the failed operation (e.g. Results[3] is a FailureResult and its OGNL path is operations[0].operand.ad.url, so the failing operator in the original list is 3 + 0 = operations[3]).

A BulkMutateJob may also return the following error results, even though they're not common:

LostResult: This happens when BMJS has lost results for some of a job's operations. You can recover from this error by checking numFailedOperations and numUnprocessedOperations for the job. If both are zero, or if the number of errors in ApiError matches numFailedOperations, then the operations were successful, but the results are missing. In any other case, you have to retrieve the entities that were targeted by the operations and see if the operation actually succeeded or not.

UnprocessedResult: This indicates that the mutate operation was not processed. This could happen because the workflow implementation prevented the operation from being processed (e.g. the operations exceed API quota). You can recover from this issue by checking the job status, making sure that you have sufficient API quota left, and simply retrying the operations in a new job.

In an upcoming blog post, we will cover the most common errors you may get in v2009, and how to address them. We've included support for error handling in all of our client libraries to help get you started, so please try it out and share your feedback with us on the forum or the projects' issue trackers.

-- Anash P. Oommen, AdWords API Team

Discover v2009: Asynchronous requests with the BulkMutateJobService

Delegation of tasks: we use this concept in our everyday lives when we pay someone to make us coffee, ask a colleague to take care of something while we run to the dentist, or tell our kids or younger siblings to handle a chore. What's common in all these situations? Well, you assign someone a task, possibly consisting of several parts, and then you wait for it to be completed. You can focus on other things while the task is being worked on, but you can still periodically ask what the status is (not too often, or they'll think you're annoying). Eventually they'll tell you it's done, and let you know whether it was successful or not. It's a simple process, and we intuitively know what to expect from it.

Well, you can now delegate complex tasks via the AdWords API! v2009 introduced a unified interface across all services for performing changes: the mutate method. This unification allowed us to develop the BulkMutateJobService, where you can provide a list of mutate operations to be performed for a number of different services, by scheduling a job which gets executed asynchronously. Much like the v13 ReportService, you can then poll for the job status, and once you get back the signal that it's been completed, you can ask for the results.

Let's dive into the details. BulkMutateJobs consist of the following:

• BulkMutateRequests: these are the different parts of the job. The first one you submit should include a count of how many parts your job has; you submit one at a time, and once the last one is submitted, the job automatically starts. This helps build very large jobs, by avoiding the sending of a single, gigantic request.
• OperationStreams: a BulkMutateRequest contains one or more OperationStreams, which can be executed in parallel (only in certain conditions, namely if their scopes target different campaigns).
• Operations: each OperationStream contains a list of Operations, which are performed sequentially.

So to create a job with two parts, you would simply do:

BulkMutateJob job = new BulkMutateJob();
job.setNumRequestParts(2);

and fill in the details for the first part:

BulkMutateRequest part1 = new BulkMutateRequest();
part1.setPartIndex(0);

Then create an operation stream, consisting of several operations (they can even use different operators) and use it in the part:

OperationStream adStream = new OperationStream();
adStream.setScopingEntityId(scopingEntityId);
adStream.setOperations(new Operation[] {
new AdGroupAdOperation(Operator.ADD, null, adGroupAd1, null),
new AdGroupAdOperation(Operator.ADD, null, adGroupAd2, null)});
part1.setOperationStreams(new OperationStream[] {adStream});

Once we're done with that, we can create the JobOperation and submit the first part to the service:

JobOperation jobOperation = new JobOperation();
jobOperation.setOperand(job);
jobOperation.setOperator(Operator.ADD);
job = bulkMutateJobService.mutate(jobOperation);
Long jobId = job.getId();

Submitting the second part is similar, but now we must use the SET operator, since we're adding to an existing job, not creating a new one. Also, we must provide the job ID, so that the system knows which job we're submitting a new part for.

BulkMutateRequest part2 = new BulkMutateRequest();
part2.setPartIndex(1);
// ... OperationStream code goes here ...
job = new BulkMutateJob();
job.setId(jobId); // Here we set the ID to the one we got from
// the response to the first part
job.setRequest(part2);
job = bulkMutateJobService.mutate(new JobOperation(Operator.SET, null, job));

Once all the parts are submitted, the system automatically starts the job. Querying the status is easy, just perform a get with the job ID:

BulkMutateJobSelector selector = new BulkMutateJobSelector();
selector.setJobIds(new long[] {jobId});
BulkMutateJob[] jobs = bulkMutateJobService.get(selector);
job = jobs[0];

Don't query too often, though, or you may run trigger a rate-limiting protection! We recommend 30 seconds as a good value to wait between queries. Once the response comes back with a COMPLETED status, we can retrieve the results for both parts, by setting the resultPartIndex:

selector.setResultPartIndex(0);
jobs = bulkMutateJobService.get(selector);
job = jobs[0];
OperationResult[] operationResults1 =
job.getResult().getOperationStreamResults()[0].getOperationResults();

selector.setResultPartIndex(1);
jobs = bulkMutateJobService.get(selector);
job = jobs[0];
OperationResult[] operationResults2 =
job.getResult().getOperationStreamResults()[0].getOperationResults();

The BulkMutateResult object in the result field of the response includes a list of OperationStreamResults, which themselves are an array of OperationResults. Going through this hierarchy allows you to see which operations were successful and which ones failed, and act accordingly.

Note: If you're looking for a more complete example of using the BulkMutateJobService, please check the examples/demo folder in the client library of your choice. Also check Performing Bulk Jobs with BulkMutateJobService in the API documentation.

With the BulkMutateJobService you can provide as much work as possible to the API in one go and let it do its processing, without worrying about timeouts or broken connections. It won't affect your error handling, either, since the responses detail exactly what happened to all of the operations you performed. It's just an efficient way of getting a lot of work done in a single task. Come in and have a seat, the AdWords API is ready to take your order!

--Sérgio Gomes, AdWords API Team

This is the first of a series of posts on the exciting features of AdWords API v2009. In next week's edition of Discover v2009, we'll tell you all about getting ideas from the new TargetingIdeaService.