Index pages with the Java Google Indexing API
The Google Indexing API allows any site owner to directly notify Google when pages are added or removed. This allows Google to schedule pages for a fresh crawl, which can lead to higher quality user traffic. For websites with many short-lived pages like job postings or livestream videos, the Indexing API keeps content fresh in search results because it allows updates to be pushed individually.
Currently, the Indexing API can only be used to crawl pages with either "JobPosting" or "BroadcastEvent embedded in a VideoObject".
Here are some of the things you can do with the Google Indexing API:
Supported | Description |
Update a URL | Notify Google of a new URL to crawl or that content at a previously-submitted URL has been updated. |
Remove a URL | After you delete a page from your servers, notify Google so that we can remove the page from our index and so that we don't attempt to crawl the URL again. |
Get the status of a request | Check the last time Google received each kind of notification for a given URL. |
Send batch indexing requests | Instead of submitting index requests for individual URL, you can combine up to 100 URLs to notify Google in one request. |
1. Requirements
First you need to register and create a project on Google Cloud Console.
Enable the "Google Indexing" service for your account.
Create a credential of type "Service Account" and download it as a JSON file.
After the above step, you are provided with an email in the format "xxx@yyy.iam.gserviceaccount.com".
No ADS
On Google Search Console, add the email you got in the step above as owner for your website.
2. Library
<!-- SEE ALL VERSIONS -->
<!-- https://libraries.io/maven/com.google.apis:google-api-services-indexing/versions -->
<dependency>
<groupId>com.google.apis</groupId>
<artifactId>google-api-services-indexing</artifactId>
<version>v3-rev20230927-2.0.0</version>
</dependency>
<!-- https://mvnrepository.com/artifact/com.google.auth/google-auth-library-oauth2-http -->
<dependency>
<groupId>com.google.auth</groupId>
<artifactId>google-auth-library-oauth2-http</artifactId>
<version>1.20.0</version>
</dependency>
<!-- https://mvnrepository.com/artifact/com.google.api-client/google-api-client-gson -->
<dependency>
<groupId>com.google.api-client</groupId>
<artifactId>google-api-client-gson</artifactId>
<version>2.2.0</version>
</dependency>
<!-- https://mvnrepository.com/artifact/commons-io/commons-io -->
<dependency>
<groupId>commons-io</groupId>
<artifactId>commons-io</artifactId>
<version>2.15.1</version>
</dependency>
3. HttpRequestInitializer
To send requests to Google services via the Google Java API, you need to create an HttpRequestInitializer object.
HttpRequestInitializer will initialize the necessary information included in requests each time they are sent. The most important of which is credentials, to ensure that you have access to relevant services and resources.
No ADS
In the previous step you created a credential and downloaded it as a JSON file. Now we will write a utility class to create an HttpRequestInitializer object.
MyUtils.java
package org.o7planning.googleapis.utils;
import java.io.ByteArrayInputStream;
import java.io.File;
import java.io.IOException;
import java.io.InputStream;
import org.apache.commons.io.FileUtils;
import com.google.api.client.http.HttpRequest;
import com.google.api.client.http.HttpRequestInitializer;
import com.google.api.client.json.JsonObjectParser;
import com.google.api.client.json.gson.GsonFactory;
import com.google.api.client.util.ObjectParser;
import com.google.auth.http.HttpCredentialsAdapter;
import com.google.auth.oauth2.GoogleCredentials;
import com.google.auth.oauth2.ServiceAccountCredentials;
public class MyUtils {
public static final String SERVICE_ACCOUNT_FILE_PATH = "/Volumes/New/_gsc/test-google-search-console-key.json";
private static byte[] serviceAccountBytes;
public static HttpRequestInitializer createHttpRequestInitializer(String... scopes) throws IOException {
InputStream serviceAccountInputStream = getServiceAccountInputStream();
GoogleCredentials credentials = ServiceAccountCredentials //
.fromStream(serviceAccountInputStream) //
.createScoped(scopes);
HttpRequestInitializer requestInitializer = new HttpRequestInitializer() {
@Override
public void initialize(HttpRequest request) throws IOException {
HttpCredentialsAdapter adapter = new HttpCredentialsAdapter(credentials);
adapter.initialize(request);
//
if (request.getParser() == null) {
ObjectParser parser = new JsonObjectParser(GsonFactory.getDefaultInstance());
request.setParser(parser);
}
//
request.setConnectTimeout(60000); // 1 minute connect timeout
request.setReadTimeout(60000); // 1 minute read timeout
}
};
return requestInitializer;
}
public static synchronized InputStream getServiceAccountInputStream() throws IOException {
if (serviceAccountBytes == null) {
serviceAccountBytes = FileUtils.readFileToByteArray(new File(SERVICE_ACCOUNT_FILE_PATH));
}
return new ByteArrayInputStream(serviceAccountBytes);
}
}
You can also create GoogleCredentials with "API Key" or "OAuth Client ID",...
- Java Google APIs commons Credentials
4. Notify to update a URL
The Google Indexing API allows you to submit indexing requests for certain URL on your website. Note that this request will be put into a queue, it will not be executed immediately no matter how many such requests you send. The time you last requested to update a URL will be saved by Google.
Each website will have a daily quota to send requests to Google. If the quota is exceeded, the request will be denied. Quotas may vary for different websites, depending on the quality of your website, specifically user traffic.
No ADS
A utility class for sending an indexing request to Google.
NotifyToUpdateUrlUtils.java
package org.o7planning.java_14285_google_apis;
import org.o7planning.googleapis.utils.MyUtils;
import com.google.api.client.googleapis.javanet.GoogleNetHttpTransport;
import com.google.api.client.http.HttpRequestInitializer;
import com.google.api.client.http.HttpTransport;
import com.google.api.client.json.gson.GsonFactory;
import com.google.api.services.indexing.v3.Indexing;
import com.google.api.services.indexing.v3.Indexing.UrlNotifications;
import com.google.api.services.indexing.v3.IndexingScopes;
import com.google.api.services.indexing.v3.model.PublishUrlNotificationResponse;
import com.google.api.services.indexing.v3.model.UrlNotification;
import com.google.api.services.indexing.v3.model.UrlNotificationMetadata;
public class NotifyToUpdateUrlUtils {
private static final String MY_APPLICATION = "My-Application";
/**
* <pre>
* {
* url: "http://foo.com/page",
* latestUpdate: {
* url: "http://foo.com/page",
* type: "URL_UPDATED",
* notifyTime: "2017-09-31T19:30:54.524457662Z"
* },
* latestRemove: {
* url: "http://foo.com/page",
* type: "URL_DELETED",
* notifyTime: "2017-08-31T19:30:54.524457662Z"
* }
* }
* </pre>
*/
public static UrlNotificationMetadata requestToUpdatePageUrl(String applicationName, String pageUrl)
throws Exception {
HttpRequestInitializer httpRequestInitializer //
= MyUtils.createHttpRequestInitializer(IndexingScopes.INDEXING);
//
HttpTransport httpTransport = GoogleNetHttpTransport.newTrustedTransport();
//
Indexing indexing = new Indexing.Builder(httpTransport, //
GsonFactory.getDefaultInstance(), //
httpRequestInitializer) //
.setApplicationName(applicationName) //
.build();
UrlNotification content = new UrlNotification();
content.setType("URL_UPDATED");
content.setUrl(pageUrl);
//
UrlNotifications.Publish publish = indexing.urlNotifications().publish(content);
PublishUrlNotificationResponse respone = publish.execute();
UrlNotificationMetadata urlNotification = respone.getUrlNotificationMetadata();
return urlNotification;
}
public static UrlNotificationMetadata updatePageUrlIndexing(String pageUrl) throws Exception {
return requestToUpdatePageUrl(MY_APPLICATION, pageUrl);
}
public static void main(String[] args) throws Exception {
// Test:
UrlNotificationMetadata metadata = updatePageUrlIndexing(MyTestConstants.URL_TO_TEST);
System.out.println("Return: " + metadata.toPrettyString());
}
}
Output:
{
url: "http://foo.com/page",
latestUpdate: {
url: "http://foo.com/page",
type: "URL_UPDATED",
notifyTime: "2017-09-31T19:30:54.524457662Z"
},
latestRemove: {
url: "http://foo.com/page",
type: "URL_DELETED",
notifyTime: "2017-08-31T19:30:54.524457662Z"
}
}
5. Notify to remove a URL
Sometimes there are some URLs on your website that are no longer needed and you want to remove them from Google Search's index. You can send a deletion request to Google, and this URL must return a 404 error code.
No ADS
NotifyToDeleteUrlUtils.java
package org.o7planning.java_14285_google_apis;
import org.o7planning.googleapis.utils.MyUtils;
import com.google.api.client.googleapis.javanet.GoogleNetHttpTransport;
import com.google.api.client.http.HttpRequestInitializer;
import com.google.api.client.http.HttpTransport;
import com.google.api.client.json.gson.GsonFactory;
import com.google.api.services.indexing.v3.Indexing;
import com.google.api.services.indexing.v3.Indexing.UrlNotifications;
import com.google.api.services.indexing.v3.IndexingScopes;
import com.google.api.services.indexing.v3.model.PublishUrlNotificationResponse;
import com.google.api.services.indexing.v3.model.UrlNotification;
import com.google.api.services.indexing.v3.model.UrlNotificationMetadata;
public class NotifyToDeleteUrlUtils {
private static final String MY_APPLICATION = "My-Application";
/**
* <pre>
* {
* "urlNotificationMetadata": {
* "url": "http://foo.com/path",
* "latestUpdate": {
* "url": "http://foo.com/path",
* "type": "URL_UPDATED",
* "notifyTime": "2023-11-28T12:49:16.868776828Z"
* },
* "latestRemove": {
* "url": "http://foo.com/path",
* "type": "URL_DELETED",
* "notifyTime": "2023-11-28T13:50:29.662337626Z"
* }
* }
* }
* </pre>
*/
public static PublishUrlNotificationResponse requestToDeletePageUrl(String applicationName, String pageUrl)
throws Exception {
HttpRequestInitializer httpRequestInitializer //
= MyUtils.createHttpRequestInitializer(IndexingScopes.INDEXING);
//
HttpTransport httpTransport = GoogleNetHttpTransport.newTrustedTransport();
//
Indexing indexing = new Indexing.Builder(httpTransport, //
GsonFactory.getDefaultInstance(), //
httpRequestInitializer) //
.setApplicationName(applicationName) //
.build();
UrlNotification content = new UrlNotification();
content.setType("URL_DELETED");
content.setUrl(pageUrl);
//
UrlNotifications.Publish publish = indexing.urlNotifications().publish(content);
PublishUrlNotificationResponse respone = publish.execute();
return respone;
}
public static PublishUrlNotificationResponse deletePageUrlIndexing(String pageUrl) throws Exception {
return requestToDeletePageUrl(MY_APPLICATION, pageUrl);
}
public static void main(String[] args) throws Exception {
// Test:
PublishUrlNotificationResponse response = deletePageUrlIndexing(MyTestConstants.URL_TO_TEST);
System.out.println("Delete Response: " + response.toPrettyString());
UrlNotificationMetadata meta = response.getUrlNotificationMetadata();
System.out.println("\nMetadata: " + meta.toPrettyString());
}
}
Output:
{
"urlNotificationMetadata": {
"url": "http://foo.com/path",
"latestUpdate": {
"url": "http://foo.com/path",
"type": "URL_UPDATED",
"notifyTime": "2023-11-28T12:49:16.868776828Z"
},
"latestRemove": {
"url": "http://foo.com/path",
"type": "URL_DELETED",
"notifyTime": "2023-11-28T13:50:29.662337626Z"
}
}
}
6. Get information of a URL
The Google Indexing API allows you to see the last time you submitted a request to index or remove a specific URL. This information does not include the actual status of the URL (updated or removed).
If you want to know the real status of a URL, continue with the article below:
No ADS
GetUrlNotificationMetaDataUtils.java
package org.o7planning.java_14285_google_apis;
import org.o7planning.googleapis.utils.MyUtils;
import com.google.api.client.googleapis.javanet.GoogleNetHttpTransport;
import com.google.api.client.http.HttpRequestInitializer;
import com.google.api.client.http.HttpTransport;
import com.google.api.client.json.gson.GsonFactory;
import com.google.api.services.indexing.v3.Indexing;
import com.google.api.services.indexing.v3.IndexingScopes;
import com.google.api.services.indexing.v3.model.UrlNotificationMetadata;
public class GetUrlNotificationMetaDataUtils {
private static final String MY_APPLICATION = "My-Application";
/**
* <pre>
* {
* url: "http://foo.com/page",
* latestUpdate: {
* url: "http://foo.com/page",
* type: "URL_UPDATED",
* notifyTime: "2017-07-31T19:30:54.524457662Z"
* },
* latestRemove: {
* url: "http://foo.com/page",
* type: "URL_DELETED",
* notifyTime: "2017-08-31T19:30:54.524457662Z"
* }
* }
* </pre>
*/
public static UrlNotificationMetadata getUrlNotifications(String applicationName, String pageUrl) throws Exception {
HttpRequestInitializer httpRequestInitializer //
= MyUtils.createHttpRequestInitializer(IndexingScopes.INDEXING);
HttpTransport httpTransport = GoogleNetHttpTransport.newTrustedTransport();
//
Indexing indexing = new Indexing.Builder(httpTransport, //
GsonFactory.getDefaultInstance(), //
httpRequestInitializer) //
.setApplicationName(applicationName) //
.build();
Indexing.UrlNotifications.GetMetadata getMetadataRequest = indexing.urlNotifications().getMetadata();
getMetadataRequest.setUrl(pageUrl);
UrlNotificationMetadata metadata = getMetadataRequest.execute();
return metadata;
}
public static UrlNotificationMetadata getUrlNotifications(String pageUrl) throws Exception {
return getUrlNotifications(MY_APPLICATION, pageUrl);
}
public static void main(String[] args) throws Exception {
// Test:
UrlNotificationMetadata infos = getUrlNotifications(MyTestConstants.URL_TO_TEST);
System.out.println("Return: " + infos.toPrettyString());
}
}
Output:
{
url: "http://foo.com/page",
latestUpdate: {
url: "http://foo.com/page",
type: "URL_UPDATED",
notifyTime: "2017-09-31T19:30:54.524457662Z"
},
latestRemove: {
url: "http://foo.com/page",
type: "URL_DELETED",
notifyTime: "2017-08-31T19:30:54.524457662Z"
}
}
7. Request URL batch indexing
Instead of submitting index requests for individual URLs, you can combine up to 100 URLs to notify Google in one request.
BatchUpdateUrlUtils.java
package org.o7planning.java_14285_google_apis;
import java.io.IOException;
import org.o7planning.googleapis.utils.MyUtils;
import com.google.api.client.googleapis.batch.BatchRequest;
import com.google.api.client.googleapis.batch.json.JsonBatchCallback;
import com.google.api.client.googleapis.javanet.GoogleNetHttpTransport;
import com.google.api.client.googleapis.json.GoogleJsonError;
import com.google.api.client.http.HttpHeaders;
import com.google.api.client.http.HttpRequestInitializer;
import com.google.api.client.http.HttpTransport;
import com.google.api.client.json.gson.GsonFactory;
import com.google.api.services.indexing.v3.Indexing;
import com.google.api.services.indexing.v3.IndexingScopes;
import com.google.api.services.indexing.v3.model.PublishUrlNotificationResponse;
import com.google.api.services.indexing.v3.model.UrlNotification;
public class BatchUpdateUrlUtils {
private static final String MY_APPLICATION = "My-Application";
public static void executeBatchIndexing(String applicationName, String... pageUrls) throws Exception {
HttpRequestInitializer httpRequestInitializer //
= MyUtils.createHttpRequestInitializer(IndexingScopes.INDEXING);
//
HttpTransport httpTransport = GoogleNetHttpTransport.newTrustedTransport();
Indexing indexing = new Indexing.Builder(httpTransport, //
GsonFactory.getDefaultInstance(), //
httpRequestInitializer) //
.setApplicationName(applicationName) //
.build();
//
JsonBatchCallback<PublishUrlNotificationResponse> callback = new JsonBatchCallback<PublishUrlNotificationResponse>() {
@Override
public void onSuccess(PublishUrlNotificationResponse res, HttpHeaders responseHeaders) {
try {
System.out.println(res.toPrettyString());
} catch (IOException e) {
e.printStackTrace();
}
}
@Override
public void onFailure(GoogleJsonError e, HttpHeaders responseHeaders) throws IOException {
System.out.println("Error Message: " + e.getMessage());
}
};
BatchRequest batch = indexing.batch();
for (String url : pageUrls) {
UrlNotification unf = new UrlNotification();
unf.setUrl(url);
unf.setType("URL_UPDATED");
indexing.urlNotifications().publish(unf).queue(batch, callback);
}
batch.execute();
}
public static void updatePageUrlIndexing(String... pageUrls) throws Exception {
executeBatchIndexing(MY_APPLICATION, pageUrls);
}
public static void main(String[] args) throws Exception {
// Test:
updatePageUrlIndexing(MyTestConstants.URL_TO_TEST, MyTestConstants.URL2_TO_TEST);
}
}
Output:
{
"urlNotificationMetadata": {
"latestRemove": {
"notifyTime": "2023-12-01T14:37:38.721117189Z",
"type": "URL_DELETED",
"url": "http://foo.com/page"
},
"latestUpdate": {
"notifyTime": "2023-12-02T12:43:43.074657919Z",
"type": "URL_UPDATED",
"url": "http://foo.com/page"
},
"url": "http://foo.com/page"
}
}
{
"urlNotificationMetadata": {
"latestUpdate": {
"notifyTime": "2023-12-02T12:43:43.074745791Z",
"type": "URL_UPDATED",
"url": "http://foo.com/page2"
},
"url": "http://foo.com/page2"
}
}
8. Error handling
An error may arise when you send a request to Google. For example, when you have exceeded the allowed quota, or submitted a request to index a URL of a website that is not yours, etc.
{
"code": 403,
"errors": [
{
"domain": "global",
"message": "Permission denied. Failed to verify the URL ownership.",
"reason": "forbidden"
}
],
"message": "Permission denied. Failed to verify the URL ownership.",
"status": "PERMISSION_DENIED"
}
In case of error, the Google Indexing API will throw an HttpResponseException.
HandleError.java
package org.o7planning.java_14285_google_apis;
import java.util.List;
import com.google.api.client.googleapis.json.GoogleJsonError;
import com.google.api.client.googleapis.json.GoogleJsonError.ErrorInfo;
import com.google.api.client.googleapis.json.GoogleJsonResponseException;
import com.google.api.client.http.HttpResponseException;
import com.google.api.services.indexing.v3.model.UrlNotificationMetadata;
public class HandleError {
public static void main(String[] args) throws Exception {
// Test:
try {
UrlNotificationMetadata infos = GetUrlNotificationMetaDataUtils
.getUrlNotifications("https://notmywebsite.com");
System.out.println("Return: " + infos.toPrettyString());
} catch (Exception e) {
if (e instanceof HttpResponseException) {
HttpResponseException ex = (HttpResponseException) e;
int statusCode = ex.getStatusCode();
String content = ex.getContent();
System.err.println("statusCode: " + statusCode);
System.err.println("content: " + content);
if (e instanceof GoogleJsonResponseException) {
GoogleJsonResponseException ex2 = (GoogleJsonResponseException) e;
GoogleJsonError detailError = ex2.getDetails();
//
List<ErrorInfo> errorInfos = detailError.getErrors();
for (ErrorInfo einfo : errorInfos) {
System.out.println("Error: domain: " + einfo.getDomain());
System.out.println("Error: message: " + einfo.getMessage());
System.out.println("Error: reason: " + einfo.getReason());
}
}
} else {
// Other Error
System.out.println("Error: " + e.getMessage());
e.printStackTrace();
}
}
}
}
Output:
statusCode: 403
content: {
"code": 403,
"errors": [
{
"domain": "global",
"message": "Permission denied. Failed to verify the URL ownership.",
"reason": "forbidden"
}
],
"message": "Permission denied. Failed to verify the URL ownership.",
"status": "PERMISSION_DENIED"
}
Error: domain: global
Error: message: Permission denied. Failed to verify the URL ownership.
Error: reason: forbidden
No ADS
- Creating a Google API Console project and OAuth2 Client ID
- Register Google Map API Key
- Get Started with the Google Search Console API
- Index pages with the Java Google Indexing API
- Use Google reCAPTCHA in Java Web Application
- Manipulating files and folders on Google Drive using Java
- Create a Google Service Account
- List, add and delete Sites with Google Search Java API
- List, submit and delete Sitemaps with Google Search Java API
Show More