List, submit and delete Sitemaps with Google Search Java API
1. Requirements
The requirements for working with Google Search Java API are mentioned in the article below. This article also covers different ways to create an HttpRequestInitializer object (With API Key, OAuth Client Id or Service Account).
No ADS
A utility class to create an HttpRequestInitializer object, which provides credentials for requests each time they are sent to Google:
MyUtils.java
package org.o7planning.googleapis.utils;
import java.io.ByteArrayInputStream;
import java.io.File;
import java.io.IOException;
import java.io.InputStream;
import org.apache.commons.io.FileUtils;
import com.google.api.client.http.HttpRequest;
import com.google.api.client.http.HttpRequestInitializer;
import com.google.api.client.json.JsonObjectParser;
import com.google.api.client.json.gson.GsonFactory;
import com.google.api.client.util.ObjectParser;
import com.google.auth.http.HttpCredentialsAdapter;
import com.google.auth.oauth2.GoogleCredentials;
import com.google.auth.oauth2.ServiceAccountCredentials;
public class MyUtils {
public static final String SERVICE_ACCOUNT_FILE_PATH = "/Volumes/New/_gsc/test-google-search-console-key.json";
private static byte[] serviceAccountBytes;
public static HttpRequestInitializer createHttpRequestInitializer(String... scopes) throws IOException {
InputStream serviceAccountInputStream = getServiceAccountInputStream();
GoogleCredentials credentials = ServiceAccountCredentials //
.fromStream(serviceAccountInputStream) //
.createScoped(scopes);
HttpRequestInitializer requestInitializer = new HttpRequestInitializer() {
@Override
public void initialize(HttpRequest request) throws IOException {
HttpCredentialsAdapter adapter = new HttpCredentialsAdapter(credentials);
adapter.initialize(request);
//
if (request.getParser() == null) {
ObjectParser parser = new JsonObjectParser(GsonFactory.getDefaultInstance());
request.setParser(parser);
}
//
request.setConnectTimeout(60000); // 1 minute connect timeout
request.setReadTimeout(60000); // 1 minute read timeout
}
};
return requestInitializer;
}
public static synchronized InputStream getServiceAccountInputStream() throws IOException {
if (serviceAccountBytes == null) {
serviceAccountBytes = FileUtils.readFileToByteArray(new File(SERVICE_ACCOUNT_FILE_PATH));
}
return new ByteArrayInputStream(serviceAccountBytes);
}
}
A utility class for creating a SearchConsole.Sitemaps object, an important object that is related to all the examples in this article.
MySitemapsUtils.java
package org.o7planning.googleapis.a14297.sitemap;
import java.io.IOException;
import java.security.GeneralSecurityException;
import org.o7planning.googleapis.utils.MyUtils;
import com.google.api.client.googleapis.javanet.GoogleNetHttpTransport;
import com.google.api.client.http.HttpRequestInitializer;
import com.google.api.client.http.HttpTransport;
import com.google.api.client.json.gson.GsonFactory;
import com.google.api.services.searchconsole.v1.SearchConsole;
import com.google.api.services.searchconsole.v1.SearchConsoleScopes;
public class MySitemapsUtils {
private static final String APPLICATION_NAME = "My-Application";
public static SearchConsole.Sitemaps getSitemapsClient(String applicationName) //
throws IOException, GeneralSecurityException {
//
// @scopes: WEBMASTERS or WEBMASTERS_READONLY
//
HttpRequestInitializer httpRequestInitializer = MyUtils
.createHttpRequestInitializer(SearchConsoleScopes.WEBMASTERS);
//
HttpTransport httpTransport = GoogleNetHttpTransport.newTrustedTransport();
//
SearchConsole client = new SearchConsole.Builder( //
httpTransport, GsonFactory.getDefaultInstance(), httpRequestInitializer) //
.setApplicationName(applicationName) //
.build();
SearchConsole.Sitemaps sitemaps = client.sitemaps();
return sitemaps;
}
public static SearchConsole.Sitemaps getSitemapsClient() //
throws IOException, GeneralSecurityException {
return getSitemapsClient(APPLICATION_NAME);
}
}
2. List Sitemaps
With the Google Search API, you can list sitemap files that have been submitted to Google Search Console. By default they include the sitemap files you declared in the "robots.txt" file.
Take a look at Apple's "robots.txt" file:
No ADS
ListSitemapExample.java
package org.o7planning.googleapis.a14297.sitemap;
import java.io.IOException;
import java.security.GeneralSecurityException;
import java.util.List;
import org.o7planning.googleapis.MyTestConstants;
import com.google.api.services.searchconsole.v1.SearchConsole;
import com.google.api.services.searchconsole.v1.model.SitemapsListResponse;
import com.google.api.services.searchconsole.v1.model.WmxSitemap;
import com.google.api.services.searchconsole.v1.model.WmxSitemapContent;
public class ListSitemapExample {
//
// @siteUrl: https://foo.com/ (IMPORTANT: needs suffix "/")
//
private static void listSitemaps(String siteUrl) //
throws IOException, GeneralSecurityException {
SearchConsole.Sitemaps sitemapClient = MySitemapsUtils.getSitemapsClient();
SearchConsole.Sitemaps.List listRequest = sitemapClient.list(siteUrl);
SitemapsListResponse listResponse = listRequest.execute();
System.out.println("Sitemaps.List Response: " + listResponse.toPrettyString());
List<WmxSitemap> sitemapList = listResponse.getSitemap();
for (WmxSitemap sitemap : sitemapList) {
System.out.println("\nSitemap: " + sitemap.getPath());
System.out.println("- error: " + sitemap.getErrors());
System.out.println("- warnings: " + sitemap.getWarnings());
System.out.println("- isPending: " + sitemap.getIsPending());
System.out.println("- isSitemapsIndex: " + sitemap.getIsPending());
System.out.println("- lastDownloaded: " + sitemap.getLastDownloaded());
System.out.println("- lastSubmitted: " + sitemap.getLastSubmitted());
//
List<WmxSitemapContent> contentList = sitemap.getContents();
if (contentList != null) {
for (WmxSitemapContent content : contentList) {
System.out.println(" - indexed: " + content.getIndexed());
System.out.println(" - submitted: " + content.getSubmitted());
System.out.println(" - type: " + content.getType());
}
}
}
}
public static void main(String[] args) throws IOException, GeneralSecurityException {
listSitemaps(MyTestConstants.siteUrl);
}
}
Output:
Sitemaps.List Response: {
"sitemap": [
{
"contents": [
{
"indexed": "0",
"submitted": "890",
"type": "web"
}
],
"errors": "0",
"isPending": false,
"isSitemapsIndex": false,
"lastDownloaded": "2023-12-03T06:47:32.933Z",
"lastSubmitted": "2023-12-03T06:47:02.495Z",
"path": "http://foo.com/sitemap.xml",
"type": "sitemap",
"warnings": "0"
}
]
}
Sitemap: http://foo.com/sitemap.xml
- error: 0
- warnings: 0
- isPending: false
- isSitemapsIndex: false
- lastDownloaded: 2023-12-03T06:47:32.933Z
- lastSubmitted: 2023-12-03T06:47:02.495Z
- indexed: 0
- submitted: 890
- type: web
3. View and download a specific Sitemap
In this example we will retrieve a sitemap information by its URL.
The "siteUrl" parameter can be:
- sc-domain:yourdomain.com
- http://yourdomain.com/
- http://sub.yourdomain.com/
- https://langlearning.net/
- https://langlearning.net/app/en/
No ADS
GetSitemapExample.java
package org.o7planning.googleapis.a14297.sitemap;
import java.io.IOException;
import java.security.GeneralSecurityException;
import java.util.List;
import org.o7planning.googleapis.MyTestConstants;
import com.google.api.services.searchconsole.v1.SearchConsole;
import com.google.api.services.searchconsole.v1.model.WmxSitemap;
import com.google.api.services.searchconsole.v1.model.WmxSitemapContent;
public class GetSitemapExample {
//
// @siteUrl: https://foo.com/ (Needs suffix "/") (for a URL-prefix property)
// @siteUrl: sc-domain:foo.com (for a Domain property)
// @feedpath: https://foo.com/sitemap.xml
//
private static void getOneSitemapInfo(String siteUrl, String feedpath)
throws IOException, GeneralSecurityException {
SearchConsole.Sitemaps sitemapClient = MySitemapsUtils.getSitemapsClient();
SearchConsole.Sitemaps.Get getRequest = sitemapClient.get(siteUrl, feedpath);
WmxSitemap sitemap = getRequest.execute();
System.out.println("\nSitemap: " + sitemap.toPrettyString());
System.out.println("- path: " + sitemap.getPath());
System.out.println("- error: " + sitemap.getErrors());
System.out.println("- warnings: " + sitemap.getWarnings());
System.out.println("- isPending: " + sitemap.getIsPending());
System.out.println("- isSitemapsIndex: " + sitemap.getIsPending());
System.out.println("- lastDownloaded: " + sitemap.getLastDownloaded());
System.out.println("- lastSubmitted: " + sitemap.getLastSubmitted());
//
List<WmxSitemapContent> contentList = sitemap.getContents();
for (WmxSitemapContent content : contentList) {
System.out.println(" - indexed: " + content.getIndexed());
System.out.println(" - submitted: " + content.getSubmitted());
System.out.println(" - type: " + content.getType());
}
}
public static void main(String[] args) throws IOException, GeneralSecurityException {
getOneSitemapInfo(MyTestConstants.siteUrl, MyTestConstants.feedpath);
}
}
Output:
Sitemap: {
"contents": [
{
"indexed": "0",
"submitted": "890",
"type": "web"
}
],
"errors": "0",
"isPending": false,
"isSitemapsIndex": false,
"lastDownloaded": "2023-12-03T06:47:32.933Z",
"lastSubmitted": "2023-12-03T06:47:02.495Z",
"path": "https://langlearning.net/sitemap.xml",
"type": "sitemap",
"warnings": "0"
}
- path: https://langlearning.net/sitemap.xml
- error: 0
- warnings: 0
- isPending: false
- isSitemapsIndex: false
- lastDownloaded: 2023-12-03T06:47:32.933Z
- lastSubmitted: 2023-12-03T06:47:02.495Z
- indexed: 0
- submitted: 890
- type: web
For example, download a sitemap:
DownloadSitemapExample.java
package org.o7planning.googleapis.a14297.sitemap;
import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.security.GeneralSecurityException;
import org.o7planning.googleapis.MyTestConstants;
import com.google.api.services.searchconsole.v1.SearchConsole;
public class DownloadSitemapExample {
//
// @siteUrl: https://foo.com/ (IMPORTANT: needs suffix "/")
// @feedpath: https://foo.com/sitemap.xml
//
private static void downloadExample(String siteUrl, String feedpath) //
throws IOException, GeneralSecurityException {
SearchConsole.Sitemaps sitemapClient = MySitemapsUtils.getSitemapsClient();
SearchConsole.Sitemaps.Get getRequest = sitemapClient.get(siteUrl, feedpath);
System.out.println("Sitemaps.Get Request: " + getRequest.toString());
String outputFilePath = "/Volumes/New/Test/output/sitemap.xml";
FileOutputStream fileOutputStream = new FileOutputStream(new File(outputFilePath));
getRequest.executeAndDownloadTo(fileOutputStream);
fileOutputStream.close();
System.out.println("Done!");
}
public static void main(String[] args) throws IOException, GeneralSecurityException {
downloadExample(MyTestConstants.siteUrl, MyTestConstants.feedpath);
}
}
4. Add a Sitemap
Submitting a new sitemap file to Google means you tell Google to track this file and crawl the URLs specified within it. After this action the sitemap file you just submitted will appear on the list you get through the API. If you submit a sitemap file that is already on the list, nothing will happen.
SubmitSitemapExample.java
package org.o7planning.googleapis.a14297.sitemap;
import java.io.IOException;
import java.security.GeneralSecurityException;
import org.o7planning.googleapis.MyTestConstants;
import com.google.api.services.searchconsole.v1.SearchConsole;
public class SubmitSitemapExample {
//
// @siteUrl: https://foo.com/ (IMPORTANT: needs suffix "/")
// @feedpathToSubmit: https://foo.com/sitemap.xml
//
private static void submitSitemap(String siteUrl, String feedpathToSubmit) //
throws IOException, GeneralSecurityException {
SearchConsole.Sitemaps sitemapClient = MySitemapsUtils.getSitemapsClient();
SearchConsole.Sitemaps.Submit submitRequest //
= sitemapClient.submit(siteUrl, feedpathToSubmit);
submitRequest.execute();
System.out.println("Done!");
}
public static void main(String[] args) throws IOException, GeneralSecurityException {
submitSitemap(MyTestConstants.siteUrl, MyTestConstants.feedpathToSubmit);
}
}
5. Delete a Sitemap
No ADS
Using the Google Console API you can delete a sitemap file, which will be removed from the list of sitemaps returned from the Google Search API.
Over time, Google's system will stop trying to use the sitemap file you just deleted. Note - deleting the sitemap file does not prevent the URLs specified in the sitemap from being crawled. Because Google can detect links on your website when parsing the content of each page. However, sitemaps are necessary to help Google crawl more effectively, which is often why Google recommends using this file.
In addition, an important thing for Google to stop tracking the sitemap file you just deleted is that this sitemap must return error code 404 or 410.
Example, deleting a sitemap with the Google Search Java API:
DeleteSitemapExample.java
package org.o7planning.googleapis.a14297.sitemap;
import java.io.IOException;
import java.security.GeneralSecurityException;
import org.o7planning.googleapis.MyTestConstants;
import com.google.api.services.searchconsole.v1.SearchConsole;
public class DeleteSitemapExample {
//
// @siteUrl: https://foo.com/ (IMPORTANT: needs suffix "/")
// @feedpathToDelete: https://foo.com/sitemap2.xml
//
private static void deleteSitemap(String siteUrl, String feedpathToDelete) //
throws IOException, GeneralSecurityException {
SearchConsole.Sitemaps sitemapClient = MySitemapsUtils.getSitemapsClient();
SearchConsole.Sitemaps.Delete deleteRequest //
= sitemapClient.delete(siteUrl, feedpathToDelete);
deleteRequest.execute();
System.out.println("Done!");
}
public static void main(String[] args) throws IOException, GeneralSecurityException {
deleteSitemap(MyTestConstants.siteUrl, MyTestConstants.feedpathToDelete);
}
}
No ADS
- Creating a Google API Console project and OAuth2 Client ID
- Register Google Map API Key
- Get Started with the Google Search Console API
- Index pages with the Java Google Indexing API
- Use Google reCAPTCHA in Java Web Application
- Manipulating files and folders on Google Drive using Java
- Create a Google Service Account
- List, add and delete Sites with Google Search Java API
- List, submit and delete Sitemaps with Google Search Java API
Show More