Package google.registry.bigquery

Class BigqueryConnection

java.lang.Object
google.registry.bigquery.BigqueryConnection
All Implemented Interfaces:
AutoCloseable
public class BigqueryConnection extends Object implements AutoCloseable
Class encapsulating parameters and state for accessing the Bigquery API.

  • Nested Class Summary

    Nested Classes
    Modifier and Type
    Class
    Description
    static class 
    BigqueryConnection.Builder
    Builder for a BigqueryConnection, since the latter is immutable once created.
    static class 
    BigqueryConnection.DestinationTable
    Class that wraps a normal Bigquery API Table object to make it immutable from the client side and give it additional semantics as a "destination" for load or query jobs, with an overwrite flag set by the client upon creation.

  • Field Summary

    Fields
    Modifier and Type
    Field
    Description
    static final String
    DEFAULT_DATASET_NAME
    Default name of the default dataset to use for requests to the API.

  • Method Summary

    Modifier and Type
    Method
    Description
    BigqueryConnection.DestinationTable.Builder
    buildDestinationTable(String tableName)
    Returns a partially built DestinationTable with the default dataset and overwrite behavior.
    BigqueryConnection.DestinationTable.Builder
    buildTemporaryTable()
    Returns a partially built DestinationTable with a randomly generated name under the default temporary table dataset, with the default TTL and overwrite behavior.
    boolean
    checkDatasetExists(String datasetName)
    Helper that returns true if a dataset with this name exists.
    boolean
    checkTableExists(String datasetName, String tableName)
    Helper that returns true if a table with this name and dataset name exists.
    void
    close()
    Closes the BigqueryConnection object by shutting down the executor service.
    com.google.api.services.bigquery.model.TableReference
    ensureTable(com.google.api.services.bigquery.model.TableReference table, String sqlQuery)
    Create a table from a SQL query if it doesn't already exist.
    com.google.common.util.concurrent.ListenableFuture<String>
    extract(BigqueryConnection.DestinationTable sourceTable, String destinationUri, BigqueryUtils.DestinationFormat destinationFormat, boolean printHeader)
    Starts an asynchronous job to extract the specified source table or view and output it to the given GCS filepath in the specified destination format, optionally printing headers.
    com.google.common.util.concurrent.ListenableFuture<String>
    extractQuery(String querySql, String destinationUri, BigqueryUtils.DestinationFormat destinationFormat, boolean printHeader)
    Starts an asynchronous job to run the provided query, store the results in a temporary table, and then extract the contents of that table to the given GCS filepath in the specified destination format, optionally printing headers.
    com.google.api.services.bigquery.model.DatasetReference
    getDataset()
    Returns dataset reference that can be used to avoid having to specify dataset in SQL code.
    String
    getDatasetId()
    Returns the dataset name that this bigquery connection uses by default.
    com.google.api.services.bigquery.model.TableReference
    getTable(String tableName)
    Returns table reference with the projectId and datasetId filled out for you.
    com.google.common.util.concurrent.ListenableFuture<com.google.common.collect.ImmutableTable<Integer,com.google.api.services.bigquery.model.TableFieldSchema,Object>>
    queryToLocalTable(String querySql)
    Starts an asynchronous query job to dump the results of the specified query into a local ImmutableTable object, row-keyed by the row number (indexed from 1), column-keyed by the TableFieldSchema for that column, and with the value object as the cell value.
    com.google.common.collect.ImmutableTable<Integer,com.google.api.services.bigquery.model.TableFieldSchema,Object>
    queryToLocalTableSync(String querySql)
    Returns the result of calling queryToLocalTable, but synchronously to avoid spawning new background threads, which App Engine doesn't support.
    com.google.common.util.concurrent.ListenableFuture<BigqueryConnection.DestinationTable>
    startQuery(String querySql, BigqueryConnection.DestinationTable dest)
    Starts an asynchronous query job to populate the specified destination table with the results of the specified query, or if the table is a view, to update the view to reflect that query.

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Field Details

    • DEFAULT_DATASET_NAME

      public static final String DEFAULT_DATASET_NAME
      Default name of the default dataset to use for requests to the API.
      See Also:

  • Method Details

    • close

      public void close()
      Closes the BigqueryConnection object by shutting down the executor service. Clients should only call this after all ListenableFutures obtained from BigqueryConnection methods have resolved; this method does not block on their completion.
      Specified by:
      close in interface AutoCloseable

    • buildDestinationTable

      public BigqueryConnection.DestinationTable.Builder buildDestinationTable(String tableName)
      Returns a partially built DestinationTable with the default dataset and overwrite behavior.

    • buildTemporaryTable

      public BigqueryConnection.DestinationTable.Builder buildTemporaryTable()
      Returns a partially built DestinationTable with a randomly generated name under the default temporary table dataset, with the default TTL and overwrite behavior.

    • startQuery

      public com.google.common.util.concurrent.ListenableFuture<BigqueryConnection.DestinationTable> startQuery(String querySql, BigqueryConnection.DestinationTable dest)
      Starts an asynchronous query job to populate the specified destination table with the results of the specified query, or if the table is a view, to update the view to reflect that query. Returns a ListenableFuture that holds the same destination table object on success.

    • queryToLocalTable

      public com.google.common.util.concurrent.ListenableFuture<com.google.common.collect.ImmutableTable<Integer,com.google.api.services.bigquery.model.TableFieldSchema,Object>> queryToLocalTable(String querySql)
      Starts an asynchronous query job to dump the results of the specified query into a local ImmutableTable object, row-keyed by the row number (indexed from 1), column-keyed by the TableFieldSchema for that column, and with the value object as the cell value. Note that null values will not actually be null, but they can be checked for using Data.isNull().

      Returns a ListenableFuture that holds the ImmutableTable on success.

    • queryToLocalTableSync

      public com.google.common.collect.ImmutableTable<Integer,com.google.api.services.bigquery.model.TableFieldSchema,Object> queryToLocalTableSync(String querySql)
      Returns the result of calling queryToLocalTable, but synchronously to avoid spawning new background threads, which App Engine doesn't support.
      See Also:

    • extract

      public com.google.common.util.concurrent.ListenableFuture<String> extract(BigqueryConnection.DestinationTable sourceTable, String destinationUri, BigqueryUtils.DestinationFormat destinationFormat, boolean printHeader)
      Starts an asynchronous job to extract the specified source table or view and output it to the given GCS filepath in the specified destination format, optionally printing headers. Returns a ListenableFuture that holds the destination GCS URI on success.

    • extractQuery

      public com.google.common.util.concurrent.ListenableFuture<String> extractQuery(String querySql, String destinationUri, BigqueryUtils.DestinationFormat destinationFormat, boolean printHeader)
      Starts an asynchronous job to run the provided query, store the results in a temporary table, and then extract the contents of that table to the given GCS filepath in the specified destination format, optionally printing headers.

      Returns a ListenableFuture that holds the destination GCS URI on success.

    • checkDatasetExists

      public boolean checkDatasetExists(String datasetName) throws IOException
      Helper that returns true if a dataset with this name exists.
      Throws:
      IOException

    • checkTableExists

      public boolean checkTableExists(String datasetName, String tableName) throws IOException
      Helper that returns true if a table with this name and dataset name exists.
      Throws:
      IOException

    • getDatasetId

      public String getDatasetId()
      Returns the dataset name that this bigquery connection uses by default.

    • getDataset

      public com.google.api.services.bigquery.model.DatasetReference getDataset()
      Returns dataset reference that can be used to avoid having to specify dataset in SQL code.

    • getTable

      public com.google.api.services.bigquery.model.TableReference getTable(String tableName)
      Returns table reference with the projectId and datasetId filled out for you.

    • ensureTable

      public com.google.api.services.bigquery.model.TableReference ensureTable(com.google.api.services.bigquery.model.TableReference table, String sqlQuery)
      Create a table from a SQL query if it doesn't already exist.