Class BigqueryConnection

  • All Implemented Interfaces:
    java.lang.AutoCloseable

    public class BigqueryConnection
    extends java.lang.Object
    implements java.lang.AutoCloseable
    Class encapsulating parameters and state for accessing the Bigquery API.
    • Nested Class Summary

      Nested Classes 
      Modifier and Type Class Description
      static class  BigqueryConnection.Builder
      Builder for a BigqueryConnection, since the latter is immutable once created.
      static class  BigqueryConnection.DestinationTable
      Class that wraps a normal Bigquery API Table object to make it immutable from the client side and give it additional semantics as a "destination" for load or query jobs, with an overwrite flag set by the client upon creation.
    • Field Summary

      Fields 
      Modifier and Type Field Description
      static java.lang.String DEFAULT_DATASET_NAME
      Default name of the default dataset to use for requests to the API.
    • Method Summary

      All Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      BigqueryConnection.DestinationTable.Builder buildDestinationTable​(java.lang.String tableName)
      Returns a partially built DestinationTable with the default dataset and overwrite behavior.
      BigqueryConnection.DestinationTable.Builder buildTemporaryTable()
      Returns a partially built DestinationTable with a randomly generated name under the default temporary table dataset, with the default TTL and overwrite behavior.
      boolean checkDatasetExists​(java.lang.String datasetName)
      Helper that returns true if a dataset with this name exists.
      boolean checkTableExists​(java.lang.String datasetName, java.lang.String tableName)
      Helper that returns true if a table with this name and dataset name exists.
      void close()
      Closes the BigqueryConnection object by shutting down the executor service.
      com.google.api.services.bigquery.model.TableReference ensureTable​(com.google.api.services.bigquery.model.TableReference table, java.lang.String sqlQuery)
      Create a table from a SQL query if it doesn't already exist.
      com.google.common.util.concurrent.ListenableFuture<java.lang.String> extract​(BigqueryConnection.DestinationTable sourceTable, java.lang.String destinationUri, BigqueryUtils.DestinationFormat destinationFormat, boolean printHeader)
      Starts an asynchronous job to extract the specified source table or view and output it to the given GCS filepath in the specified destination format, optionally printing headers.
      com.google.common.util.concurrent.ListenableFuture<java.lang.String> extractQuery​(java.lang.String querySql, java.lang.String destinationUri, BigqueryUtils.DestinationFormat destinationFormat, boolean printHeader)
      Starts an asynchronous job to run the provided query, store the results in a temporary table, and then extract the contents of that table to the given GCS filepath in the specified destination format, optionally printing headers.
      com.google.api.services.bigquery.model.DatasetReference getDataset()
      Returns dataset reference that can be used to avoid having to specify dataset in SQL code.
      java.lang.String getDatasetId()
      Returns the dataset name that this bigquery connection uses by default.
      com.google.api.services.bigquery.model.TableReference getTable​(java.lang.String tableName)
      Returns table reference with the projectId and datasetId filled out for you.
      com.google.common.util.concurrent.ListenableFuture<com.google.common.collect.ImmutableTable<java.lang.Integer,​com.google.api.services.bigquery.model.TableFieldSchema,​java.lang.Object>> queryToLocalTable​(java.lang.String querySql)
      Starts an asynchronous query job to dump the results of the specified query into a local ImmutableTable object, row-keyed by the row number (indexed from 1), column-keyed by the TableFieldSchema for that column, and with the value object as the cell value.
      com.google.common.collect.ImmutableTable<java.lang.Integer,​com.google.api.services.bigquery.model.TableFieldSchema,​java.lang.Object> queryToLocalTableSync​(java.lang.String querySql)
      Returns the result of calling queryToLocalTable, but synchronously to avoid spawning new background threads, which App Engine doesn't support.
      com.google.common.util.concurrent.ListenableFuture<BigqueryConnection.DestinationTable> startLoad​(BigqueryConnection.DestinationTable dest, BigqueryUtils.SourceFormat sourceFormat, java.lang.Iterable<java.lang.String> sourceUris)
      Starts an asynchronous load job to populate the specified destination table with the given source URIs and source format.
      com.google.common.util.concurrent.ListenableFuture<BigqueryConnection.DestinationTable> startQuery​(java.lang.String querySql, BigqueryConnection.DestinationTable dest)
      Starts an asynchronous query job to populate the specified destination table with the results of the specified query, or if the table is a view, to update the view to reflect that query.
      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
    • Field Detail

      • DEFAULT_DATASET_NAME

        public static final java.lang.String DEFAULT_DATASET_NAME
        Default name of the default dataset to use for requests to the API.
        See Also:
        Constant Field Values
    • Method Detail

      • close

        public void close()
        Closes the BigqueryConnection object by shutting down the executor service. Clients should only call this after all ListenableFutures obtained from BigqueryConnection methods have resolved; this method does not block on their completion.
        Specified by:
        close in interface java.lang.AutoCloseable
      • buildDestinationTable

        public BigqueryConnection.DestinationTable.Builder buildDestinationTable​(java.lang.String tableName)
        Returns a partially built DestinationTable with the default dataset and overwrite behavior.
      • buildTemporaryTable

        public BigqueryConnection.DestinationTable.Builder buildTemporaryTable()
        Returns a partially built DestinationTable with a randomly generated name under the default temporary table dataset, with the default TTL and overwrite behavior.
      • startQuery

        public com.google.common.util.concurrent.ListenableFuture<BigqueryConnection.DestinationTable> startQuery​(java.lang.String querySql,
                                                                                                                  BigqueryConnection.DestinationTable dest)
        Starts an asynchronous query job to populate the specified destination table with the results of the specified query, or if the table is a view, to update the view to reflect that query. Returns a ListenableFuture that holds the same destination table object on success.
      • queryToLocalTable

        public com.google.common.util.concurrent.ListenableFuture<com.google.common.collect.ImmutableTable<java.lang.Integer,​com.google.api.services.bigquery.model.TableFieldSchema,​java.lang.Object>> queryToLocalTable​(java.lang.String querySql)
        Starts an asynchronous query job to dump the results of the specified query into a local ImmutableTable object, row-keyed by the row number (indexed from 1), column-keyed by the TableFieldSchema for that column, and with the value object as the cell value. Note that null values will not actually be null, but they can be checked for using Data.isNull().

        Returns a ListenableFuture that holds the ImmutableTable on success.

      • queryToLocalTableSync

        public com.google.common.collect.ImmutableTable<java.lang.Integer,​com.google.api.services.bigquery.model.TableFieldSchema,​java.lang.Object> queryToLocalTableSync​(java.lang.String querySql)
        Returns the result of calling queryToLocalTable, but synchronously to avoid spawning new background threads, which App Engine doesn't support.
        See Also:
        App Engine Runtime
      • extract

        public com.google.common.util.concurrent.ListenableFuture<java.lang.String> extract​(BigqueryConnection.DestinationTable sourceTable,
                                                                                            java.lang.String destinationUri,
                                                                                            BigqueryUtils.DestinationFormat destinationFormat,
                                                                                            boolean printHeader)
        Starts an asynchronous job to extract the specified source table or view and output it to the given GCS filepath in the specified destination format, optionally printing headers. Returns a ListenableFuture that holds the destination GCS URI on success.
      • extractQuery

        public com.google.common.util.concurrent.ListenableFuture<java.lang.String> extractQuery​(java.lang.String querySql,
                                                                                                 java.lang.String destinationUri,
                                                                                                 BigqueryUtils.DestinationFormat destinationFormat,
                                                                                                 boolean printHeader)
        Starts an asynchronous job to run the provided query, store the results in a temporary table, and then extract the contents of that table to the given GCS filepath in the specified destination format, optionally printing headers.

        Returns a ListenableFuture that holds the destination GCS URI on success.

      • checkDatasetExists

        public boolean checkDatasetExists​(java.lang.String datasetName)
                                   throws java.io.IOException
        Helper that returns true if a dataset with this name exists.
        Throws:
        java.io.IOException
      • checkTableExists

        public boolean checkTableExists​(java.lang.String datasetName,
                                        java.lang.String tableName)
                                 throws java.io.IOException
        Helper that returns true if a table with this name and dataset name exists.
        Throws:
        java.io.IOException
      • getDatasetId

        public java.lang.String getDatasetId()
        Returns the dataset name that this bigquery connection uses by default.
      • getDataset

        public com.google.api.services.bigquery.model.DatasetReference getDataset()
        Returns dataset reference that can be used to avoid having to specify dataset in SQL code.
      • getTable

        public com.google.api.services.bigquery.model.TableReference getTable​(java.lang.String tableName)
        Returns table reference with the projectId and datasetId filled out for you.
      • ensureTable

        public com.google.api.services.bigquery.model.TableReference ensureTable​(com.google.api.services.bigquery.model.TableReference table,
                                                                                 java.lang.String sqlQuery)
        Create a table from a SQL query if it doesn't already exist.