Clojurians Log v2

I think you need to call .build on your Builder after you set the jobId?

So in your example @oliver.marks:

(.create s
         (-> (query)
             (JobInfo/newBuilder)
             (.setJobId (JobId/of (str (java.util.UUID/randomUUID))))
             (.build)))

Not that I’ve used that before, but I just know from other Java GCP libs that use the builder pattern…

For BQ we eventually opted for using the API’s directly.

But there are other things you’ll need to do then, like parsing the super weird result structure (I can share our bad code if you like)

I would be interested, I did consider working direct with the api but had not seen anyone else doing this, all other code seems to work with the java api's I would like to see a gcp equivalent of the congnitech amazon api that would be nice 🙂

thanks for the tip, I had missed out the .build step I can now run the queries and gets some results 🙂

Awesome!

I'll share some code when I land.

here you go @oliver.marks :

(declare row->clj)

(defn value->clj [field-schema type val]
  (when val
    (case type
      "STRING" val
      "FLOAT" (Float/parseFloat val)
      "INTEGER" (Long/parseLong val)
      "BOOLEAN" (Boolean/parseBoolean val)
      "DATETIME" (time.coerce/to-date (time.coerce/to-date-time val))
      "DATE" (time.coerce/to-date val)
      "TIMESTAMP" (java.util.Date. (-> (Double/parseDouble val)
                                       (.longValue)
                                       (* 1000)))

      "RECORD" (row->clj field-schema val))))
(defn field->clj [field-schema field]
  (let [value (:v field)
        type (:type field-schema)
        mode (:mode field-schema)
        value (if (= mode "REPEATED")
                (map #(value->clj field-schema type (:v %)) value)
                (value->clj field-schema type value))]
    {(keyword (:name field-schema)) value}))

(defn row->clj [row-schema row]
  (into {}
        (map field->clj (:fields row-schema) (:f row))))

(defn bq-response->clj [resp]
  (let [schema (:schema resp)
        rows (:rows resp)]
    (map #(row->clj schema %) rows)))

used like this:

(defn get-gcp-token []
  (let [credentials-client (-> (GoogleCredential/getApplicationDefault))
        _ (.refreshToken credentials-client)
        token (.getAccessToken credentials-client)]
    token))

(defn fetch-paged-bq-results [project job-id page-token]
  (let [token (get-gcp-token)
        response (-&gt; (client/get (format "<https://bigquery.googleapis.com/bigquery/v2/projects/%s/queries/%s>" project job-id)
                                 {:query-params {:pageToken page-token}
                                  :oauth-token token
                                  :throw-exceptions false})
                     :body
                     (json/decode true))
        page-token (:pageToken response)]
    (if (:error response)
      nil
      (if page-token
        (concat (fetch-paged-bq-results project (:jobId (:jobReference response)) page-token) (bq-response-&gt;clj response))
        (bq-response-&gt;clj response)))))

(defn execute-bq-query [project query]
  (let [token (get-gcp-token)
        response (-&gt; (client/post (format "<https://bigquery.googleapis.com/bigquery/v2/projects/%s/queries>" project)
                                  {:body (json/encode {:query query
                                                       :useLegacySql false
                                                       :maxResults 100000
                                                       :timeoutMs 100000})
                                   :oauth-token token
                                   :throw-exceptions false})
                     :body
                     (json/decode true))
        page-token (:pageToken response)]
    (if (:error response)
      nil
      (if page-token
        (concat (fetch-paged-bq-results project (:jobId (:jobReference response)) page-token) (bq-response-&gt;clj response))
        (bq-response-&gt;clj response)))))

running locally, you need to have run gcloud auth application-default login

This can certainly be made better. We haven’t had time to do some much needed refactoring yet though.

@domparry thank you very much for that, looks like it could be made much nicer to work with than the java api

yes. Much more repl friendly….

yeah exactly, I had been using this as a reference previously https://github.com/RakutenReady/hugsql-bq but using java

If i get my task complete I will try it out and probably use that going forward 🙂

@domparry just curious where does this come from ? GoogleCredential/getApplicationDefault is this from java ?

Yes, that’s correct. From here: com.google.api.client.googleapis.auth.oauth2.GoogleCredential