[Fixed]-Receiving intermittent error when querying BigQuery database from Django App hosted on Apache2 Server

After trying many different methods to make a workaround for this issue, I ended up developing a hack that works for now I guess. But I really think there is a bug in Google API’s Authentication. I mean if you want to test it out, visit www.blesque.tv. You can see that now it works good. Slower than if I was hitting Google’s API directly, but works. I really didn’t change anything in my BigQuery Controller’s Authentication. I simply started executing it through a “subprocess” and then read the results of a query from “stdout”. But that’s not the fix, it is a hack. Google needs to fix the bug that is blocking me and I bet many others from being able to use Bigquery API the correct way. Code to my fix is below. Feel free to comment. I mean maybe someone still knows better way to fix this issue and I hope in the future Google will fix their API Authentication bugs. There is some conflict when authenticating from within the application running on Django platform when it tries to create a new token or acquire existing one. Bug basically that Google needs to fix.

My BigQuery Controller:

__author__ = 'igor_vishnevskiy'


#Gogle's API Refference page:
#https://cloud.google.com/bigquery/docs/reference/v2/jobs#configuration.query
#https://cloud.google.com/bigquery/docs/reference/v2/tables


from oauth2client.service_account import ServiceAccountCredentials
from httplib2 import Http
from apiclient.discovery import build
import os
from googleapiclient.errors import HttpError
import random
import json
import sys


class BQController():

    def __init__(self):
        self.authenticate_one()
        query = sys.argv[1]
        print self.runQuery(query)


    def authenticate_one(self):

        configFile = "pass_to_config_file_containing_p12_key_client_email_and_project_id"

        #Authorizing connection
        with open(configFile, "r") as conConf:
            filedata = conConf.read()
            jsonData = self.getJsonContent(filedata)
            KEY_FILE = jsonData["signin_credentials"]["key_file"]
            self.PROJECT_ID = jsonData["signin_credentials"]["project_id"]
            CLIENT_EMAIL = jsonData["signin_credentials"]["client_email"]

        scopes = ['https://www.googleapis.com/auth/bigquery']

        self.credentials = ServiceAccountCredentials.from_p12_keyfile(
            CLIENT_EMAIL, KEY_FILE, scopes=scopes)

        self.http_auth = self.credentials.authorize(Http())#ca_certs=os.environ['REQUESTS_CA_BUNDLE']))
        self.credentials.refresh(self.http_auth)
        self.bigquery = build('bigquery', 'v2',  http=self.http_auth)

    def runQuery(self, queryStr):
        try:
            query_request = self.bigquery.jobs()
            query_data = {
                'query': (
                    queryStr)
            }

            try:
                query_response = query_request.query(
                    projectId=self.PROJECT_ID,
                    body=query_data).execute()
                queryset = query_response['rows']
                return queryset
            except:
                self.credentials.refresh(self.http_auth)
                self.runQuery(queryStr)

        except HttpError as err:
            queryset = []
            print('Error: {}'.format(err.content))
            raise err
            return queryset


    def getJsonContent(self, jsonStr):
        jsonData = json.loads(jsonStr)
        return jsonData

BQController()

Code that I used to convert stdout of BQController() into a queryset object that then I pass to my Django template:

def my_view(request):

    subProcessResponse = runShellCommandWithLogReturnedAsString("cd /home/businessai/webapps/blesque_tv/trydjango19/product_upload/controllers/; python bigquery_run.py 'SELECT product_id as id, supplier_name as db_name, image_file as image_url, title, description as clean_content, msrp as marked_up_price, product_sku, price, ship_cost, drop_ship_fee, condition FROM products.doba_inventory LIMIT 10'")
    queryset = eval(subProcessResponse)

    context = {
        "object_list" : queryset,
    }

    return render(request, "template.html", context)

def runShellCommandWithLogReturnedAsString(commandAsStr):
    logToReturn = ""
    try:
        popen = subprocess.Popen(commandAsStr, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, shell=True)
        lines_iterator = iter(popen.stdout.readline, b"")
        for line in lines_iterator:
            logToReturn = logToReturn + str(line)
    except Exception,e:
        print e

    return logToReturn