[Fixed]-Where to instantiate boto s3 client so it is reused during a request?

by

3👍

If you use a lambda client, go with global. The client lets you reuse execution environments which has cost and performance savings

Take advantage of execution environment reuse to improve the performance of your function. Initialize SDK clients and database connections outside of the function handler

https://docs.aws.amazon.com/lambda/latest/dg/best-practices.html

Otherwise I think this is a stylistic choice dependent on your app.

If your client will never change, global seems like a safe way to do it. The drawback is since it’s a constant, you shouldn’t change it during runtime. This has consequences, e.g. this makes changing Session hard. You could use a singleton but the code would become more verbose.

If you instantiate clients everywhere, you run the risk of making a client() call signature change a large effort, eg if you need to pass client('s3', verify=True), you’d have to add verify=True everywhere which is a pain. It’s unlikely you’d do this though. The only param you’re likely to override is config which you can pass through the session using set_default_config.

You could make it its own module, eg

foo.bar.aws.clients

session = None
ecs_client = None
eks_client = None

def init_session(new_session):
  session = new_session
  ecs_client = session.client('ecs')
  eks_client = session.client('eks')

You can call init_session from an appropriate place or have defaults and an import hook to auto instatiate. This file will get larger as you use more clients but at least the ugliness is contained. You could also do a hack like

def init_session(s):
  session = s
  clients = ['ecs', 'iam', 'eks', …]
  for c in clients:
    globals()[f'{c}_client'] = session.client(c)

The problem is the indirection that this hack adds, eg intelliJ is not smart enough to figure out where your clients came from and will say you are using an undefined variable.

👤wonton

1👍

My best approach is to use functools.partial and have all the constant variables such as bucket and other metadata frozen in a partial and then pust pass in variable data. However, boto3 is still slow as hell to create the signed urls, compared to a simple string format it is ~x100 slower.

👤Cristiano Coelho

Leave a comment