Basic process explanation: A process has some initialization parameters that have to be set dynamically at runtime. For example data for a certain amount of dates (final_lookback) have to be pulled from sql. The amount of dates is based on a configuration parameter (base_lookback) + an amount that is pulled from another table. Therefore the final_lookback is only know after the process starts. It is necessary to save the dynamic configuration parameters to compare the next time the process runs. It is therefore also important that calculated parameters are not accidently changed later in the code thereby changing the logic of the running process. The process has multiple dynamic configuration parameters - final_lookback is an example of one. An additional complication is that the process is threaded and some of the dynamic parameters are general for all threads and others are thread specific and set when the thread is started. This question has two parts:

1. In order to manage all dynamically calculated input parameters so that the whole process has access to the same parameters and to ensure that they cannot be changed later in the process we created a singleton class/object to hold all dynamically modified runtime parameters. Basically a key value list similar to how configparser holds data. With the option to lock it after all parameters are set. Additionally to manage the thread specific parameters we considered creating a dictionary of dictionaries with each thread having it's own version of the dictionary of runtime parameters. So question 1 Is there a better way of managing dynamically set configuration parameters for the type of process described above?
2. If we were only dealing with one variable contextvars might be the place to store a dynamically changed variable with a wrapper function to lock it. However it has to be passed to threads and more importantly in our scenario we have multiple variables. Which brings up the ideas of using contextvars to hold a mutable object (dictionary or class). It seems that contextvars can hold a mutable object but it is unclear how exactly to implement it. Should the runtime parameters class be set once as a singleton on startup and the contextvars be included in the class? With contextvars holding a dictionary of key-values? Or create contextvars separately and have the runtime class access it - which would mean each file would need two imports one for contextvars and one for the class? So second question: Has the idea of using contextvars to hold a mutable container been tried? Would it be a valid solution for the process described and would passing to threads work so that each thread could update the contextvars variables it needs without affecting the other threads?

Basic process explanation: A process has some initialization parameters that have to be set dynamically at runtime. For example data for a certain amount of dates (final_lookback) have to be pulled from sql. The amount of dates is based on a configuration parameter (base_lookback) + an amount that is pulled from another table. Therefore the final_lookback is only know after the process starts. It is necessary to save the dynamic configuration parameters to compare the next time the process runs. It is therefore also important that calculated parameters are not accidently changed later in the code thereby changing the logic of the running process. The process has multiple dynamic configuration parameters - final_lookback is an example of one. An additional complication is that the process is threaded and some of the dynamic parameters are general for all threads and others are thread specific and set when the thread is started. This question has two parts:

1. In order to manage all dynamically calculated input parameters so that the whole process has access to the same parameters and to ensure that they cannot be changed later in the process we created a singleton class/object to hold all dynamically modified runtime parameters. Basically a key value list similar to how configparser holds data. With the option to lock it after all parameters are set. Additionally to manage the thread specific parameters we considered creating a dictionary of dictionaries with each thread having it's own version of the dictionary of runtime parameters. So question 1 Is there a better way of managing dynamically set configuration parameters for the type of process described above?
2. If we were only dealing with one variable contextvars might be the place to store a dynamically changed variable with a wrapper function to lock it. However it has to be passed to threads and more importantly in our scenario we have multiple variables. Which brings up the ideas of using contextvars to hold a mutable object (dictionary or class). It seems that contextvars can hold a mutable object but it is unclear how exactly to implement it. Should the runtime parameters class be set once as a singleton on startup and the contextvars be included in the class? With contextvars holding a dictionary of key-values? Or create contextvars separately and have the runtime class access it - which would mean each file would need two imports one for contextvars and one for the class? So second question: Has the idea of using contextvars to hold a mutable container been tried? Would it be a valid solution for the process described and would passing to threads work so that each thread could update the contextvars variables it needs without affecting the other threads?

• python
• multithreading
• dynamic
• singleton
• python-contextvars

• Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking. – Community Bot Commented Jan 20 at 10:13
• Thanks, I've reworded the question. – shoshanie Commented Jan 21 at 8:17

2 Answers 2

So, for part 1 - yes - what you are doing there reads like a sane approach, and I don't see any motives to change it.

As for (2): first, where to place the contextvar: it would work either as a top-level module value, or as an attribute in your class (or even on the instance, once it is a singleton). Since all things are equal, you can just attach it to your singleton anyway, so it can be passed along.

as for several key/values attached to the contextvar: if you have a single starting point in your threads where you can set all the key/values for that thread, a plain contextvar holding a dictionary will do. Just keep in mind it requires a new dictionary for each thread, and all keys should be populated at this point.

If you want to use the contextvar as a namespace, just like threading.local works, you can use the extracontext library (disclosure: I am the author) - you could them have a "ContextLocal" instance directly in your class, and use that as a namespace. (extracontext use Python's contextvas in the backend by default)

(demo in an ipython REPL:)

In [4]: %pip install python-extracontext

In [5]: from extracontext import ContextLocal

In [6]: class _Config:
   ...:     local = ContextLocal()
   ...: 

In [7]: Config = _Config()  # simplest singleton pattern! Just import "Config" from here.

In [9]: def worker(n):
   ...:     config.local.value = n   # create new attributes dynamically, just with one plain assingment!
   ...:     print(config.local.value)
   ...: 


In [11]: def worker(n):
    ...:     Config.local.value = n
    ...:     print(Config.local.value)
    ...: 

In [12]: worker(5)
5

In [14]: Config.local.value
Out[14]: 5

In [15]: import threading

In [16]: threading.Thread(target=worker, args=(23,)).start()
23

In [17]: Config.local.value
Out[17]: 5

Should you like to have a "locking" mechanism to prevent changes to ContextLocal attributes just like you do on your config class, and are willing to use extracontext, just contact me (could be through an issue on the extracontext github project) - I could implement that.

There is a single starting point for each thread where all dynamically set parameters are initialized. However some of the parameters are the same across the entire process for all threads and other parameters are thread specific. Ideally it would be best if in the process later there is no need to know which parameters were set by which part and that they could all be accessed in the same way. For example:

General settings

groupname = abc
base_lookback = 5

Thread-specific settings

final_lookback = 8

Ideally, access to groupname, base_lookback, and final_lookback should be the same ctx.base_lookback or ctx.final_lookback should both work without the thread having to know if the parameters are thread specific. Having a new dictionary for each thread would mean that the two parts of the data are separate or we would have to copy the general data into each thread specific dictionary.

From what I understood about extracontext, it essentially does what we were trying to do - set up a contextvars which allows multiple variables. If we instantiate extracontext` one time so that it acts as a singleton.

context.py:

ctx  = ContextLocal()

file1.py:

from context import ctx

file2.py

from context import ctx

We would then have the same contextlocal used throughout the process. It can be initialized at the beginning with all general parameters and then per thread with thread-specific parameters and it would be thread and async safe. Am I understanding correctly?

We could then also add a decorator around it to lock it.