Pulp RecursionError: maximum recursion depth exceeded while calling a Python object

[ruksarsultana]

I'm working on a MIP problem using pulp on a large dataset with approx 0.3 million variables & numerous constraints which takes a lot of computation time in constraint formulation. To optimize the code, I have recently tried creating the constraints using Pool option from python's multiprocessing package. I'm storing the constraints (left part, right part, sense) in a list & then adding the constraints to the pulp model using lpmodel += left part <= right part by looping through the list. Attaching the pulp model (MPS file) from a sample data for reference

Issue - I'm stuck with the below error when running the model - lpmodel.solve(PULP_CBC_CMD(msg=1)) RecursionError:

model_pulp.zip
model_pulp.zip

maximum recursion depth exceeded while calling a Python object

I have also tried setting higher recursion limit using sys.setrecursionlimit(1500), however that isn't solving the problem too.

Details - pulp version (2.3), Solver (PULP_CBC_CMD), Operating system (Windows 64-bit)

[pchtsp]

Using the multiprocessing package usually causes problems in pulp. There are issues with how the variables are stored and referenced inside the constraints, I think. Because the references to the pulp objects created inside each worker in the pool are not correctly handled afterwards.

If you want to improve the time it takes to build the model, you could share the code and we could help you make it more efficient.
For example, here there is some advice: https://stackoverflow.com/a/64497199/6508131

Note that if i use your first file, it does solve:

import pulp
file = 'model_pulp/model_pulp.json'
variables, model = pulp.LpProblem.from_json(file)
model.solve(pulp.PULP_CBC_CMD())

If you really want to go the multiprocessing route, you would have to export variables and constraints from each worker using the toDict() method, and then rejoining everything in the master node and then recreate the model with something like pulp.LpProblem.from_dict() or pulp.LpProblem.from_json().

This is not documented because it's usually a bad idea. Usually, the time it takes to build the model in a pulp code that follows best practices takes a very small fraction of the time it takes to actually solve.

[ruksarsultana]

Thanks for the response. I was able to resolve the Recursion error by storing the model to a dictionary/ json & then reloading the model from the dictionary/ json (as suggested above).

Please excuse I don't have a minimum reproducible code, so I won't be able to share the code due to DLP reasons.

[tle4336]

@ruksarsultana May I know at which step did you export your model to a dictionary/json? Is it after you added all the constraints to the Lp problem of class pulp?

[tle4336]

@pchtsp Hi Mr. Pescheria, if I use another package like mpire and also create a pool with threading as a start method (see here for that package -- https://github.com/sybrenjansen/mpire) , does the same issue of the OP also hold? It seems to me that the same problem persists, but I would love to have you confirm.