AbstractModel, ConcreteModel, DataPortal and Problem dumps with Pyomo

AbstractModel, ConcreteModel, DataPortal and Problem dumps with Pyomo

And flex your muscles!

·

9 min read

In the previous article we solved our first problem with Pyomo and SCIP. Today we solve a second problem. We introduce the concepts of AbstractModel, ConcreteModel and DataPortal. We also learn how to export a Problem into a text file for debugging.

The Truck Allocation Problem

We run a generalist product supplier business that sells a variety of products worldwide. Every day a truck comes to the warehouse to load the items that must be delivered to retailers. Usually, the number total number of items to be shipped exceed truck's capacity. Therefore we must decide which items we load into the truck and which items stay on the warehouse until the next truck.

Retailers do not pay in advance, meaning that we only get paid for the products that we actually send to them. Hence, we want to maximize the profits of all items that we ship with the truck.

As we already mentioned in the first article, a well defined Mathematical Programming Problem packs all the information that we need to understand and solve this problem. The following Pyomo implementation should be clear enough.

💡
Notice how doc, docstrings, variable names and comments are crucial to understand the code and the Problem itself. They are the lighthouse for code readers and maintainers. Stick this in your mind.
from pyomo.environ import AbstractModel

model = AbstractModel(name='truck')

Sets and Parameters

from pyomo.environ import (
    NonNegativeIntegers,
    NonNegativeReals,
    Param,
    Set,
)

model.products = Set(
    name='products',
    doc='Set of products to load into the truck.',
)

model.stock = Param(
    model.products,  # One parameter value for each element in products set
    name='stock',
    doc='Total number of items requested of each product',
    domain=NonNegativeIntegers,
)

model.profits = Param(
    model.products,  # One parameter value for each element in products set
    name='profits',
    doc=(
        'Profits obtained for each item that we load into the truck. '
        'Units are €/item'
    ),
    domain=NonNegativeReals,
)

model.volume = Param(
    model.products,  # One parameter value for each element in products set
    name='volume',
    doc='Volume of each product in cubic meters',
    domain=NonNegativeReals,
)

model.truck_volume = Param(  # Just one parameter
    name='truck_volume',
    doc='The volume of the truck in cubic meters',
    domain=NonNegativeReals,
)

Variables

from typing import Tuple

from pyomo.environ import AbstractModel, NonNegativeIntegers, Var


def number_of_items_bounds(
    model: AbstractModel,
    product: str,
) -> Tuple[int, int]:
    """Get the lower bound and upper bound of variable stock."""
    lower_bound = 0
    upper_bound = model.stock[product]
    return lower_bound, upper_bound

model.number_of_items = Var(
    model.products,  # One variable for each element in products set
    name='number_of_items',
    doc='Number of items to load into the truck',
    domain=NonNegativeIntegers,
    bounds=number_of_items_bounds,
)

We define a variable that takes non negative nteger values (0, 1, 2, ...). The number of items to ship is upper bounded by the total number of items to be sent.

The argument bounds only admits a callable that return two floats. If you need to bound a variable with a Pyomo expression, you must use a constraint.

Constraints

from typing import Tuple

from pyomo.environ import AbstractModel, Constraint
from pyomo.core.expr.relational_expr import InequalityExpression


def constraint_volume_of_items_cannot_exceed_truck_volume(
    model: AbstractModel,
) -> InequalityExpression:
    """The volume of items in the container cannot exceed truck volume."""
    shipped_volume = sum(
        model.number_of_items[product] * model.volume[product]
        for product in model.products
    )
    return shipped_volume <= model.truck_volume

# Only one constraint.
model.volume_of_items_cannot_exceed_truck_volume = Constraint(
    name='volume_of_items_cannot_exceed_truck_volume',
    doc=constraint_volume_of_items_cannot_exceed_truck_volume.__doc__,
    rule=constraint_volume_of_items_cannot_exceed_truck_volume,
)

Objective

Thanks to the function name, the objective is crystal clear. We want to maximize the shipped_profits.

from pyomo.environ import AbstractModel, Objective, maximize
from pyomo.core.expr.numeric_expr import LinearExpression


def shipped_profits(model: AbstractModel) -> LinearExpression:
    """Sum the profits of all items that we load into the truck."""
    shipped_profits = sum(
        model.number_of_items[product] * model.profits[product]
        for product in model.products
    )
    return shipped_profits

model.objective_function = Objective(rule=shipped_profits ,sense=maximize)

Instantiate a ConcreteModel

Mathematics build algorithms that solve Abstract Problems theoretically. This problem we're solving here is known as the Knapsack Problem, and it is already solved mathematically for any set and parameters that you choose.

In Pyomo, an AbstractModel is a template with no data. Still the values of the sets and parameters are not defined. The bounds of the variables and the actual expression of the constraint is not defined neither. Thus, a Solver cannot solve instances of AbstractModel because they have no data.

The Solver works with instances of ConcreteModel. Think of a ConcreteModel an AbstractModel (the templated) filled with actual data. Eventually, the Solver also stores the optimal solution (and other non-optimal solutions as well) into the ConcreteModel.

One way to load data with Pyomo is by using a DataPortal. In this example the DataPortal loads the file data.json containing the values of Sets and parameters. Here's the file.

# File: data.json
{
    "products": ["toys", "treadmill", "locker", "costumes"],
    "stock": {
        "toys": 15,
        "treadmill": 10,
        "locker": 5,
        "costumes": 20
    },
    "profits": {
        "toys": 12,
        "treadmill": 95.0,
        "locker": 200,
        "costumes": 40.0
    },
    "volume": {
        "toys": 0.5,
        "treadmill": 3.4,
        "locker": 7.2,
        "costumes": 1.4
    },
    "truck_volume": 50.0
}

Data seems (and is) unrealistic. Bear with that. It has been crafted specifically to generate an interesting discussion later on.

from typing import Tuple

from pyomo.environ import ConcreteModel, DataPortal, SolverFactor


# Load data from json file. Pyomo admits loading data in  
# many formats. Here are more examples
# https://pyomo.readthedocs.io/en/stable/working_abstractmodels/data/dataportals.html
data = DataPortal(filename='data.json')

# Get an instance of the problem with data.
instance: ConcreteModel = model.create_instance(
    name='Small container',
    data=data,
)

# Get the SCIP solver
solver = SolverFactory('scip')

# Finally, solve the problem!
solution = solver.solve(
    instance,
    tee=True,  # Log the progress of the execution of the Solver
)

Notice that the separation of AbstractModel and ConcreteModel is really convenient. We can define just one AbstractModel, but solve many instances ConcreteModel instances with different data each.

Report results

from pyomo.environ import value


profits = value(instance.objective_function)
print(f'Profits are {profits}€')

for product in instance.products:
    number_of_items = round(instance.number_of_items[product].value)
    volume_of_an_item = instance.volume[product]
    volume_of_items = volume_of_an_item * number_of_items
    print(
        f'Put {number_of_items} units of {product!r}. '
        f'This occupies {volume_of_items} cubic meters'
    )

Notice that we read results from the ConcreteModelinstance. Try to execute profits = value(model.objective_function) and Pyomo will warn you that model does not hold any data, so no value can be computed.

The output of the Solver

We'll use the same environment we installed in the previous article.

Run this file. The solver shows logs of the whole process to help us understand the complexity of the problem and the execution process. For now, let's focus on the solution of the problem. In subsequent articles we'll understand logs in detail.

Summary of the problem

Near the beginning there's a summary of our problem.

original problem has 4 variables (0 bin, 4 int, 0 impl, 0 cont) and 1 constraints

The problem has 4 integer variables, one for each product, and just one constraint.

For completeness, bin refers to binary (as we saw with the Rooks Problem), cont refers to continuous and impl refers to implications (haven't seen any examples yet).

The optimal solution

Near the end there's the solver veredict. It found 12 solutions in 0.00 seconds. One of them is the optimal solution.

SCIP Status        : problem is solved [optimal solution found]
Solving Time (sec) : 0.00
<line omited>
Primal Bound       : +1.41000000000000e+03 (12 solutions)
<lines omited>
Profits are 1410.0€
Load 0 unit of 'toys'. This occupies 0.0 cubic meters
Load 6 unit of 'treadmill'. This occupies 20.4 cubic meters
Load 1 unit of 'locker'. This occupies 7.2 cubic meters
Load 16 unit of 'costumes'. This occupies 22.4 cubic meters

Few things to consider:

  • It found 12 solutions. This means, 12 valid item arrangements in the truck. Finally, the solver asserts that (at least) one of them is the optimal solution to the Problem.

  • Notice that we don't reach the stock limit of any product. Also, the volume of all items that we ship (50) does not exceed the volume of the truck (50). The Solver did a great job at filling every corner of the truck.

  • The locker is the product with higher profits per item. However, we're not shipping all units that we can, but just one.

    💡
    Can you think why this happens?
    💡
    Try adding a constraint so that the number of lockers to load into the truck is two. The new solution won't as good, meaning that the profits will be lower or equal!

Debugging our implementation

With larger and more complex AbstractModel implementations we may need to double check if we typed our Sets, parameters, variables and constraints are correctly. We can export a ConcreteModel into a text file. Use it often, it is life-saver.

# If the instance has been already solved, the dump also contains
# the values of the optimal solution.
with open('concrete_model_dump.txt', 'wt') as f:
    instance.pprint(f)

Here's the file

💡
Again, notice how helpful the descriptions in doc arguments and docstrings are!
1 Set Declarations
    products : Set of products to load into the truck.
        Size=1, Index=None, Ordered=Insertion
        Key  : Dimen : Domain : Size : Members
        None :     1 :    Any :    4 : {'toys', 'treadmill', 'locker', 'costumes'}

4 Param Declarations
    profits : Profits obtained for each item that we load into the truck. Units are €/item
        Size=4, Index=products, Domain=NonNegativeReals, Default=None, Mutable=False
        Key       : Value
         costumes :  40.0
           locker :   200
             toys :    12
        treadmill :  95.0
    stock : Total number of items requested of each product
        Size=4, Index=products, Domain=NonNegativeIntegers, Default=None, Mutable=False
        Key       : Value
         costumes :    20
           locker :     5
             toys :    15
        treadmill :    10
    truck_volume : The volume of the truck in cubic meters
        Size=1, Index=None, Domain=NonNegativeReals, Default=None, Mutable=False
        Key  : Value
        None :  50.0
    volume : Volume of each product in cubic meters
        Size=4, Index=products, Domain=NonNegativeReals, Default=None, Mutable=False
        Key       : Value
         costumes :   1.4
           locker :   7.2
             toys :   0.5
        treadmill :   3.4

1 Var Declarations
    number_of_items : Number of items to load into the truck
        Size=4, Index=products
        Key       : Lower : Value              : Upper : Fixed : Stale : Domain
         costumes :     0 :               16.0 :    20 : False : False : NonNegativeIntegers
           locker :     0 : 1.0000000000000053 :     5 : False : False : NonNegativeIntegers
             toys :     0 :                0.0 :    15 : False : False : NonNegativeIntegers
        treadmill :     0 :  5.999999999999989 :    10 : False : False : NonNegativeIntegers

1 Objective Declarations
    objective_function : Size=1, Index=None, Active=True
        Key  : Active : Sense    : Expression
        None :   True : maximize : 12*number_of_items[toys] + 95.0*number_of_items[treadmill] + 200*number_of_items[locker] + 40.0*number_of_items[costumes]

1 Constraint Declarations
    volume_of_items_cannot_exceed_truck_volume : The volume of items in the container cannot exceed truck volume.
        Size=1, Index=None, Active=True
        Key  : Lower : Body                                                                                                                     : Upper : Active
        None :  -Inf : 0.5*number_of_items[toys] + 3.4*number_of_items[treadmill] + 7.2*number_of_items[locker] + 1.4*number_of_items[costumes] :  50.0 :   True

8 Declarations: products stock profits volume truck_volume number_of_items volume_of_items_cannot_exceed_truck_volume objective_function

This dump also offers valuable information. For instance, notice that the column Body of constraint volume_of_items_cannot_exceed_truck_volume computes the volume of all the items that we load into the truck. We can use Pyomo to get the expression evaluated with the optimal solution.

from pyomo.environ import value


volume_of_all_items_loaded_into_the_truck = (
    value(instance.volume_of_items_cannot_exceed_truck_volume.body)
)

Attributions