Mario Francisco

In this work, deep reinforcement learning methodology takes advantage of transfer learning methodology to achieve a reasonable trade-off between environmental impact and operating costs in the activated sludge process of Wastewater treatment plants (WWTPs). WWTPs include complex nonlinear biological processes, high uncertainty, and climatic disturbances, among others. The dynamics of complex real processes are difficult to accurately approximate by mathematical models due to the complexity of the process itself. Consequently, model-based control can fail in practical application due to the mismatch between the mathematical model and the real process. Control based on the model-free reinforcement deep learning (RL) methodology emerges as an advantageous method to arrive at suboptimal solutions without the need for mathematical models of the real process. However, convergence of the RL method to a reasonable control for complex processes is data-intensive and time-consuming. For this reason, the RL method can use the transfer learning approach to cope with this inefficient and slow data-driven learning. In fact, the transfer learning method takes advantage of what has been learned so far so that the learning process to solve a new objective does not require so much data and time. The results demonstrate that cumulatively achieving conflicting objectives can efficiently be used to approach the control of complex real processes without relying on mathematical models.

This article presents a distributed model predictive control algorithm including fuzzy negotiation among subsystems and a dynamic setpoint generation method, applied to a simulated sewerage network. The methodology considers WWTP as an additional objective of control. To improve the performance of a DMPC using a hydraulic model for prediction, a more detailed model has been considered including suspended solids concentration (TSS). The results obtained with the proposed methodology have been validated on a benchmark simulation model for sewer systems developed to test and compare methodologies, showing good performance.

This paper proposes a novel solution for using deep neural networks with reinforcement learning as a valid option in negotiating distributed hierarchical controller agents. The proposed method is implemented in the upper layer of a hierarchical control architecture composed at its lowest levels by distributed control based on local models and negotiation processes with fuzzy logic. The advantage of the proposal is that it does not require the use of models in the negotiation, and it facilitates the minimization of any dynamic behavior index and the specification of constraints. Specifically, it uses a reinforcement learning policy gradient algorithm to achieve a consensus among the agents. The algorithm is successfully applied to a level system composed of eight interconnected tanks that are quite difficult to control due to their non-linear nature and the high interaction among their subsystems.