A reinforcement learning approach for individualizing erythropoietin dosages in hemodialysis patients

This paper presents a reinforcement learning (RL) approach for anemia management in patients undergoing chronic renal failure. Erythropoietin (EPO) is the treatment of choice for this kind of anemia but it is an expensive drug and with some dangerous side-effects that should be considered especially for patients who do not respond to the treatment. Therefore, an individualized treatment appears to be necessary. RL is a suitable approach to tackle this problem. Moreover, resulting policies are similar to medical protocols, and hence, they can easily be transferred to daily practice. A cohort of 64 patients are included in the study. An implementation of the Q-learning algorithm based on a state-aggregation table and another implementation using the multi-layer perceptron as a function approximator (Q-MLP) are compared with the protocols followed in the Nephrology Unit. The policy obtained by the Q-MLP approach outperforms the hospital policy in terms of the ratio of patients that are within the targeted range of hemoglobin (11.5-12.5g/dl) at the end of the analyzed period, since an increase of 25% is observed. It ensures an improvement in patients' quality-of-life and considerable economic savings for the health care system due to both the expensiveness of EPO treatment and the costs incurred by the health care system in order to alleviate problems related to EPO over-dosing. It should be pointed out that the approach presented here is completely general, and therefore, it can be applied to any problem of drug dosage optimization.

[1]  A. Barto,et al.  Improved Temporal Difference Methods with Linear Function Approximation , 2004 .

[2]  John N. Tsitsiklis,et al.  Neuro-Dynamic Programming , 1996, Encyclopedia of Machine Learning.

[3]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[4]  R. Bellazzi Drug delivery optimization through Bayesian networks: an application to erythropoietin therapy in uremic anemia. , 1993, Computers and biomedical research, an international journal.

[5]  Antonio J. Serrano,et al.  Dosage individualization of erythropoietin using a profile-dependent support vector regression , 2003, IEEE Transactions on Biomedical Engineering.

[6]  Antonio J. Serrano,et al.  Use of neural networks for dosage individualisation of erythropoietin in patients with secondary anemia to chronic renal failure , 2003, Comput. Biol. Medicine.

[7]  P. Werbos,et al.  Beyond Regression : "New Tools for Prediction and Analysis in the Behavioral Sciences , 1974 .

[8]  E. S. Olivas,et al.  Use of neural networks for dosage individualisation of erythropoietin in patients with secondary anemia to chronic renal failure. , 2003, Computers in biology and medicine.

[9]  David P Steensma,et al.  Phase III study of two different dosing schedules of erythropoietin in anemic patients with cancer. , 2006, Journal of clinical oncology : official journal of the American Society of Clinical Oncology.

[10]  Jacek M. Zurada,et al.  Individualization of pharmacological anemia management using reinforcement learning , 2005, Neural Networks.

[11]  R. Bellazzi,et al.  Mathematical modeling of erythropoietin therapy in uremic anemia. Does it improve cost-effectiveness? , 1994, Haematologica.