Journals

1. Q-learning for Markov decision processes with a satisfiability criterion (Systems and Control Letters, 2018)
2. Revisiting random walk based sampling in networks: evasion of burn-in period and frequent regenerations (Computational Social Networks, 2018)
3. Controlled equilibrium selection in stochastically perturbed dynamics (Annals of Probability, 2018)
4. Gradient estimation with simultaneous perturbation and compressive sensing (Journal of Machine Learning Research, 2018)
5. Opportunistic scheduling as restless bandits (IEEE Transactions on Control of Network Systems, 2018)
6. Metastability in stochastic replicator dynamics (Dynamic Games and Applications, 2018)
7. A Concentration Bound for Stochastic Approximation via Alekseev's Formula (Stochastic Systems, 2019)
8. Empirical Q-value iteration (Stochastic Systems, 2021)
9. Distributed stochastic approximation with local projections (SIAM Journal on Optimization, 2018)
10. Linear programming formulation of long run average optimal control problem (Journal of Optimization Theory and Applications, 2019)
11. Whittle indexability in egalitarian process sharing systems (Annals of Operations Research, 2019)
12. Aerial monitoring of slow moving convoys using elliptical orbits (European Journal of Control, 2019)
13. Reinforcement learning, sequential Monte Carlo and the EM algorithm (Sadhana, 2018)
14. Low complexity online radio access technology selection algorithm for LTE-WiFi HETNET (IEEE Transactions on Mobile Computing, 2019)
15. Stochastic approximation algorithms for rumor source inference on graphs (Performance Evaluation, 2019)
16. Distributed sever allocation for content delivery networks (Queueing Models and Service Management, 2019)
17. LP Formulations of Discrete Time Long-Run Average Optimal Control Problems: the Nonergodic Case (SIAM Journal on Control and Optimization, 2019)
18. Non-asymptotic error bounds for constant stepsize stochastic approximation for tracking mobile agents (Mathematics of Control, Signals and Systems, 2019)
19. On the fastest finite Markov processes (Journal of Mathematical Analysis and Applications, 2020)
20. A variational characterization of the optimal exit rate for controlled diffusions (Theory of Probability and Mathematical Statistics, 2020)
21. Simultaneous small noise limit for singularly perturbed slow-fast coupled diffusions (Applied Mathematics and Optimization, 2021)
22. Scheduling in wireless networks with spatial reuse of spectrum as restless bandits (Performance Evaluation, 2021)
23. Online reinforcement learning of optimal threshold policies for Markov decision processes (IEEE Transactions on Automatic Control, 2022)
24. Prospect-theoretic Q-learning (Systems and Control Letters, 2021)
25. A concentration bound for contractive stochastic approximation (Systems and Control Letters, 2021)
26. Opinion shaping in social networks using reinforcement learning (IEEE Trans. Control of Network Systems, 2022)
27. Dynamic social learning under graph constraints (IEEE Trans. Control of Network Systems, 2022)
28. Whittle index based Q-learning for restless bandits with average reward (Automatica, 2022)
29. Revisiting SIR in the age of COVID-19: explicit solutions and control problems (SIAM Journal on Control and Optimization, 2022)
30. Concentration of contractive stochastic approximation and reinforcement learning (to appear in Stochastic Systems, 2022)
31. (S. K. Singh, V. S. Borkar and G. S. Kasbekar) “User association in dense mmwave networks as restless bandits”, IEEE Transactions on Vehicular Technology, 71(7), 2022, 7919-7929..

Book Chapters

1. (V. S. Borkar, J. A. Filar) “Postponing collapse : ergodic control with a probabilistic constraint”, Modeling, Stochastic Control, Optimization, and Applications (G. George Yin, Q. Zhang, eds.), IMA Volume in Mathematics and Its Applications No. 164, Springer Nature, Cham, Switz., 2019., Springer, 2013.
2. (A. Arapostathis, V. S. Borkar) “ “Controlled” versions of the Collatz- Wielandt and Donsker-Varadhan formulae”, In: Applied Probability and Stochastic Processes (Joshua V., Varadhan S., Vishnevsky V. (eds)), Infosys Science Foundation Series, Springer, Singapore, 2020.
3. (A. Arapostathis, V. S. Borkar) “On the relative value iteration with a risk-sensitive criterion”, Proc. of Stochastic Control and Modeling, Simons Semester No. 10, Banach Centre Publications No. 122, Polish Academy of Sciences, Warsaw, 2020..
4. (K. Avrachenkov, V. S. Borkar, H. P. Dolhare and K. Patil) “Full gradient DQN reinforcement learning: a provably convergent scheme”, In: Modern Trends in Controlled Stochastic Processes (pp. 192-220). Springer, Cham, 2021.

Popular Expository Articles

1. ‘Markov Chain Monte Carlo (MCMC): A short overvieq’ + ‘The birth of MCMC’, Resonance, 27(6), 2022, 1107-1115 + 1105-1106.

Books

1. 1. Stochastic Approximation: A Dynamical Systems Viewpoint (second enlarged edition), Hindustan Publ. Agency, New Delhi, and Springer Nature, 2022

Conferences

1. (V. S. Borkar, K. Chadha) "A reinforcement learning algorithm for restless bandits", Proc. Indian Control Conference, Kanpur, Jan. 2018, 89-94.
2. (V. S. Borkar, S. M. Shah) "Distributed algorithms: Tsitsiklis and beyond", Workshop on Info. Theory and Appl., San Diego, Feb. 2018 (Invited talk).
3. (V. S. Borkar) "A variational formula for risk-sensitive control", Workshop on Stochastic Control, Computational Methods and Applications, Institute for Mathematics and Applications, Minneapolis, May 7-11, 2018 (Invited talk).
4. (V. S. Borkar) "Controlled Diffusion Processes", Institute for Mathematical Statistics Asia-Pacific Rim Meeting, Singapore, June 2018 (Distinguished lecture).
5. (V. S. Borkar) "Nonlinear filtering and averaging in two time scale systems", Annual Meeting of the American Institute of Mathematical Sciences, Taipei, July 2018 (Invited talk).
6. (V. S. Borkar, V. Dwaracherla, N. Sahasrabudhe) "Sparse regression using compressive sensing with input shaping", 23rd Intl. Symp. on Mathematical Theory of Networks and Systems, Hong Kong, July 16-20, 2018.
7. (V. S. Borkar) "Stochastic approximation revisited: some new wine in old bottle", Workshop in honour of Prof. Sean Meyn, 57th IEEE Conference on Decision and Control, Miami Beach, Florida, Dec. 16, 2018 (Invited talk).
8. (V. S. Borkar, K. Chadha) "A reinforcement learning algorithm for restless bandits", Workshop on Learning Theory, TIFR, Mumbai, Jan. 2-6, 2019 (Invited talk).
9. (A. Arapostathis, V. S. Borkar) "Relative value iteration for ergodic control", Workshop on Recent Problems of Stochastic Control Theory, Banach Centre, Warsaw, Jan. 28 - Feb. 2, 2019 (Invited talk).
10. (A. Roy, V. S. Borkar, A. Karandikar and P. Chaporkar) "A structure-aware online learning algorithm for Markov decision processes", VALUETOOLS 2019, Barcelona, March 2019.
11. (V. S. Borkar) "Small noise limits", National Conference on Stochastic Differential Equations and Applications, IIST, Thiruvananthapuram, June 6-7, 2019 (Invited talk).
12. (V. S. Borkar) "A representation theorem for risk-sensitive value", SIAM Conf. on Control and Its Applications, Chengdu, China, June 19-21, 2019 (Invited talk).
13. (V. S. Borkar) "Crawling towards ephemera", IEEE Bombay Section Signature Conference, IIT, Mumbai, July 2019 (Plenary talk).
14. (V. S. Borkar) "A representation theorem for risk-sensitive reward", Workshop on Advances in Applied Probability, Intl. Centre for Theoretical Sciences, Bengaluru, Aug. 2019 (Invited talk).
15. (K. Avrachenkov, V. S. Borkar) "A learning algorithm for the Whittle index policy for scheduling web crawlers", Proc. 57th Allerton Conference on Communication, Control, and Computing, Monticello, IL, 2019 (Invited talk).
16. (V. S. Borkar) "Stochastic approximation: an overview", Dagstuhl Seminar on Theory of Randomized Optimization Heuristics, Oct. 2019 (Invited talk).
17. (A. Arapostathis, V. S. Borkar) `Linear programs for risk-sensitive control', Workshop on Modern Trends in Controlled Stochastic Processes: Theory and Applications, Liverpool, UK, July 5-9, 2021.
18. (V. S. Borkar) Kab Q aur kahaan? (Variations on a theme of Watkins), Plenary Lecture at National Communications Conference, IIT Kanpur, July 29, 2021.
19. (V. S. Borkar) Stochastic approximation: Robbins-Monro and its variants, International Conference on Emerging trends in Statistics and Data Science in conjunction with 40th Annual Convention of Indian Society of Probability and Statistics, September 9, 2021 (Prof. G. Sankaranarayanan Endowment Lecture).
20. (Priyadarshini, K., S. Chaudhuri, V. S. Borkar and S. Chaudhuri), A Unified Batch Selection Policy for Active Metric Learning, In Joint European Conference on Machine Learning and Knowledge Discovery in Databases (pp. 599-616), Springer, Cham, 2021.
21. (V. S. Borkar) Dynamic social learning under graph constraints, Workshop on Deep Reinforcement Learning, IISc, Bengaluru, Oct. 22, 2021.
22. (V. S. Borkar, Syomantak Chaudhuri) Accelerating MCMC by rare intermittent resets, 14th EAI International Conference on Performance Evaluation Methodologies and Tools, Guangzhou, China, Oct. 31, 2021.
23. 'Dynamic choice under graph constraints', Workshop on Retrial Queues, Kottayam, Dec. 8, 2021.
24. (A. Arapostathis and V. S. Borkar) Linear and dynamic programs for risk-sensitive cost minimization, 60th IEEE Conf. on Decision and Control, Austin, Texas, Dec. 15, 2021.
25. (K. Avrachenkov, V. S. Borkar, K. Patil) Deep reinforcement learning for web crawling, In The Seventh Indian Control Conference, IIT Bombay, Mumbai, Dec. 22, 2021.
26. (V. S. Borkar, S. Chandak and P. Dodhia) Concentration for reinforcement learning and the 'curse of non-Markovianity', SIGMETRICS Workshop on Learning-based Control of Queues and Networks, IIT Bombay, June 6, 2022.
27. "In search of Markov controls", Symposium on Stochastic Control and Applications, Dept. of Mathematics, IISc, Bengaluru (invited talk), 25/7/2022.
28. "Some interesting dynamics motivated by reinforcement learning", 19th International Symposium on Dynamic Games, Porto, Portugal, 25/7/2022 (invited talk).
29. Dolhare, H. and Borkar, V., 2022, September. A Concentration Bound for Distributed Stochastic Approximation. In 2022 58th Annual Allerton Conference on Communication, Control, and Computing, Monticello, IL, Sept. 2022. (invited talk)