Stochastic Methods

This course covers foundational and advanced topics in stochastic processes, focusing on both theoretical principles and practical applications. Students learn to model randomness in systems, analyze probabilistic phenomena, and apply stochastic methods in various domains, including optimization, inference, networks, and simulation.

Course Overview

The course covers a broad range of stochastic techniques, structured into ten main topics, each with corresponding lecture notes:

1. Randomness

📄 Lecture 1: Randomness

• Random number generators (RNGs) and their properties
• Pseudorandom number generation: Linear Congruential Generators (LCG), PCG64
• Probability distributions: uniform, discrete, continuous
• Inverse transform sampling

2. Random Variables

📄 Lecture 2: Random Variables

• Discrete vs. continuous random variables
• Probability mass functions (PMF) and probability density functions (PDF)
• Cumulative distribution functions (CDF) and their properties
• Binomial, Poisson, and normal distributions

3. Expectation & Limit Theorems

📄 Lecture 3: Expectation

• Definition of expectation and its properties
• Linearity of expectation
• The Law of Large Numbers (LLN)
• The Central Limit Theorem (CLT) and its implications
• Monte Carlo simulation for numerical approximations

4. Variance & Monte Carlo Methods

📄 Lecture 4: Variance

• Monte Carlo integration and importance sampling
• Variance reduction techniques (stratified sampling, control variates, antithetic variates)
• Rejection sampling and proof of correctness
• Dependence and independence of random variables
• Bayesian inference and Bayes’ theorem

5. Networks & Random Graphs

📄 Lecture 5: Networks

• Introduction to random networks
• Graph theory: adjacency matrices and connectivity
• Erdős–Rényi random graphs
• Structural Equation Models (SEM) and causality in networks
• Applications in epidemiology and network analysis

6. Markov Processes

📄 Lecture 6: Markov Processes

• Markov chains: transition matrices, Chapman-Kolmogorov equation
• Random walks and applications in financial models
• Stationary distributions and ergodicity
• Multi-step transition probabilities and long-term behavior
• Applications in epidemiology and finance

7. Stochastic Simulation

📄 Lecture 7: Stochastic Simulation

• Stochastic Cellular Automata and probabilistic grid-based models
• Agent-Based Modeling (ABM) and emergent behavior
• Example: Forest fire model (spread of wildfire simulation)
• Example: Ant foraging simulation (pheromone-based search strategies)

8. Stochastic Inference

📄 Lecture 8: Stochastic Inference

• Linear Regression and least squares estimation
• Logistic Regression and Maximum Likelihood Estimation (MLE)
• Support Vector Machines (SVMs) for classification
• Neural networks and stochastic optimization methods

9. Stochastic Optimization

📄 Lecture 9: Stochastic Optimization

• Evolutionary algorithms: Genetic algorithms (GA) & Differential Evolution (DE)
• Stochastic Gradient Descent (SGD) and its variants
• ADAM optimizer: adaptive learning rates for optimization
• Expectation-Maximization (EM) algorithm and Monte Carlo EM
• Markov Chain Monte Carlo (MCMC) methods

10. Stochastic Systems & Multi-Agent Models

📄 Lecture 10: Stochastic Systems

• Multi-agent stochastic systems: Defining agents, tasks, tools, and processes
• Markov Decision Processes (MDP)
• Process optimization: sequential, parallel, hierarchical, and event-driven systems
• Randomness in AI-generated outputs and bias in language models

Key Learning Objectives

1. Master Probability & Stochastic Concepts: Develop a strong foundation in probability theory, random variables, and stochastic modeling.
2. Analyze Markov Processes & Networks: Construct and analyze Markov Chains and random networks, understanding long-term behaviors.
3. Simulate & Optimize Stochastic Systems: Apply Monte Carlo methods, agent-based models, and stochastic gradient descent for practical applications.
4. Explore Stochastic Optimization: Learn techniques like evolutionary algorithms, SGD, and MCMC for solving complex problems.
5. Work with Multi-Agent Stochastic Models: Understand stochastic decision-making, AI-generated randomness, and process optimization.

References

• Introduction to Probability Models by Sheldon M. Ross
• Stochastic Processes by Sheldon M. Ross
• Probability Models for Computer Science by Sheldon M. Ross
• First-order and Stochastic Optimization Methods for Machine Learning by Guanghui Lan
• Basic Stochastic Processes by Zdzisław Brzeźniak and Tomasz Zastawniak