Instructor: Tim Chumley
Office: Clapp 404b
Phone: 538-2299
e-mail: tchumley
Office Hours: tentatively Mondays 10:00–11:00, Tuesdays 4:00–5:00, Wednesdays 3:00–4:00, Thursdays 9:00–10:00, and by appointment

Textbook: Introduction to Stochastic Processes with R by Robert P. Dobrow, ISBN: 9781118740651;
on library reserve under QC20.7.S8 D63 2016;
available as a free e-text


Announcements will be posted here throughout the semester.

April 29: Here are office hours for the coming week:

  • Monday, April 30: 10:00 – 11:00
  • Tuesday, May 1: 11:00 – 12:00, 4:00 – 5:00
  • Wednesday, May 2: 2:00 – 3:00

and by appointment. Please note that I’ll be away Thursday and Friday, but I’ll be able to reply to email, albeit with some delay.

April 28: The final exam was handed out in class and is posted here.

April 4: The next homework due date was pushed back to April 11 to give you some time to study for Friday’s exam. Also, we’ll cancel class on April 13 for Senior Symposium.

Feb 3: Some clarifying remarks about rewrites have been posted below in the Homework section.

Feb 2: Here is another modeling competition opportunity taking place at Amherst College April 21 called SCUDEM (in addition to the one I wrote to you about taking place next week).


Check the syllabus for all the important class policies (grades, attendance, etc.).


There will be weekly homework assignments throughout the semester to be turned in, as well as suggested problems to be considered as additional practice.

  • Written assignments. A selection of problems taken from the textbook or other sources will be assigned to be due Fridays at 7pm.
    • These assignments are given so that you can practice writing mathematics and receive feedback on your progress.
    • If you would like feedback on a particular steps in a problem, then you can indicate this on your assignment.
    • Each problem will be given a score out of 5. A score of 1 will be given to indicate that to stay on track you should get help from your instructor.
    • Please read these guidelines for writing up your homework solutions.
  • Submitting to Gradescope. We’ll be using an online submission tool called Gradescope to collect, grade, and return homework. Here are some notes:
    • Begin by signing up as a student on the Gradescope home page with Entry Code MWY6EK.
    • I’ve posted a short tutorial to help with Gradescope if you haven’t used it before.
    • Gradescope has also made a guide to help with scanning and submitting.
    • More information is available in this general help guide.
    • Please let me know of any issues that come up!
  • Suggested problems. The book has a good variety of problems, both in terms of content and difficulty level. As you study throughout the semester, you might consider browsing and trying the problems at the end of the chapter for extra practice. I’ll always be happy to help over email or in office hours if you’re stuck.
  • Rewrites. You’ll be allowed to redo some missed homework problems this semester. Here are some details:
    • You can rewrite one problem per problem set and earn back full points.
    • You should include with your rewrite an explanation that makes it easy to see what has changed in your answer (ie. what went wrong originally and what you’ve done to fix it).
    • You should turn in your rewrite in class on the Friday after the original due date. (Generally graded work will be released on Gradescope two or three days after the due date.)
Assignment Due
Problem set 0 (solutions) Jan 29
Problem set 1 (solutions) Feb 2
Problem set 2 (solutions) Feb 9
Problem set 3 (solutions) Feb 16
Problem set 4 (solutions) Feb 23
Problem set 5 (solutions) Mar 9
Problem set 6 (solutions) Mar 23
Problem set 7 (solutions) Mar 30
Problem set 8 (solutions) Apr 11
1-2 page project topic summary Apr 16
Problem set 9 (solutions) Apr 20
Problem set 10 (solutions) Apr 27


There will be three exams. The dates for the two mid-terms are subject to change slightly:

Exam Date Time and Location Material Study material Solutions
Exam 1 Mar 2 in class, take-home 1.1 – 3.6 study guide (solutions) in-class, take-home
Exam 2 Apr 6 in class, take-home 3.8, 4.1 – 4.4, 5.1, 5.2 study guide (solutions) in-class, take-home
Final May 3 - 7 take-home Ch. 6, 7

Course plan

Our plan is to cover the textbook chapters 1 – 3, and parts of 4 – 7. Below is a rough outline of what is to be covered week by week through the semester. Please check back regularly for precise details on what is covered, as well as postings of in-class worksheets and other activities. Also note that the days of the week under Sections in the table below provide links to class notes.

Week Sections In-class activities Problems
Jan 22 - Jan 26 Wednesday 1.1, 1.2, 2.1
Friday: 2.2
First day info, State spaces, index sets
Markov transition matrix, graphs
1.1, 1.2, 1.3, 1.4, 1.9, 1.35
2.8, 2.9, 2.10
Jan 29 - Feb 2 Monday: 2.3, 2.4
Wednesday: 2.3
Friday: 2.5
n-step distribution (solutions)
Simulation, RMarkdown
More simulation
2.1, 2.2, 2.6, 2.25, 2.26
2.4, 2.5, 2.11, 2.17
2.23, 2.24, 2.27
Feb 5 - Feb 9 Monday: 3.1, 3.2
Wednesday: snow day
Friday: 3.2, 3.3
Stationary distribution

Communication classes
2.18, 3.1, 3.2, 3.5, 3.8, 3.10

3.13, 3.14a, 3.63
Feb 12 - Feb 16 Monday: 3.3
Wednesday: 3.3
Friday: 3.4
Canonical decomposition
3.28, 3.29
3.17, 3.22
3.14, 3.18, 3.34
Feb 19 - Feb 23 Monday: 3.5
Wednesday: 3.6, 3.8
Friday: 3.8

Ergodic chains

3.50, 3.51
Feb 26 - Mar 2 Monday: 3.8
Wednesday: review
Friday: exam
Absorbing chains 3.54
Mar 5 - Mar 9 Monday: 5.1
Wednesday: snow day
Friday: 5.2
TSP worksheet, TSP tutorial, RMarkdown

Crytography, R demo
5.1, 5.3
Mar 12 - Mar 16 spring break
Mar 19 - Mar 23 Monday: 4.1, 4.2
Wednesday: 4.3
Friday: 4.4
Extinction in subcritical case
Probability generating functions
4.1, 4.2, 4.3, 4.4, 4.9
4.7, 4.12, 4.13
4.14, 4.16, 4.19
Mar 26 - Mar 30 Monday: 6.1
Wednesday: 6.2
Friday: 6.2
Poisson process introduction
Arrival times
6.1, 6.3, 6.4, 6.6
6.2, 6.7
Apr 2 - Apr 6 Monday: 6.4
Wednesday: review
Friday: exam
Thinning and superposition
Review MCMC solution
6.8, 6.13, 6.15, 6.16
Apr 9 - Apr 13 Monday: 7.1, 7.2
Wednesday: 7.3
Friday: no class
Transition rates, holding times, embedded chain 7.1, 7.2
7.4, 7.5, 7.6
Apr 16 - Apr 20 Monday: 7.3
Wednesday: 7.4
Friday: 7.6
Infinitesimal generator
Limiting and stationary distributions
7.7, 7.14, 7.23
7.11, 7.13
7.19, 7.25, 7.26
Apr 23 - Apr 27 Monday: 7.6
Wednesday: presentations
Friday: presentations
Queueing theory (solution) 7.20, 7.22, 7.24, 7.30, 7.33
Apr 30 - May 4 Monday: presentations


We’ll devote the last week of the semester to a mini-symposium of short group presentations. Since the field of stochastic processes is rich with interesting examples and topics, more than we could cover in a single semester, each group of 2-3 students will choose a topic/application that we might otherwise not have time for in class.
We’ll also plan to have a writing component to the project and some preliminary deliverables in the lead up to the final week. More details will be discussed in class later in the semester.

  • Presentation information and suggested topics
  • Upcoming dates:
    • Friday, March 30: select two potential topics and turn in 1-2 paragraphs about each. Give a synopsis of each topic and why you might like to present on it.
    • Monday, April 16: submit 1-2 page (single spaced, written or typed) summary of the topic (one submission per group).
    • week of Monday, April 16: meet with me with a draft of slides and/or plan for the talk.
    • April 25, 27, 30: presentation days in class.
    • Monday, May 7: submit 3-5 page report on the topic.
  • Presentation rubric
Group members Topic Presentation day Slides
Nicole, Sally, Younghoo Moran process Wednesday, April 25 pdf
Jessica, Sunan, Weijia Optimal stopping Wednesday, April 25 pdf
Nhu, Uyen Martingales, betting strategies Friday, April 27 pdf
Emma, Juliana, Shirley Card shuffling Friday, April 27 pdf
Faryal, Mirha, Momal Birth and death processes Monday, April 30 pdf
Ranjini, Rebecca, Surabhi Hidden Markov model Monday, April 30 pdf
Puyang An application of MCMC to ecology Monday, April 30 pdf

LaTeX resources

LaTeX is a typesetting system that makes it easy to type math. Here are some resources you might find useful as you type reports or make presentation slides.

  • MacTeX/TeXLive. An installation package for getting LaTeX running on your computer. MacTeX is for macOS and TeXLive is for other operating systems.

  • LaTeXiT. This small program for macOS lets you quickly make a mathematical expression using LaTeX commands and export it as an image or pdf file that can be inserted into a PowerPoint presentation. This is an alternative to making a full LaTeX document; it doesn’t require much time investment aside from first installing MacTeX or TeXLive. (LaTeXiT comes with a full MacTeX/TeXLive installation.)

  • quick reference. This file has a list of common commands.

  • LaTeX file template. This is a simple file that lets you get started making a full LaTeX document. This is the pdf file generated by the template.

  • Beamer template. If you’d like to use LaTeX to make presentation slides, there is a package called Beamer that is an alternative to systems like PowerPoint. Here is a template for a Beamer presentation. This is the pdf file generated by the template.

Getting help

Here’s a few ways to get help:

  • Office hours (listed at the top of this page). These are times I hold on my schedule to be available for students. I like to use these times as open drop in sessions where there may be multiple students getting help simultaneously, but it’s expected that you come with specific questions about notes, homework problems, or the text that you have thought about already. If you need to schedule a time to meet with me one-on-one, either to discuss something private or because you can’t make my office hours, please send me an email with times that you are available to meet, and I will find a time that works for both of us.
  • Study groups. I strongly encourage you to form small groups and work together on problems. My advice though is to try problems by yourself before discussing them. We have several spaces on the fourth floor of Clapp where students can work together for extended periods of time.


  • Everyone is invited to join DataCamp, which provides an introductory R tutorial. It’s a convenient way to gain some familiarity with R, a useful tool for our course and beyond. Our textbook also provides a thorough tutorial of some R basics in the appendix.

  • Our textbook also has a useful collection of R scripts available here; contained there are all the R code snippets you’ll notice interspersed in the text.