How I (finally) did my area exam

0. Picking a topic area

I have “decided” to do my area exam many times before now. Area exams are easy to postpone, because as PhD students it’s easy to feel like there’s always something more urgent to be working on. Over the years I have repeatedly fallen in to the trap of thinking of myself as a code- and paper-writing machine (which means no papers === I’m a complete failure! This machine is broken! Now I need to write twice as fast to make up for my humiliating failure! Etc.).

But besides the institutional requirement to get it done, the main thing giving me forward momentum while working on my exam this time is having a clear idea of what I want to discuss. The area exam talk is about 40 minutes, which is both intimidatingly longer than any talk I’ve ever given and shorter than I expected. You’re not going to be able to sum up, say, all of neurosymbolic programming or planning for physical HRI in 40 minutes, and trying to do so will both waste your time and give you unnecessary anxiety. My topic is something that I have at least started some research work in, so I finally feel I have an okay grasp on what’s relevant and what’s not.

My topic is AI for the facilitation of task handover, which is (1) honestly still too broad for a 40-minute talk, but I did my best and refined as I went, and (2) an interesting choice because AI is not currently used to facilitate task handover (at least, not by anyone but me). The goal of the area exam is to demonstrate your understanding of the state of the art in your chosen area. Because my area is an application domain rather than a technique, and a currently-unrealized application domain at that, my talk will aim to look at the state of the various component technologies that would be applied to this problem space. This (in theory) lets me look at huge areas like LLM summarization and pick out the current papers that are most relevant to my desired application.

1. Deciding to do the area exam, and setting a timeline

If you have completed step 0, this should really only be the logistics step. Talk to your advisor to finalize your topic and, with their help, figure out 2 other faculty members that care about that topic and ask them to be on your committee. (This request does not have to be a big to-do – a fairly short email with a description of your topic and a brief explanation of why you think they would be good to have on your committee given their expertise in XYZ worked just fine for me).

Schedule yourself a solid ~3 weeks where you can focus mostly or completely on the exam, then send out a when2meet or equivalent to your commitee for some 1-2 week period after that. Once you’ve confirmed everyone’s availability, send out that calendar invite, let your labmates know, and book a conference room (at CU you can do this at CU EMS, or by contacting the CS front office directly to book a room in ECES or DLC). I’d strongly recommend also scheduling a practice talk a few days beforehand and inviting labmates and/or friends.

In terms of a progress timeline, your major milestones should probably be something like:

a. Finalizing a paper list with your advisor: 2 weeks before exam b. Sending your advisor your full slide deck for feedback: 1 week before exam c. Giving a practice talk to your lab to get more feedback: 2+ days before exam

If at all possible, schedule multiple meetings with your advisor ahead of time for discussion and feedback, especially for steps (a) and (b).

2. Making your outline

I had already done a fair amount of reading on all the component areas I wanted to touch on, but didn’t know exactly what papers I would be discussion or what structure I would use. After much consideration and scrapped attempts, I ended up taxonomizing the component parts of the task handover problem this way:

a. forming a state estimate from input data b. acquiring a listener model for producing model updates c. performing model updates given a difference in mental models

My advisor initially recommended I pick 2 relevant papers for each of these problem spaces, so 6 papers total. Note that this is a pretty uncommonly small number of papers; in most prelim exams I typically see more in the 8-12 range, sometimes more. Again, my situation is somewhat unique in that I’m using a potential application area as a lens to discuss relevant findings. That means that each paper needs to be dicussed in a fair bit of depth, because I need to spend more time connecting it back to the topic at hand.

Lastly, I came up with a few broad thesis statements that I would use to frame each of my papers:

1. Recent advances in related areas of AI can be applied to task handover​
2. LLMs have made these advances more applicable to real-world communication scenarios

These are hardly the most controversial or specific theses, but having them was helpful for me in motivating my narrative and I’d highly recommend it. You can fall back on your taxonomy and have something as simple as “each of these N approaches has its own strengths” or “X new technology has enabled us to take this field in Y direction” or “Z is an ongoing problem that remains unsolved in this area, and here’s what we know about it / how we have progressed towards solving it.”

3. Choosing your papers

At our next meeting my advisor told me to focus my 6-paper main lit review on the first two areas, and leave the last section – “performing model updates” – as an abbreviated sort of current-work teaser at the end. This gave me room to discuss 2 more papers in the “listener model acquisition” section.

For each paper, it’s really important that you have a clear idea of what that paper’s contributions are, and why your listeners should care about it. There are also other aspects to consider.

• While you’re definitely going to learn new things during this process and step outside of your comfort zone somewhat, you also don’t want to be choosing papers that are wildly outside your area of expertise. Remember the goal here is to show off the expertise you’ve developed in your PhD so far – leverage what you already know about as much as possible.
• Relatedly, you might have already done some research work in your topic area, so use this as an opportunity to highlight what prior work has been most relevant to you as a scientist in this area.
• Avoid including papers that are more than a few years old, unless you have good reason (e.g. they were particularly impactful, laid the foundation for future papers you’re also going to discuss, etc.). AI moves very fast, and even papers from 2-3 years ago can become quite outdated. Try looking at works that have cited a relevant paper instead to see what’s been done more recently.
• Avoid including papers that your committee members authored. This is tempting if they’ve done relevant work, but you’re setting yourself up for a potentially uncomfortable Q&A session.

For an example, the final paper list my advisor approved looked like this:

1. Obtaining a model of state and environment (i.e., task-relevant state) (n.b.: don’t focus on environment dynamics)
   • Zhang et al. Hierarchical Cross-Modality Semantic Correlation Learning Model for Multimodal Summarization - a recent-ish (2022) and well-cited paper that had a lot of success on summarization metrics by using semantic correlation learning to combine image and textual data into a single representation and text output. This paper showcases one of many strategies that can be used to extract meaning from multimodal input, making AI better able to recover task relevant information from the world around it. Key point: We can extract meaningful from multiple modalities (for developing a state estimation)
   • Das, Chernova & Kim, NeurIPS 2023. State2Explanation: Concept-based explanations to benefit agent learning and user understanding. Cited by 24. Learns an embedding model between agent states and associated concept-based explanations and leverages such learned model to (1) inform reward shaping for agent learning, and (2) improve end-user understanding of the agent’s decision making at deployment. Key point: We can create an explanation-grounded internal model that can then be used to generate future explanations
2. Obtaining a model of the receiver (i.e. the human model) – giving a model update requires knowing what model you’re updating common thread: what does this paper mean in the age of LLMs? What’s the lesson we can transfer into an LLM-powered method?
   • Zhang & Soh, Large Language models as zero-shot human models for HRI. https://ieeexplore.ieee.org/abstract/document/10341488 Key point: In some situations, we may not need a bespoke human model
   • Salomons, Akdere & Scassellati. IJCAI 2021. BKT-POMDP: Fast Action Selection for User Skill Modeling over Tasks with Multiple Skills.
   • Yang et al., Machine Learning and Knowledge Discovery in Databases 2020. GIKT: A Graph-based interaction model for knowledge tracing. https://link.springer.com/chapter/10.1007/978-3-030-67658-2_18 Cons: IDK this conference. Pros: 150+ citations so clearly it was impactful. Contrasts with Nicole’s paper above by using a graph convolutional network for knowledge tracing, showcasing a different technical approach that might be more appropriate in some scenarios
   • Chen, Fong & Soh, MIRROR: Differentiable Deep Social Projection for assistive HRC https://arxiv.org/pdf/2203.02877 Key point: Rather than knowledge tracing, this system learns observational & policy residuals in advance, then uses that to predict what human is and is not aware of and what their tendencies are
3. Delivering model updates: Conclusion, touching on my current work & a few related works (very briefly)
   • Chakraboti et al, 2017. Plan Explanations as model reconciliation: Moving beyond explanation as soliloquy. Key point: proposing computational approaches to various explanation types, given a human model
   • Tabrez, SPEAR (policy elicitation). SPEAR uses the goal of policy elicitation to determine how best to update the user’s reward function via Boolean algebra and an MIL to optimize for coverage and conciseness. Key point: Builds on the above and showcases a particular approach to model reconciliation via policy elicitation

This took a non-neglible amount of back-and-forth with my advisor to settle on, so talk with them early & often in this process for best results!

4. Making your slides

I am by no means a genius at PowerPoint creation, so your mileage may vary, but here’s what worked for me.

• Start with a sketch of your presentation’s structure. I made a new PowerPoint and literally just wrote titles for each slide to get a sense of what I was going to talk about and where, and put things like “[illustrative figure goes here]” for my future self to figure out.
  • Intro: where you give the necessary background on your research area and why people should care about it.
  • Thesis: Some kind of takeaway from all your papers and why you’re talking about them. What do you want your listeners to be convinced of by the end of your talk? If they remember 1-2 things, what should they be?
  • Roadmap: Introduce the taxonomy you came up with, if you have one, or at least some way of grouping your papers to give your talk structure.
  • Lit review: Go section by section from your outline, and then paper by paper. For each one, you won’t have time to give a full in-depth overview of what the authors did – instead you’ll have only a few slides (maybe only 1, depending on how many papers you have!) to say very specifically what’s unique about the paper and how it relates to your thesis. At the end of each section, I recommend having a quick recap slide to summarize the key points from the papers you just discussed.
  • Conclusion: This is where you can briefly talk about your current work, where you think the field is going & how you want to contribute to it. Where do you fit in in the story you’ve just told?

• Build out your intro (and maybe even conclusion) next_, because it will force you to get a very solid idea of the perspective you’re taking on each of your papers. (I actually did slides for my papers first, then went back after I wrote my intro and realized I wanted to rework literally all of them. Don’t be like me.) This might be where you have to make some figures or illustrations as well, to convey your problem space and outline visually. In your conclusion you might also get to talk about your own past or present work.

• Build out your paper slides: Honestly the easiest part, but some things to keep in mind:
  • Include citation information where appropriate, including for images.
  • Pictures are worth a thousand words – if the paper has decent figures, use them. But–
  • If you’re going to use a figure from the paper, it should either be completely self-explanatory (as in, very simple / just a photograph / etc.) OR you should take an appropriate amount of time to contextualize it or reference it directly in your talk.
  • Same goes for any math, data tables, etc.: if it’s on your slide, you should speak to it. (That’s a Brad Hayes-ism, for anyone curious.) Don’t use them just for set dressing!
  • It’s okay if your slides are very text-heavy on the first draft, but eventually you’ll want to minimize on-slide text – you want people listening to you, not reading. Optimize for clarity, spread info across multiple slides if needed.
  • Similarly, complicated figures like system diagrams or tables can be overwhelming and/or distracting for your audience, even if you explain them. Include only the most relevant figures/parts of figures, and consider doing ‘progressive reveals’ where you un-hide parts of the figure/text as you discuss it to help listeners follow along. This was suggested to me by a labmate at my practice talk, and now I’m a big progressive reveal fan and I think everyone else should be too.
  • The key takeaway from each paper should be written out, super explicitly, on the slide – listeners should be 100% clear on why you care about that paper and they should too.
• Check your clarity & time. Are you going to be able to get through all those slides in 40-45 minutes, without rushing through? As I wrote my presentation, I timed myself doing a quick-and-dirty run through of each section to get a sense of where I was at. (For example, my initial pass at my intro took me about 15 minutes(!), so obviously I had some trimming to do.) This is also where I condensed text where possible, and moved a lot of text off my slides and into my presentation notes.

Once you have a rough draft, it’s time to send the slides to your advisor for feedback & approval.

5. Prepping & practicing your talk

I relied heavily on the Presenter View feature in PowerPoint (also in Google Slides) so that I could reference my notes on my laptop while I was speaking. I tried to limit these to bullet pointed almost-sentences that still covered everything I wanted to say at each slide. Still, you don’t want to be reading off your laptop the whole talk. It took me a few practice runs to get the talk to a half-memorized state, where I could take quick glances at each bullet point to know what to say next.

(I still definitely read off my screen excessively because my memory fails me under pressure and I dislike eye contact. It’s important to know your weaknesses.)

The practice talk I gave for my labmates was incredibly helpful: for the practice, for the feedback they gave, and for the validation of something I’d been working on for weeks. Even though I was nervous to give it, the confidence boost it gave me going into my actual exam was extremely worth it. They also asked some great general questions that made me feel more prepared for the Q&A.

6. Giving your talk

At this point, your advisor has ok’d your papers and your slides, your labmates have ok’d your talk, you’ve practiced – your success is more or less guaranteed. Try not to be nervous! Speak clearly and confidently. Give you (and your audience) time to breathe; have some water nearby and take a sip between slides as needed.

For the closed Q&A session, my committee mercifully did not grill me on any technical details. Their questions were really testing my ability to articulate my opinions and decision-making: about what I included in my talk, what is or is not exciting about a particular paper, what the biggest challenges are, and so on. I came out of it feeling pretty good, which was a pleasant surprise.

7. The aftermath

I passed! You can view the slides from my talk here. If you’re a CU affiliate you can view a recording of it here.

I’m so grateful to have gotten through this process with my pride mostly intact. I’m also very thankful to my committee for their patience with me – it took me a long time and a lot of iteration to get the presentation to a state we all were happy with. I wrote this post in hopes that it would help make the process a bit less intimidating for future students. Good luck!