Trends in Deep Learning Seminar at UBC

The University of British Columbia

Year: Winter Session II 2022

Time: Monday and Wednesday 1:00-2:30

Location: Zoom

Instructor: Dr. Muhammad Abdul-Mageed

Office location: Ling 242

Office phone: 6048-274-530

Office hours: By appointment

E-mail address: muhammad.mageed@ubc.ca

Student Portal: http://canvas.ubc.ca

---

1. Course Rationale & Goal:

Rationale/Background: Deep learning, a class of machine learning methods inspired by information processing in the human brain, has revolutionized the way we build machines, automate processes, analyze data, and just problem-solve in a fast-increasing host of domains. These transformational changes have disrupted whole industries, and are expected to impact wide sectors of society. Scholarship in the field has been growing very rapidly, with significant funding provided for both basic research and applications.

Goal: This is a graduate-level, directed resarch course aimed at bridging the gap between intermediate deep learning knowledge and novel and bleeding-edge deep learning research. The course will emphasize deep learning methods developed over the past 1-2 years. Areas will include the below:

• Graph Representation Learning
• Contrastive Learning
• Unsupervised Deep Learning for Speech

Potential audiences for this course are:

• This course is by invitation only.

---

2. Course Objectives:

Upon completion of this course students will be able to:

• identify a sub-area of deep learning research, engineer a solution, and write a high-quality research paper for a top venue.

• prepare and deliver a presentation based on a novel deep learning research area or paper.

• understand a range of deep learning topics at a research level theoritically, and develop related engineering solutions

3. Course Topics:

TBA (But see above). There will be some space to customize some select sessions based on student interests which might not be covered in areas above.

4. Prerequisites:

• Knwoledge of basic and intermediate deep learning methods (as demonstrated by course work or at least one publication).

5. Format of the course:

• This is not a regular course. Rather, this is a directed research/reading course that will operate in a seminar fashion where students and the instructor meet twice a week to read and discuss novel research papers. Some sessions will be focused on engineering. Presentations will rotate among students, with enough prior time for preparation. This course has no homework. Output will be a research paper by the end of the semester.

6. Course syllabus:

7. Calendar / Weekly schedule (tentative)

Note: Will be arranged further upon organizational meeting/discussion in class.

Format the below with new content

Time	Topic/Paper	Presenter	Slides
TBA	[TBA]	TBA	[slides]
TBA	[TBA]	TBA	[slides]
TBA	[TBA]	TBA	[slides]
TBA	[TBA]	TBA	[slides]
TBA	[TBA]	TBA	[slides]
TBA	[TBA]	TBA	[slides]

---

8. Readings (tentative):

Graph Representation Learning

• Book: Graph Representation Learning [pdf]

• List of Papers on Graph for NLP [Link]

• [Neural Sentiment Classification with User and Product Attention]

• [A Lexicon-Based Supervised Attention Model for Neural Sentiment Analysis]

• [Weakly Supervised One-Shot Detection with Attention Siamese Networks]

• [Graph-Attention Networks]

• [Abusive Language Detection with Graph Convolutional Networks]

• [Modeling Ideological Agenda Setting and Framing in Polarized Online Groups with Graph Neural Networks and Structured Sparsity]

• [You Shall Know a User by the Company It Keeps: Dynamic Representations for Social Media Users in NLP]

• [Overcoming Language Variation in Sentiment Analysis with Social Attention]

• [GPT-GNN: Generative Pre-Training of Graph Neural Networks]

• [Graph Contrastive Learning with Augmentations]

• [Hierarchical Heterogeneous Graph Representation Learning for Short Text Classification]

• [Iterative Deep Graph Learning for Graph Neural Networks: Better and Robust Node Embeddings]

---

Contrastive Learning

• [SimCSE: Simple Contrastive Learning of Sentence Embeddings]
• [Fast, Effective, and Self-Supervised: Transforming Masked Language Models into Universal Lexical and Sentence Encoders]
• [Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models]
• [Contrastive Learning for Many-to-many Multilingual Neural Machine Translation]
• [Contrastive Learning with Adversarial Perturbations for Conditional Text Generation]
• [Contrastive Learning for Context-aware Neural Machine Translation Using Coreference Information]
• [COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining]
• [Improving Gradient-based Adversarial Training for Text Classification by Contrastive Learning and Auto-Encoder]
• [Improved Text Classification via Contrastive Adversarial Training]

---

Automatic Speech Recognition

• [Voice Recognition Using MFCC Algorithm]
• [Deep Speech: Scaling up end-to-end speech recognition]
• [Attention-Based Models for Speech-Recognition]
• [Listen, Attend and Spell]
• [Deep Speech 2: End-to-End Speech Recognition in English and Mandarin]
• [Advances in Joint CTC-Attention based E2E ASR with a Deep CNN Encoder and RNN-LM]
• [Attention Is All You Need]
• [State-of-the-art Speech Recognition with Sequence-to-Sequence Models]
• [An Analsis Of Incorporating An External Language Model Into A Sequence-to-Sequence Model]
• [Speech-Transformer: A No-Recurrence Sequence-to-Sequence Model for Speech Recognition]
• [On the Choice of Modeling Unit for Sequence-to-Sequence Speech Recognition]
• [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition]
• [wav2vec: Unsupervised Pre-training for Speech Recognition]
• [Transformers with convolutional context for ASR]
• [Jasper: An End-to-End Convolutional Neural Acoustic Model]
• [End-to-end ASR: from Supervised to Semi-Supervised Learning with Modern Architectures]
• [SpecAugment on Large Scale Datasets]
• [ContextNet: Improving Convolutional Neural Networks for ASR with Global Context]
• [Conformer: Convolution-augmented Transformer for Speech Recognition]
• [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations]
• [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units]

---

Normalizing Flows

---

Efficient Deep Learning Models

• [Efficient Transformers: A Survey]
• [PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination]
• [Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Search]
• [Sparse is Enough in Scaling Transformers]
• [Scaling Laws for Neural Language Models]
• [Scaling Laws for Neural Machine Translation]
• [Data and Parameter Scaling Laws for Neural Machine Translation]
• A Survey on Dynamic Neural Networks for Natural Language Processing
• Neural Architecture Search: A Survey
• Primer: Searching for Efficient Transformers for Language Modeling
• Searching for Convolutions and a More Ambitious NAS
• Unified Scaling Laws for Routed Language Models
• Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
• Mixture-of-Experts with Expert Choice Routing
• AutoDistil : Few-shot Task-agnostic Neural Architecture Search for Distilling Large Language Models
• The Efficiency Misnomer

---

Diversity in Language Generation

• Presentation slides
• Notes
• The Curious Case of Neural Text Degeneration
• Comparison of Diverse Decoding Methods from Conditional Language Models
• Trading Off Diversity and Quality in Natural Language Generation
• Mirostat: A Neural Text Decoding Algorithm that Directly Controls Perplexity
• COD3S: Diverse Generation with Discrete Semantic Signatures
• A Systematic Characterization of Sampling Algorithms for Open-ended Language Generation
• How Decoding Strategies Affect the Verifiability of Generated Text
• What Do You Get When You Cross Beam Search with Nucleus Sampling?
• A Diversity-Promoting Objective Function for Neural Conversation Models
• A Contrastive Framework for Neural Text Generation
• Noisy Parallel Approximate Decoding for Conditional Recurrent Language Model
• A Simple, Fast Diverse Decoding Algorithm for Neural Generation
• Diverse Beam Search: Decoding Diverse Solutions from Neural Sequence Models

9. Course Assignments/Grades:

Assignment	Due date	Weight
Professionalization & Class Participation	Throughout	10%
First presentation		10%
Second presentation		10%
Third presentation		10%
Individual or Group assignment: Project Proposal		Ungraded
Individual or Group assignment: Term Project		60%

Notes on Assignments:

Presentations:

Presentations will be graded with a check mark upon completion, meaning students will be assigned full presentation grade upon delivery. Since this is a directed reading/research course, students are expected to do their best to deliver high quality presentations. This includes making sure a student starts preparing early, does her/his best to digest and explain the matrial, and include enough examples to illustrate a concept. Concievably, some material will be techincal and/or will need enough attention. Discussions in class will be complementary to the student presentations and all students will be expected to participate by asking questions and possibly answering others' questions..

**Deliverables:

A presentation (with slides and possibly code)

Project Proposal:

Students will prepare a 2-page proposal of a research project. The proposal should have enough details about the method, data, brief lietrature review, expected outcome, and (in group projects) a breakdown of work over time (and among members).Students will have the opportunity to discuss their ideas closely with the instructor and their peers.

**Deliverables:

A 2-page project proposal.

Final Project:

The major assignment in this course is a research paper that presents a novel deep learning method to solve a problem, or a novel application of one of the deep learning methods covered in class. The paper should be of high-quality and follow the norms of academic publishing within the DL and/or NLP community. Students will have the opportunity to work closely with the isntructors and peers on their projects.

**Deliverables:

An 8-page, publishable, research paper + all relevant code.

---

10. Course Policies

Attendance: Since this is a directed research/reading course, some Friday sessions may be canceled upon agreement in class, with prior notice. Otherwise, students are expected to attend regularly and come prepared. The UBC calendar states: “Regular attendance is expected of students in all their classes (including lectures, laboratories, tutorials, seminars, etc.). Students who neglect their academic work and assignments may be excluded from the final examinations. Students who are unavoidably absent because of illness or disability should report to their instructors on return to classes.”

Evaluation: All assignments will be marked using the evaluative criteria given in this syllabus.

Access & Diversity: Access & Diversity works with the University to create an inclusive living and learning environment in which all students can thrive. The University accommodates students with disabilities who have registered with the Access and Diversity unit. You must register with the Disability Resource Centre to be granted special accommodations for any on-going conditions. Religious Accommodation: The University accommodates students whose religious obligations conflict with attendance, submitting assignments, or completing scheduled tests and examinations. Please let your instructor know in advance, preferably in the first week of class, if you will require any accommodation on these grounds. Students who plan to be absent for varsity athletics, family obligations, or other similar commitments, cannot assume they will be accommodated, and should discuss their commitments with the instructor before the course drop date. UBC policy on Religious Holidays.

Academic Integrity

Plagiarism Plagiarism is the most serious academic offence that a student can commit. Regardless of whether or not it was committed intentionally, plagiarism has serious academic consequences and can result in expulsion from the university. Plagiarism involves the improper use of somebody else's words or ideas in one's work.

It is your responsibility to make sure you fully understand what plagiarism is. Many students who think they understand plagiarism do in fact commit what UBC calls "reckless plagiarism." Below is an excerpt on reckless plagiarism from UBC Faculty of Arts' leaflet, Plagiarism Avoided: Taking Responsibility for Your Work.

"The bulk of plagiarism falls into this category. Reckless plagiarism is often the result of careless research, poor time management, and a lack of confidence in your own ability to think critically. Examples of reckless plagiarism include:

• Taking phrases, sentences, paragraphs, or statistical findings from a variety of sources and piecing them together into an essay (piecemeal plagiarism);
• Taking the words of another author and failing to note clearly that they are not your own. In other words, you have not put a direct quotation within quotation marks;
• Using statistical findings without acknowledging your source;
• Taking another author's idea, without your own critical analysis, and failing to acknowledge that this idea is not yours;
• Paraphrasing (i.e. rewording or rearranging words so that your work resembles, but does not copy, the original) without acknowledging your source;
• Using footnotes or material quoted in other sources as if they were the results of your own research; and
• Submitting a piece of work with inaccurate text references, sloppy footnotes, or incomplete source (bibliographic) information."

Bear in mind that this is only one example of the different forms of plagiarism. Before preparing for their written assignments, students are strongly encouraged to familiarize themselves with the following source on plagiarism: the Academic Integrity Resource Centre.

If after reading these materials you still are unsure about how to properly use sources in your work, please ask me for clarification. Students are held responsible for knowing and following all University regulations regarding academic dishonesty. If a student does not know how to properly cite a source or what constitutes proper use of a source it is the student's personal responsibility to obtain the needed information and to apply it within University guidelines and policies. If evidence of academic dishonesty is found in a course assignment, previously submitted work in this course may be reviewed for possible academic dishonesty and grades modified as appropriate. UBC policy requires that all suspected cases of academic dishonesty must be forwarded to the Dean for possible action.