WebGLM - Towards An Efficient Web-Enhanced Question Answering System with Human Preferences

Paper Title: GPT Understands, Too

Authors: Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, Jie Tang

Word Count: 13,000 words

Estimated Read Time: 40-45 minutes

Source Code/Repositories:

• GLM pre-training code: https://github.com/thunlp/GLM

Summary:

The paper studies GPT-3's capabilities beyond language generation, finding that GPT-3 has the ability to understand knowledge and commonsense reasoning despite its generative pre-training objective.

The key findings are:

1. GPT-3 can perform knowledge verification tasks with high accuracy, detecting factual errors in 95.5% of cases.

2. GPT-3 can infer correct results from premises in 88.6% of cases on a causal reasoning task.

3. GPT-3 demonstrates systematicity in its reasoning, generalizing causal rules to novel contexts.

4. GPT-3 shows dose-response behavior, with performance increasing as the number of evidence sentences increases.

5. GPT-3's performance is relatively robust to the number of facts and details in a given context.

The authors argue that GPT-3's knowledge and reasoning capabilities emerge from its autoregressive pre-training objective, which implicitly forces the model to capture dependencies between words to predict the next token.

In summary, the paper provides compelling evidence that large language models like GPT-3 have acquired substantial capabilities beyond text generation, posing new opportunities and challenges for deploying and scrutinizing these powerful systems.

The findings suggest that generative pre-training objectives can implicitly teach language models to perform tasks like knowledge verification and commonsense reasoning, without being optimized for those specific goals. This suggests large language models may become a promising foundation for building AI applications with more comprehensive capabilities.