+
+ Recent studies have extensively explored different approaches to enhance LLMs’ reasoning abilities
+after pre-training. The two main approaches are continued training and instruction tuning. Continued
+training focuses on training LLMs with language model loss on large-scale math or science-centric
+documents [Lewkowycz et al., 2022, Taylor et al., 2022, Azerbayev et al., 2023, Shao et al., 2024,
+Ying et al., 2024] recalled from the general web corpus. Instruction tuning, on the other hand, seeks
+to employ supervised fine-tuning loss on curated high-quality instruction-response pairs [Ouyang
+et al., 2022, Chung et al., 2024]. While human-annotated instruction datasets [Cobbe et al., 2021,
+Hendrycks et al., 2021b, Amini et al., 2019] are often limited in scale, recent studies [Yu et al., 2023,
+Yue et al., 2023, Tosh et al., 2024, Li et al., 2024, Tang et al., 2024] attempt to prompt state-of-the-
+art models like GPT-4 with seed data to synthesize large-scale instruction datasets. However, the
+synthesized instruction data becomes highly biased towards the seed data distribution and is prone to
+a high degree of hallucination.
+
+To address these limitations, we propose to discover naturally existing instruction data from
+the web. We argue that the web corpus (e.g., Common Crawl) already contains a vast amount of
+high-quality instruction data that can enhance LLM reasoning. For example, textbooks, exams,
+educational resources, and forums on the web contain a wealth of instruction-following pairs across
+various domains like math, science, engineering, and humanities. We believe such readily available
+instruction data is not only diverse but also of high quality. However, such instruction data is highly
+dispersed across the corpus, which makes it particularly challenging to discover.
+
+In this paper, we aim to mine these instruction-response pairs from the web using a three-step pipeline.
+(1) Recall step: We create a diverse seed dataset by crawling several quiz websites. We use this
+seed data to train a fastText model [Joulin et al., 2016] and employ it to recall documents from
+Common Crawl [Computer, 2023]. GPT-4 is used to trim down the recalled documents by their root
+URL. We obtain 18M documents through this step. (2) Extract step: We utilize open-source LLMs
+like Mixtral [Jiang et al., 2024] to extract Q-A pairs from these documents, producing roughly 5M
+candidate Q-A pairs. (3) Refine step: After extraction, we further employ Mixtral-8×7B [Jiang et al.,
+2024] and Qwen-72B [Bai et al., 2023] to refine [Zheng et al., 2024b] these candidate Q-A pairs. This
+refinement operation aims to remove unrelated content, fix formality, and add missing explanations
+to the candidate Q-A pairs. This refinement operation is pivotal to maintaining the quality of the
+mined Q-A pairs. Eventually, we harvest a total of 10M instruction-response pairs through these
+steps. Unlike existing instruction-tuning dataset, our dataset WEBINSTRUCT is purely mined from
+the Web without any human crowdsourcing or GPT-4 distillation.
+
+To validate the effectiveness of our WEBINSTRUCT, we train MAmmoTH2, a family of open-sourced
+language models that are trained on a variety of base models, including Mistral-7B, Llama3-8B,
+Mixtral-8×7B, and Yi-34B. Our experiments demonstrate the effectiveness of scaling up instruction
+tuning on science reasoning tasks. On a range of held-out reasoning benchmarks including Theo-
+remQA, GSM8K, MATH, ARC-C, MMLU-STEM, GPQA and BBH, MAmmoTH2 exhibits remarkably
+better reasoning abilities compared to the base model. MAmmoTH2 boosts the performance of the base
+model significantly. For example, MAmmoTH2-7B boosts the performance of Mistral-7B by 5-31%
+across different datasets, while MAmmoTH2-34B boosts the Yi-34B base model by 6-24% across
+different datasets. Notably, WEBINSTRUCT does not contain any data from any of the evaluation
+benchmarks, highlighting the strong generalization of the instruction-tuned model. These findings
+reveal new perspectives on how to construct high-quality instruction dataset in scale without requiring
+crowdsourcing or distillation.
+
+
+
+
+
+
+
+
+
+
+
+
+ Figure 3: Step 1: Recall relevant documents from Common Crawl. Step 2: Extracting Q-A pairs.
+Step 3: Refine with the extracted Q-A pairs.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ Figure 4: An illustrating example from WEBINSTRUCT for the extraction and refine step
+
+
+
+
+
+
+
+ To further enhance the performance of MAmmoTH2 on more general code generation, math rea-
+ soning and instruction-following tasks, we continue to tune it on several open-source instruction
+ datasets including OpenHermes2.5, Code-Feedback, Math-plus. The further enhanced model is
+ named MAmmoTH2-Plus, which exhibits strong performance across these general tasks. For example,
+ MAmmoTH2-7B-Plus achieves top performance on HumanEval and MBPP code generation bench-
+ marks, and MAmmoTH2-8×7B leads the AlpacaEval 2.0 and Arena Hard leaderboards for general
+ language understanding and instruction-following. The consistent gain of MAmmoTH2-Plus over the
+ official instruct models (e.g. Mixtral-Instruct-v0.1) further validates the effectiveness of our dataset
+ and training approach.
+
+
+
+
+
+
+ 
+ 
+ 
+ 
+ 
+