MMLU eval failing on granite-8b-starter model: 500 Server Error

[relyt0925]

Running basic mmlu eval on the granite-8b-starter model (or any instructlab aligned models based on granite-8b fails with)

[root@tyler-machine-boot-6 root]# /root/bin/ilab.sh --config /var/mnt/instg1/instructlab/config.yaml model evaluate --model /var/mnt/instg1/instructlab/models/granite-8b-starter/  --benchmark mmlu --gpus 8
Using local model found at '/var/mnt/instg1/instructlab/models/granite-8b-starter' for '--model'
INFO 2024-12-09 05:37:38,853 numexpr.utils:145: Note: detected 80 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
INFO 2024-12-09 05:37:38,854 numexpr.utils:148: Note: NumExpr detected 80 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 16.
INFO 2024-12-09 05:37:38,854 numexpr.utils:161: NumExpr defaulting to 16 threads.
INFO 2024-12-09 05:37:40,302 datasets:59: PyTorch version 2.4.1 available.
Using gpus from --gpus or config and ignoring --tensor-parallel-size configured in serve vllm_args
INFO 2024-12-09 05:37:50,562 instructlab.model.backends.vllm:112: Trying to connect to model server at http://127.0.0.1:8000/v1
INFO 2024-12-09 05:37:51,955 instructlab.model.backends.vllm:313: vLLM starting up on pid 94 at http://127.0.0.1:49301/v1
INFO 2024-12-09 05:37:51,955 instructlab.model.backends.vllm:121: Starting a temporary vLLM server at http://127.0.0.1:49301/v1
INFO 2024-12-09 05:37:51,955 instructlab.model.backends.vllm:136: Waiting for the vLLM server to start at http://127.0.0.1:49301/v1, this might take a moment... Attempt: 1/120
INFO 2024-12-09 05:37:55,138 instructlab.model.backends.vllm:136: Waiting for the vLLM server to start at http://127.0.0.1:49301/v1, this might take a moment... Attempt: 2/120
INFO 2024-12-09 05:37:58,453 instructlab.model.backends.vllm:136: Waiting for the vLLM server to start at http://127.0.0.1:49301/v1, this might take a moment... Attempt: 3/120
INFO 2024-12-09 05:38:01,708 instructlab.model.backends.vllm:136: Waiting for the vLLM server to start at http://127.0.0.1:49301/v1, this might take a moment... Attempt: 4/120
INFO 2024-12-09 05:38:05,020 instructlab.model.backends.vllm:136: Waiting for the vLLM server to start at http://127.0.0.1:49301/v1, this might take a moment... Attempt: 5/120
INFO 2024-12-09 05:38:08,281 instructlab.model.backends.vllm:136: Waiting for the vLLM server to start at http://127.0.0.1:49301/v1, this might take a moment... Attempt: 6/120
INFO 2024-12-09 05:38:11,621 instructlab.model.backends.vllm:136: Waiting for the vLLM server to start at http://127.0.0.1:49301/v1, this might take a moment... Attempt: 7/120
INFO 2024-12-09 05:38:14,890 instructlab.model.backends.vllm:136: Waiting for the vLLM server to start at http://127.0.0.1:49301/v1, this might take a moment... Attempt: 8/120
INFO 2024-12-09 05:38:18,188 instructlab.model.backends.vllm:136: Waiting for the vLLM server to start at http://127.0.0.1:49301/v1, this might take a moment... Attempt: 9/120
INFO 2024-12-09 05:38:21,540 instructlab.model.backends.vllm:136: Waiting for the vLLM server to start at http://127.0.0.1:49301/v1, this might take a moment... Attempt: 10/120
INFO 2024-12-09 05:38:24,954 instructlab.model.backends.vllm:136: Waiting for the vLLM server to start at http://127.0.0.1:49301/v1, this might take a moment... Attempt: 11/120
INFO 2024-12-09 05:38:28,243 instructlab.model.backends.vllm:136: Waiting for the vLLM server to start at http://127.0.0.1:49301/v1, this might take a moment... Attempt: 12/120
INFO 2024-12-09 05:38:31,516 instructlab.model.backends.vllm:136: Waiting for the vLLM server to start at http://127.0.0.1:49301/v1, this might take a moment... Attempt: 13/120
INFO 2024-12-09 05:38:34,736 instructlab.model.backends.vllm:136: Waiting for the vLLM server to start at http://127.0.0.1:49301/v1, this might take a moment... Attempt: 14/120
INFO 2024-12-09 05:38:38,196 instructlab.model.backends.vllm:136: Waiting for the vLLM server to start at http://127.0.0.1:49301/v1, this might take a moment... Attempt: 15/120
INFO 2024-12-09 05:38:41,622 instructlab.model.backends.vllm:136: Waiting for the vLLM server to start at http://127.0.0.1:49301/v1, this might take a moment... Attempt: 16/120
INFO 2024-12-09 05:38:44,931 instructlab.model.backends.vllm:136: Waiting for the vLLM server to start at http://127.0.0.1:49301/v1, this might take a moment... Attempt: 17/120
INFO 2024-12-09 05:38:45,351 instructlab.model.backends.vllm:143: vLLM engine successfully started at http://127.0.0.1:49301/v1
INFO 2024-12-09 05:38:45,354 lm-eval:164: Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
INFO 2024-12-09 05:38:45,354 lm-eval:201: Initializing local-completions model, with arguments: {'base_url': 'http://127.0.0.1:49301/v1/completions', 'model': '/var/mnt/instg1/instructlab/models/granite-8b-starter/', 'tokenizer_backend': 'huggingface'}
WARNING 2024-12-09 05:38:45,354 lm-eval:99: Automatic batch size is not supported for API models. Defaulting to batch size 1.
INFO 2024-12-09 05:38:45,354 lm-eval:111: Using max length 2048 - 1
INFO 2024-12-09 05:38:45,354 lm-eval:114: Concurrent requests are disabled. To enable concurrent requests, set `num_concurrent` > 1.
INFO 2024-12-09 05:38:45,354 lm-eval:124: Using tokenizer huggingface
Using the latest cached version of the module from /var/mnt/instg1/instructlab/.cache/huggingface/modules/datasets_modules/datasets/hails--mmlu_no_train/b7d5f7f21003c21be079f11495ee011332b980bd1cd7e70cc740e8c079e5bda2 (last modified on Mon Dec  9 04:52:38 2024) since it couldn't be found locally at hails/mmlu_no_train, or remotely on the Hugging Face Hub.
WARNING 2024-12-09 05:38:58,973 datasets.load:1608: Using the latest cached version of the module from /var/mnt/instg1/instructlab/.cache/huggingface/modules/datasets_modules/datasets/hails--mmlu_no_train/b7d5f7f21003c21be079f11495ee011332b980bd1cd7e70cc740e8c079e5bda2 (last modified on Mon Dec  9 04:52:38 2024) since it couldn't be found locally at hails/mmlu_no_train, or remotely on the Hugging Face Hub.
WARNING 2024-12-09 05:39:26,568 lm-eval:270: Overwriting default num_fewshot of mmlu_world_religions from None to 5
WARNING 2024-12-09 05:39:26,568 lm-eval:270: Overwriting default num_fewshot of mmlu_virology from None to 5
WARNING 2024-12-09 05:39:26,568 lm-eval:270: Overwriting default num_fewshot of mmlu_us_foreign_policy from None to 5
WARNING 2024-12-09 05:39:26,568 lm-eval:270: Overwriting default num_fewshot of mmlu_sociology from None to 5
WARNING 2024-12-09 05:39:26,568 lm-eval:270: Overwriting default num_fewshot of mmlu_security_studies from None to 5
WARNING 2024-12-09 05:39:26,569 lm-eval:270: Overwriting default num_fewshot of mmlu_public_relations from None to 5
WARNING 2024-12-09 05:39:26,569 lm-eval:270: Overwriting default num_fewshot of mmlu_professional_psychology from None to 5
WARNING 2024-12-09 05:39:26,569 lm-eval:270: Overwriting default num_fewshot of mmlu_professional_medicine from None to 5
WARNING 2024-12-09 05:39:26,569 lm-eval:270: Overwriting default num_fewshot of mmlu_professional_law from None to 5
WARNING 2024-12-09 05:39:26,569 lm-eval:270: Overwriting default num_fewshot of mmlu_professional_accounting from None to 5
WARNING 2024-12-09 05:39:26,569 lm-eval:270: Overwriting default num_fewshot of mmlu_prehistory from None to 5
WARNING 2024-12-09 05:39:26,569 lm-eval:270: Overwriting default num_fewshot of mmlu_philosophy from None to 5
WARNING 2024-12-09 05:39:26,569 lm-eval:270: Overwriting default num_fewshot of mmlu_nutrition from None to 5
WARNING 2024-12-09 05:39:26,569 lm-eval:270: Overwriting default num_fewshot of mmlu_moral_scenarios from None to 5
WARNING 2024-12-09 05:39:26,569 lm-eval:270: Overwriting default num_fewshot of mmlu_moral_disputes from None to 5
WARNING 2024-12-09 05:39:26,569 lm-eval:270: Overwriting default num_fewshot of mmlu_miscellaneous from None to 5
WARNING 2024-12-09 05:39:26,569 lm-eval:270: Overwriting default num_fewshot of mmlu_medical_genetics from None to 5
WARNING 2024-12-09 05:39:26,569 lm-eval:270: Overwriting default num_fewshot of mmlu_marketing from None to 5
WARNING 2024-12-09 05:39:26,569 lm-eval:270: Overwriting default num_fewshot of mmlu_management from None to 5
WARNING 2024-12-09 05:39:26,569 lm-eval:270: Overwriting default num_fewshot of mmlu_machine_learning from None to 5
WARNING 2024-12-09 05:39:26,569 lm-eval:270: Overwriting default num_fewshot of mmlu_logical_fallacies from None to 5
WARNING 2024-12-09 05:39:26,569 lm-eval:270: Overwriting default num_fewshot of mmlu_jurisprudence from None to 5
WARNING 2024-12-09 05:39:26,569 lm-eval:270: Overwriting default num_fewshot of mmlu_international_law from None to 5
WARNING 2024-12-09 05:39:26,569 lm-eval:270: Overwriting default num_fewshot of mmlu_human_sexuality from None to 5
WARNING 2024-12-09 05:39:26,569 lm-eval:270: Overwriting default num_fewshot of mmlu_human_aging from None to 5
WARNING 2024-12-09 05:39:26,569 lm-eval:270: Overwriting default num_fewshot of mmlu_high_school_world_history from None to 5
WARNING 2024-12-09 05:39:26,569 lm-eval:270: Overwriting default num_fewshot of mmlu_high_school_us_history from None to 5
WARNING 2024-12-09 05:39:26,569 lm-eval:270: Overwriting default num_fewshot of mmlu_high_school_statistics from None to 5
WARNING 2024-12-09 05:39:26,569 lm-eval:270: Overwriting default num_fewshot of mmlu_high_school_psychology from None to 5
WARNING 2024-12-09 05:39:26,569 lm-eval:270: Overwriting default num_fewshot of mmlu_high_school_physics from None to 5
WARNING 2024-12-09 05:39:26,569 lm-eval:270: Overwriting default num_fewshot of mmlu_high_school_microeconomics from None to 5
WARNING 2024-12-09 05:39:26,569 lm-eval:270: Overwriting default num_fewshot of mmlu_high_school_mathematics from None to 5
WARNING 2024-12-09 05:39:26,569 lm-eval:270: Overwriting default num_fewshot of mmlu_high_school_macroeconomics from None to 5
WARNING 2024-12-09 05:39:26,569 lm-eval:270: Overwriting default num_fewshot of mmlu_high_school_government_and_politics from None to 5
WARNING 2024-12-09 05:39:26,570 lm-eval:270: Overwriting default num_fewshot of mmlu_high_school_geography from None to 5
WARNING 2024-12-09 05:39:26,570 lm-eval:270: Overwriting default num_fewshot of mmlu_high_school_european_history from None to 5
WARNING 2024-12-09 05:39:26,570 lm-eval:270: Overwriting default num_fewshot of mmlu_high_school_computer_science from None to 5
WARNING 2024-12-09 05:39:26,570 lm-eval:270: Overwriting default num_fewshot of mmlu_high_school_chemistry from None to 5
WARNING 2024-12-09 05:39:26,570 lm-eval:270: Overwriting default num_fewshot of mmlu_high_school_biology from None to 5
WARNING 2024-12-09 05:39:26,570 lm-eval:270: Overwriting default num_fewshot of mmlu_global_facts from None to 5
WARNING 2024-12-09 05:39:26,570 lm-eval:270: Overwriting default num_fewshot of mmlu_formal_logic from None to 5
WARNING 2024-12-09 05:39:26,570 lm-eval:270: Overwriting default num_fewshot of mmlu_elementary_mathematics from None to 5
WARNING 2024-12-09 05:39:26,570 lm-eval:270: Overwriting default num_fewshot of mmlu_electrical_engineering from None to 5
WARNING 2024-12-09 05:39:26,570 lm-eval:270: Overwriting default num_fewshot of mmlu_econometrics from None to 5
WARNING 2024-12-09 05:39:26,570 lm-eval:270: Overwriting default num_fewshot of mmlu_conceptual_physics from None to 5
WARNING 2024-12-09 05:39:26,570 lm-eval:270: Overwriting default num_fewshot of mmlu_computer_security from None to 5
WARNING 2024-12-09 05:39:26,570 lm-eval:270: Overwriting default num_fewshot of mmlu_college_physics from None to 5
WARNING 2024-12-09 05:39:26,570 lm-eval:270: Overwriting default num_fewshot of mmlu_college_medicine from None to 5
WARNING 2024-12-09 05:39:26,570 lm-eval:270: Overwriting default num_fewshot of mmlu_college_mathematics from None to 5
WARNING 2024-12-09 05:39:26,570 lm-eval:270: Overwriting default num_fewshot of mmlu_college_computer_science from None to 5
WARNING 2024-12-09 05:39:26,570 lm-eval:270: Overwriting default num_fewshot of mmlu_college_chemistry from None to 5
WARNING 2024-12-09 05:39:26,570 lm-eval:270: Overwriting default num_fewshot of mmlu_college_biology from None to 5
WARNING 2024-12-09 05:39:26,570 lm-eval:270: Overwriting default num_fewshot of mmlu_clinical_knowledge from None to 5
WARNING 2024-12-09 05:39:26,570 lm-eval:270: Overwriting default num_fewshot of mmlu_business_ethics from None to 5
WARNING 2024-12-09 05:39:26,570 lm-eval:270: Overwriting default num_fewshot of mmlu_astronomy from None to 5
WARNING 2024-12-09 05:39:26,570 lm-eval:270: Overwriting default num_fewshot of mmlu_anatomy from None to 5
WARNING 2024-12-09 05:39:26,570 lm-eval:270: Overwriting default num_fewshot of mmlu_abstract_algebra from None to 5
INFO 2024-12-09 05:39:26,575 lm-eval:415: Building contexts for mmlu_world_religions on rank 0...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 171/171 [00:01<00:00, 141.06it/s]
INFO 2024-12-09 05:39:27,795 lm-eval:415: Building contexts for mmlu_virology on rank 0...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 166/166 [00:01<00:00, 140.88it/s]
INFO 2024-12-09 05:39:28,981 lm-eval:415: Building contexts for mmlu_us_foreign_policy on rank 0...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100/100 [00:00<00:00, 141.09it/s]
INFO 2024-12-09 05:39:29,695 lm-eval:415: Building contexts for mmlu_sociology on rank 0...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 201/201 [00:01<00:00, 139.72it/s]
INFO 2024-12-09 05:39:31,143 lm-eval:415: Building contexts for mmlu_security_studies on rank 0...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 245/245 [00:02<00:00, 121.40it/s]
INFO 2024-12-09 05:39:33,172 lm-eval:415: Building contexts for mmlu_public_relations on rank 0...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 110/110 [00:00<00:00, 141.81it/s]
INFO 2024-12-09 05:39:33,954 lm-eval:415: Building contexts for mmlu_professional_psychology on rank 0...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 612/612 [00:04<00:00, 141.74it/s]
INFO 2024-12-09 05:39:38,298 lm-eval:415: Building contexts for mmlu_professional_medicine on rank 0...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 272/272 [00:01<00:00, 141.56it/s]
INFO 2024-12-09 05:39:40,231 lm-eval:415: Building contexts for mmlu_professional_law on rank 0...
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1534/1534 [00:10<00:00, 141.54it/s]
INFO 2024-12-09 05:39:51,136 lm-eval:415: Building contexts for mmlu_professional_accounting on rank 0...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 282/282 [00:01<00:00, 141.79it/s]
INFO 2024-12-09 05:39:53,138 lm-eval:415: Building contexts for mmlu_prehistory on rank 0...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 324/324 [00:02<00:00, 141.52it/s]
INFO 2024-12-09 05:39:55,442 lm-eval:415: Building contexts for mmlu_philosophy on rank 0...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 311/311 [00:02<00:00, 141.77it/s]
INFO 2024-12-09 05:39:57,649 lm-eval:415: Building contexts for mmlu_nutrition on rank 0...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 306/306 [00:02<00:00, 141.61it/s]
INFO 2024-12-09 05:39:59,824 lm-eval:415: Building contexts for mmlu_moral_scenarios on rank 0...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 895/895 [00:06<00:00, 142.13it/s]
INFO 2024-12-09 05:40:06,159 lm-eval:415: Building contexts for mmlu_moral_disputes on rank 0...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 346/346 [00:02<00:00, 141.72it/s]
INFO 2024-12-09 05:40:08,615 lm-eval:415: Building contexts for mmlu_miscellaneous on rank 0...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 783/783 [00:05<00:00, 142.14it/s]
INFO 2024-12-09 05:40:14,157 lm-eval:415: Building contexts for mmlu_medical_genetics on rank 0...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100/100 [00:00<00:00, 142.21it/s]
INFO 2024-12-09 05:40:14,865 lm-eval:415: Building contexts for mmlu_marketing on rank 0...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 234/234 [00:01<00:00, 142.12it/s]
INFO 2024-12-09 05:40:16,522 lm-eval:415: Building contexts for mmlu_management on rank 0...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 103/103 [00:00<00:00, 142.41it/s]
INFO 2024-12-09 05:40:17,250 lm-eval:415: Building contexts for mmlu_machine_learning on rank 0...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 112/112 [00:00<00:00, 142.14it/s]
INFO 2024-12-09 05:40:18,044 lm-eval:415: Building contexts for mmlu_logical_fallacies on rank 0...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 163/163 [00:01<00:00, 142.25it/s]
INFO 2024-12-09 05:40:19,197 lm-eval:415: Building contexts for mmlu_jurisprudence on rank 0...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 108/108 [00:00<00:00, 142.30it/s]
INFO 2024-12-09 05:40:19,961 lm-eval:415: Building contexts for mmlu_international_law on rank 0...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 121/121 [00:00<00:00, 142.49it/s]
INFO 2024-12-09 05:40:20,817 lm-eval:415: Building contexts for mmlu_human_sexuality on rank 0...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 131/131 [00:00<00:00, 142.30it/s]
INFO 2024-12-09 05:40:21,744 lm-eval:415: Building contexts for mmlu_human_aging on rank 0...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 223/223 [00:01<00:00, 142.73it/s]
INFO 2024-12-09 05:40:23,316 lm-eval:415: Building contexts for mmlu_high_school_world_history on rank 0...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 237/237 [00:01<00:00, 141.99it/s]
INFO 2024-12-09 05:40:24,996 lm-eval:415: Building contexts for mmlu_high_school_us_history on rank 0...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 204/204 [00:01<00:00, 141.71it/s]
INFO 2024-12-09 05:40:26,446 lm-eval:415: Building contexts for mmlu_high_school_statistics on rank 0...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 216/216 [00:01<00:00, 142.04it/s]
INFO 2024-12-09 05:40:27,976 lm-eval:415: Building contexts for mmlu_high_school_psychology on rank 0...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 545/545 [00:03<00:00, 142.61it/s]
INFO 2024-12-09 05:40:31,821 lm-eval:415: Building contexts for mmlu_high_school_physics on rank 0...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 151/151 [00:01<00:00, 141.90it/s]
INFO 2024-12-09 05:40:32,892 lm-eval:415: Building contexts for mmlu_high_school_microeconomics on rank 0...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 238/238 [00:01<00:00, 141.87it/s]
INFO 2024-12-09 05:40:34,581 lm-eval:415: Building contexts for mmlu_high_school_mathematics on rank 0...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 270/270 [00:01<00:00, 142.02it/s]
INFO 2024-12-09 05:40:36,494 lm-eval:415: Building contexts for mmlu_high_school_macroeconomics on rank 0...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 390/390 [00:02<00:00, 142.25it/s]
INFO 2024-12-09 05:40:39,252 lm-eval:415: Building contexts for mmlu_high_school_government_and_politics on rank 0...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 193/193 [00:01<00:00, 112.93it/s]
INFO 2024-12-09 05:40:40,970 lm-eval:415: Building contexts for mmlu_high_school_geography on rank 0...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 198/198 [00:01<00:00, 141.97it/s]
INFO 2024-12-09 05:40:42,374 lm-eval:415: Building contexts for mmlu_high_school_european_history on rank 0...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 165/165 [00:01<00:00, 141.01it/s]
INFO 2024-12-09 05:40:43,552 lm-eval:415: Building contexts for mmlu_high_school_computer_science on rank 0...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100/100 [00:00<00:00, 141.86it/s]
INFO 2024-12-09 05:40:44,262 lm-eval:415: Building contexts for mmlu_high_school_chemistry on rank 0...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 203/203 [00:01<00:00, 142.06it/s]
INFO 2024-12-09 05:40:45,700 lm-eval:415: Building contexts for mmlu_high_school_biology on rank 0...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 310/310 [00:02<00:00, 142.41it/s]
INFO 2024-12-09 05:40:47,890 lm-eval:415: Building contexts for mmlu_global_facts on rank 0...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100/100 [00:00<00:00, 141.69it/s]
INFO 2024-12-09 05:40:48,601 lm-eval:415: Building contexts for mmlu_formal_logic on rank 0...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 126/126 [00:00<00:00, 142.15it/s]
INFO 2024-12-09 05:40:49,493 lm-eval:415: Building contexts for mmlu_elementary_mathematics on rank 0...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 378/378 [00:02<00:00, 142.15it/s]
INFO 2024-12-09 05:40:52,169 lm-eval:415: Building contexts for mmlu_electrical_engineering on rank 0...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 145/145 [00:01<00:00, 142.37it/s]
INFO 2024-12-09 05:40:53,194 lm-eval:415: Building contexts for mmlu_econometrics on rank 0...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 114/114 [00:00<00:00, 141.27it/s]
INFO 2024-12-09 05:40:54,006 lm-eval:415: Building contexts for mmlu_conceptual_physics on rank 0...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 235/235 [00:01<00:00, 142.33it/s]
INFO 2024-12-09 05:40:55,667 lm-eval:415: Building contexts for mmlu_computer_security on rank 0...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100/100 [00:00<00:00, 142.16it/s]
INFO 2024-12-09 05:40:56,376 lm-eval:415: Building contexts for mmlu_college_physics on rank 0...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 102/102 [00:00<00:00, 142.08it/s]
INFO 2024-12-09 05:40:57,099 lm-eval:415: Building contexts for mmlu_college_medicine on rank 0...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 173/173 [00:01<00:00, 142.33it/s]
INFO 2024-12-09 05:40:58,322 lm-eval:415: Building contexts for mmlu_college_mathematics on rank 0...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100/100 [00:00<00:00, 142.13it/s]
INFO 2024-12-09 05:40:59,031 lm-eval:415: Building contexts for mmlu_college_computer_science on rank 0...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100/100 [00:00<00:00, 141.69it/s]
INFO 2024-12-09 05:40:59,742 lm-eval:415: Building contexts for mmlu_college_chemistry on rank 0...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100/100 [00:00<00:00, 142.62it/s]
INFO 2024-12-09 05:41:00,448 lm-eval:415: Building contexts for mmlu_college_biology on rank 0...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 144/144 [00:01<00:00, 142.88it/s]
INFO 2024-12-09 05:41:01,463 lm-eval:415: Building contexts for mmlu_clinical_knowledge on rank 0...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 265/265 [00:01<00:00, 143.02it/s]
INFO 2024-12-09 05:41:03,327 lm-eval:415: Building contexts for mmlu_business_ethics on rank 0...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100/100 [00:00<00:00, 141.88it/s]
INFO 2024-12-09 05:41:04,037 lm-eval:415: Building contexts for mmlu_astronomy on rank 0...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 152/152 [00:01<00:00, 141.79it/s]
INFO 2024-12-09 05:41:05,116 lm-eval:415: Building contexts for mmlu_anatomy on rank 0...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 135/135 [00:00<00:00, 142.25it/s]
INFO 2024-12-09 05:41:06,071 lm-eval:415: Building contexts for mmlu_abstract_algebra on rank 0...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100/100 [00:00<00:00, 142.57it/s]
INFO 2024-12-09 05:41:06,777 lm-eval:489: Running loglikelihood requests
Requesting API:   0%|▏                                                                                                                                                                             | 48/56168 [00:16<2:51:51,  5.44it/s]WARNING 2024-12-09 05:44:05,620 lm-eval:347: API request failed with error message: Internal Server Error. Retrying...
WARNING 2024-12-09 05:44:06,786 lm-eval:347: API request failed with error message: Internal Server Error. Retrying...
WARNING 2024-12-09 05:44:07,959 lm-eval:347: API request failed with error message: Internal Server Error. Retrying...
INFO 2024-12-09 05:44:16,435 instructlab.model.backends.vllm:475: Waiting for GPU VRAM reclamation...
Traceback (most recent call last):
  File "/opt/app-root/bin/ilab", line 8, in <module>
    sys.exit(ilab())
             ^^^^^^
  File "/opt/app-root/lib64/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/app-root/lib64/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/opt/app-root/lib64/python3.11/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/app-root/lib64/python3.11/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/app-root/lib64/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/app-root/lib64/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/app-root/lib64/python3.11/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/app-root/lib64/python3.11/site-packages/instructlab/clickext.py", line 323, in wrapper
    return f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^
  File "/opt/app-root/lib64/python3.11/site-packages/instructlab/model/evaluate.py", line 811, in evaluate
    overall_score, individual_scores = evaluator.run(api_base)
                                       ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/app-root/lib64/python3.11/site-packages/instructlab/eval/mmlu.py", line 144, in run
    results = self._run_mmlu(server_url)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/app-root/lib64/python3.11/site-packages/instructlab/eval/mmlu.py", line 171, in _run_mmlu
    mmlu_output = self._simple_evaluate_with_error_handling(
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/app-root/lib64/python3.11/site-packages/instructlab/eval/mmlu.py", line 187, in _simple_evaluate_with_error_handling
    return simple_evaluate(**kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/app-root/lib64/python3.11/site-packages/lm_eval/utils.py", line 397, in _wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/opt/app-root/lib64/python3.11/site-packages/lm_eval/evaluator.py", line 301, in simple_evaluate
    results = evaluate(
              ^^^^^^^^^
  File "/opt/app-root/lib64/python3.11/site-packages/lm_eval/utils.py", line 397, in _wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/opt/app-root/lib64/python3.11/site-packages/lm_eval/evaluator.py", line 500, in evaluate
    resps = getattr(lm, reqtype)(cloned_reqs)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/app-root/lib64/python3.11/site-packages/lm_eval/api/model.py", line 378, in loglikelihood
    return self._loglikelihood_tokens(new_reqs, disable_tqdm=disable_tqdm)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/app-root/lib64/python3.11/site-packages/lm_eval/models/api_models.py", line 502, in _loglikelihood_tokens
    outputs = retry(
              ^^^^^^
  File "/opt/app-root/lib64/python3.11/site-packages/tenacity/__init__.py", line 336, in wrapped_f
    return copy(f, *args, **kw)
           ^^^^^^^^^^^^^^^^^^^^
  File "/opt/app-root/lib64/python3.11/site-packages/tenacity/__init__.py", line 475, in __call__
    do = self.iter(retry_state=retry_state)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/app-root/lib64/python3.11/site-packages/tenacity/__init__.py", line 376, in iter
    result = action(retry_state)
             ^^^^^^^^^^^^^^^^^^^
  File "/opt/app-root/lib64/python3.11/site-packages/tenacity/__init__.py", line 418, in exc_check
    raise retry_exc.reraise()
          ^^^^^^^^^^^^^^^^^^^
  File "/opt/app-root/lib64/python3.11/site-packages/tenacity/__init__.py", line 185, in reraise
    raise self.last_attempt.result()
          ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/opt/app-root/lib64/python3.11/site-packages/tenacity/__init__.py", line 478, in __call__
    result = fn(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^
  File "/opt/app-root/lib64/python3.11/site-packages/lm_eval/models/api_models.py", line 350, in model_call
    response.raise_for_status()
  File "/opt/app-root/lib64/python3.11/site-packages/requests/models.py", line 1024, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http://127.0.0.1:49301/v1/completions
Requesting API:   0%|▏                                                                                                                                                                            | 48/56168 [00:37<12:02:36,  1.29it/s]

[bbrowning]

The client is getting a 500 error here, which means the vLLM server logs would be needed to diagnose what the actual error is. Do you have those available?

[relyt0925]

Yep let me get those!

[relyt0925]

Key error looks to be

INFO:     127.0.0.1:41012 - "POST /v1/completions HTTP/1.1" 500 Internal Server Error
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/opt/app-root/lib64/python3.11/site-packages/uvicorn/protocols/http/httptools_impl.py", line 401, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/app-root/lib64/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__
    return await self.app(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/app-root/lib64/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/opt/app-root/lib64/python3.11/site-packages/starlette/applications.py", line 113, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/opt/app-root/lib64/python3.11/site-packages/starlette/middleware/errors.py", line 187, in __call__
    raise exc
  File "/opt/app-root/lib64/python3.11/site-packages/starlette/middleware/errors.py", line 165, in __call__
    await self.app(scope, receive, _send)
  File "/opt/app-root/lib64/python3.11/site-packages/starlette/middleware/cors.py", line 85, in __call__
    await self.app(scope, receive, send)
  File "/opt/app-root/lib64/python3.11/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/opt/app-root/lib64/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/opt/app-root/lib64/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/opt/app-root/lib64/python3.11/site-packages/starlette/routing.py", line 715, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/opt/app-root/lib64/python3.11/site-packages/starlette/routing.py", line 735, in app
    await route.handle(scope, receive, send)
  File "/opt/app-root/lib64/python3.11/site-packages/starlette/routing.py", line 288, in handle
    await self.app(scope, receive, send)
  File "/opt/app-root/lib64/python3.11/site-packages/starlette/routing.py", line 76, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/opt/app-root/lib64/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/opt/app-root/lib64/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/opt/app-root/lib64/python3.11/site-packages/starlette/routing.py", line 73, in app
    response = await f(request)
               ^^^^^^^^^^^^^^^^
  File "/opt/app-root/lib64/python3.11/site-packages/fastapi/routing.py", line 301, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/app-root/lib64/python3.11/site-packages/fastapi/routing.py", line 212, in run_endpoint_function
    return await dependant.call(**values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/app-root/lib64/python3.11/site-packages/vllm/entrypoints/openai/api_server.py", line 334, in create_completion
    return JSONResponse(content=generator.model_dump())
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/app-root/lib64/python3.11/site-packages/starlette/responses.py", line 180, in __init__
    super().__init__(content, status_code, headers, media_type, background)
  File "/opt/app-root/lib64/python3.11/site-packages/starlette/responses.py", line 43, in __init__
    self.body = self.render(content)
                ^^^^^^^^^^^^^^^^^^^^
  File "/opt/app-root/lib64/python3.11/site-packages/starlette/responses.py", line 183, in render
    return json.dumps(
           ^^^^^^^^^^^
  File "/usr/lib64/python3.11/json/__init__.py", line 238, in dumps
    **kw).encode(obj)
          ^^^^^^^^^^^
  File "/usr/lib64/python3.11/json/encoder.py", line 200, in encode
    chunks = self.iterencode(o, _one_shot=True)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/json/encoder.py", line 258, in iterencode
    return _iterencode(o, 0)
           ^^^^^^^^^^^^^^^^^
ValueError: Out of range float values are not JSON compliant
WARNING 2024-12-09 14:55:03,056 lm-eval:347: API request failed with error message: Internal Server Error. Retrying...
DEBUG 12-09 14:55:03 launcher.py:54] port 56149 is used by process psutil.Process(pid=94, name='pt_main_thread', status='running', started='14:49:09') launched with command:

[relyt0925]

fulllogsmmlu.txt.zip

[relyt0925]

Key version info

(app-root) /$ pip list            
Package                           Version
--------------------------------- -----------------
absl-py                           2.1.0
accelerate                        1.0.1
aiofiles                          24.1.0
aiohappyeyeballs                  2.4.3
aiohttp                           3.11.2
aiosignal                         1.3.1
annotated-types                   0.7.0
anyio                             4.6.2.post1
appdirs                           1.4.4
attrs                             24.2.0
beautifulsoup4                    4.12.3
bitsandbytes                      0.44.1
certifi                           2024.8.30
chardet                           5.2.0
charset-normalizer                3.4.0
click                             8.1.7
click-didyoumean                  0.3.1
cloudpickle                       3.1.0
colorama                          0.4.6
DataProperty                      1.0.1
datasets                          2.21.0
deepsearch-glm                    0.26.1
deepspeed                         0.15.2
dill                              0.3.8
diskcache                         5.6.3
distro                            1.9.0
docker-pycreds                    0.4.0
docling                           2.5.2
docling-core                      2.3.2
docling-ibm-models                2.0.3
docling-parse                     2.0.4
docutils                          0.21.2
einops                            0.8.0
evaluate                          0.4.3
fastapi                           0.115.5
filelock                          3.16.1
filetype                          1.2.0
flash_attn                        2.6.3
frozenlist                        1.5.0
fsspec                            2024.5.0
gguf                              0.10.0
gitdb                             4.0.11
GitPython                         3.1.43
h11                               0.14.0
hjson                             3.1.0
httpcore                          1.0.7
httptools                         0.6.4
httpx                             0.27.2
huggingface_hub                   0.26.2
idna                              3.10
importlib_metadata                8.5.0
instructlab                       0.21.0
instructlab-dolomite              0.2.0
instructlab-eval                  0.4.1
instructlab-quantize              0.1.0
instructlab-schema                0.4.1
instructlab-sdg                   0.6.1
instructlab-training              0.6.1
interegular                       0.3.3
Jinja2                            3.1.4
jiter                             0.7.1
joblib                            1.4.2
jsonlines                         3.1.0
jsonpatch                         1.33
jsonpointer                       3.0.0
jsonref                           1.1.0
jsonschema                        4.23.0
jsonschema-specifications         2024.10.1
langchain-core                    0.3.19
langchain-text-splitters          0.3.2
langsmith                         0.1.143
lark                              1.2.2
llama_cpp_python                  0.2.79
llvmlite                          0.43.0
lm_eval                           0.4.5
lm-format-enforcer                0.10.6
lxml                              4.9.4
markdown-it-py                    3.0.0
marko                             2.1.2
MarkupSafe                        3.0.2
mbstrdecoder                      1.1.3
mdurl                             0.1.2
mean_average_precision            2021.4.26.0
mistral_common                    1.5.0
more-itertools                    10.5.0
mpmath                            1.3.0
msgpack                           1.1.0
msgspec                           0.18.6
multidict                         6.1.0
multiprocess                      0.70.16
nest_asyncio                      1.6.0
networkx                          3.4.2
ninja                             1.11.1.1
nltk                              3.9.1
numba                             0.60.0
numexpr                           2.10.1
numpy                             1.26.4
nvidia-ml-py                      12.560.30
openai                            1.54.4
opencv-python-headless            4.10.0.84
orjson                            3.10.11
outlines                          0.0.46
packaging                         24.2
pandas                            2.2.1
pandas-stubs                      2.2.3.241009
partial-json-parser               0.2.1.1.post4
pathspec                          0.12.1
pathvalidate                      3.2.1
peft                              0.13.2
pillow                            10.4.0
pip                               22.3.1
portalocker                       2.10.1
prometheus_client                 0.21.0
prometheus-fastapi-instrumentator 7.0.0
prompt_toolkit                    3.0.48
propcache                         0.2.0
protobuf                          4.25.5
psutil                            6.1.0
py-cpuinfo                        9.0.0
pyairports                        2.1.1
pyarrow                           16.1.0
pybind11                          2.13.6
pycountry                         24.6.1
pydantic                          2.9.2
pydantic_core                     2.23.4
pydantic-settings                 2.6.1
pydantic_yaml                     1.4.0
Pygments                          2.18.0
pypdfium2                         4.30.0
pytablewriter                     1.2.0
python-dateutil                   2.9.0.post0
python-docx                       1.1.2
python-dotenv                     1.0.1
python-pptx                       1.0.2
pytz                              2024.2
PyYAML                            6.0.2
pyzmq                             26.2.0
referencing                       0.35.1
regex                             2024.11.6
requests                          2.32.3
requests-toolbelt                 1.0.0
rich                              13.9.4
rouge_score                       0.1.2
rpds-py                           0.18.1
Rtree                             1.3.0
ruamel.yaml                       0.18.6
ruamel.yaml.clib                  0.2.12
sacrebleu                         2.4.3
safetensors                       0.4.5
scikit-learn                      1.5.2
scipy                             1.14.1
sentencepiece                     0.2.0
sentry-sdk                        2.18.0
setproctitle                      1.3.3
setuptools                        75.5.0
shellingham                       1.5.4
shortuuid                         1.0.13
six                               1.16.0
smmap                             5.0.1
sniffio                           1.3.1
soupsieve                         2.6
sqlitedict                        2.1.0
sse-starlette                     2.1.3
starlette                         0.41.2
starlette-context                 0.3.6
sympy                             1.13.3
tabledata                         1.3.3
tabulate                          0.9.0
tcolorpy                          0.1.6
tenacity                          9.0.0
tesserocr                         2.7.1
threadpoolctl                     3.5.0
tiktoken                          0.7.0
tokenizers                        0.20.3
toml                              0.10.2
torch                             2.4.1
torchvision                       0.19.0
tqdm                              4.67.0
tqdm-multiprocess                 0.0.11
transformers                      4.46.2
triton                            3.0.0
trl                               0.12.1
typepy                            1.3.2
typer                             0.12.5
types-pytz                        2024.2.0.20241003
typing_extensions                 4.12.2
tzdata                            2024.2
urllib3                           2.2.3
uvicorn                           0.32.0
uvloop                            0.21.0
vllm                              0.6.2
wandb                             0.16.6
watchfiles                        0.24.0
wcwidth                           0.2.13
websockets                        14.1
wheel                             0.45.0
word2number                       1.1
xdg-base-dirs                     6.0.2
xformers                          0.0.27.post2
XlsxWriter                        3.2.0
xxhash                            3.5.0
yamllint                          1.35.1
yarl                              1.17.1
zipp                              3.21.0
zstandard                         0.23.0

[bbrowning]

We discussed this on Slack as well, and just adding a note here that this looks likely related to vllm-project/vllm#10234 but it hasn't been definitively determined if it's that exact issue yet or not.

[relyt0925]

I have seen granite-7b-starter fail as well logs attached below

granite7bfailout.txt.zip

[RobotSail]

So the reason we are getting the error is similar to what @bbrowning found - we are getting logprobs values of nan which the starlette library doesn't properly serialize into JSON as the native library does. But the underlying root cause seems to be that Granite 8B is returning nan logprobs when a certain prompt template is used in the first place.