Spaces:
Sleeping
Sleeping
Kewen Zhao
commited on
Commit
·
f5dea60
1
Parent(s):
0c0f1e7
update readme and description
Browse files- README.md +1 -1
- code_eval_stdio.py +5 -4
README.md
CHANGED
|
@@ -16,7 +16,7 @@ description: >-
|
|
| 16 |
(https://arxiv.org/abs/2107.03374).
|
| 17 |
---
|
| 18 |
|
| 19 |
-
# Metric Card for Code Eval
|
| 20 |
|
| 21 |
## Metric description
|
| 22 |
|
|
|
|
| 16 |
(https://arxiv.org/abs/2107.03374).
|
| 17 |
---
|
| 18 |
|
| 19 |
+
# Metric Card for Code Eval StdIO
|
| 20 |
|
| 21 |
## Metric description
|
| 22 |
|
code_eval_stdio.py
CHANGED
|
@@ -76,10 +76,11 @@ Returns:
|
|
| 76 |
pass_at_k: dict with pass rates for each k
|
| 77 |
results: dict with granular results of each unittest
|
| 78 |
Examples:
|
| 79 |
-
>>>
|
| 80 |
-
>>>
|
| 81 |
-
>>>
|
| 82 |
-
>>>
|
|
|
|
| 83 |
>>> print(pass_at_k)
|
| 84 |
{'pass@1': 0.5, 'pass@2': 1.0}
|
| 85 |
"""
|
|
|
|
| 76 |
pass_at_k: dict with pass rates for each k
|
| 77 |
results: dict with granular results of each unittest
|
| 78 |
Examples:
|
| 79 |
+
>>> code_eval_stdio = evaluate.load("hage2000/code_eval_stdio")
|
| 80 |
+
>>> inputs = ["2 3"]
|
| 81 |
+
>>> references = ["5"]
|
| 82 |
+
>>> candidates = [[ "nums = list(map(int, input().split()))\nprint(sum(nums))"]]
|
| 83 |
+
>>> pass_at_k, results = code_eval_stdio.compute(references=references, predictions=candidates, inputs = inputs, k=[1, 2])
|
| 84 |
>>> print(pass_at_k)
|
| 85 |
{'pass@1': 0.5, 'pass@2': 1.0}
|
| 86 |
"""
|