Demo enhancement - #91
Conversation
| First, clone the repo: | ||
|
|
||
| ```{code-cell} bash | ||
| ```{code-block} bash |
There was a problem hiding this comment.
no. I used code-cell on purpose for rendering style
There was a problem hiding this comment.
What should be done for the cells to render? They weren't rendering on my end when I was reviewing the docs
There was a problem hiding this comment.
oh, if they't not rendering then maybe the files need different yaml header to render right. code-cell renders in md notebooks
| ```{code-cell} bash | ||
|
|
||
| ::::{note} | ||
| The default runner is ollama. Make sure ollama is running by calling `ollama serve` on another terminal window |
There was a problem hiding this comment.
| The default runner is ollama. Make sure ollama is running by calling `ollama serve` on another terminal window | |
| The default runner is `ollama`. Make sure ollama is running in advance. |
I'm pretty sure I have never run ollama serve so I don't think that was completely accurate. I'd prefer to add a link here to ollama docs for how to make sure it's running. (across platforms)
| <!-- Doesn't really run anything | ||
| File "/work/pi_brownsarahm_uri_edu/ayman_uri/BenchTools/benchtools/benchtools/task.py", line 497, in run | ||
| for (prompt_id, prompt),values in zip(id_prompt_list,self.variant_values): | ||
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | ||
| TypeError: 'NoneType' object is not iterable | ||
| --> |
There was a problem hiding this comment.
| <!-- Doesn't really run anything | |
| File "/work/pi_brownsarahm_uri_edu/ayman_uri/BenchTools/benchtools/benchtools/task.py", line 497, in run | |
| for (prompt_id, prompt),values in zip(id_prompt_list,self.variant_values): | |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
| TypeError: 'NoneType' object is not iterable | |
| --> |
Please do not include error messages unless necessary and please never include absolute paths in any commit
| There are multiple ways to creating a Task object | ||
| ``` | ||
| add_task = Task.from_txt_csv('../../demos/folderbench/tasks/add') | ||
| add_task = Task.from_txt_csv('benchtools/assets/demos/folderbench/tasks/add') | ||
| tiny_bench.add_task(add_task) |
There was a problem hiding this comment.
this was a bad example because it only works if they install after cloning; now that we have the demos installing, we should update this accordingly.
| rm -rf tiniest_demo | ||
| ``` | ||
|
|
||
| <!-- Same problem with run --> |
|
|
||
| ```{code-cell} | ||
| pre_built_yml = Bench.from_yaml('../../demos/listbench') | ||
| pre_built_yml = Bench.from_yaml('benchtools/assets/demos/listbench') |
| ``` | ||
| [{'a': 2, 'b': 3}, {'a': 3, 'b': 4}, {'a': 5, 'b': 5}] | ||
| ``` |
There was a problem hiding this comment.
what is this? I do not want manual output if at all avoidable
| [{'a': 2, 'b': 3}, {'a': 3, 'b': 4}, {'a': 5, 'b': 5}] | ||
| ``` | ||
|
|
||
| Make sure you have `ollama serve` running to run the benchmark |
There was a problem hiding this comment.
again check that this is correct
| --> | ||
|
|
||
|
|
||
| Logs will be found in `benchtools/assets/demos/listbench/logs` |
There was a problem hiding this comment.
Plse do not let this happen
brownsarahm
left a comment
brownsarahm
left a comment
There was a problem hiding this comment.
style fixes are sitll needed
| benchtool score . | ||
| ``` | ||
|
|
||
| ```{code-block} consol |
There was a problem hiding this comment.
| ```{code-block} consol | |
| ```{code-block} console |
| Let's run the scorer to score the LLM on the task at hand. | ||
|
|
||
| ```{code-block} bash | ||
| :tags: ["skip-execution"] |
There was a problem hiding this comment.
this is only for code-cells; so remove if using code-block on purpose
| ls logs/gemma3/animal/1780063281 | ||
| animal_spots-hairy-paws animal_stripes-hairy-hooves animal_stripes-hairy-paws run_info.yml |
There was a problem hiding this comment.
dont'e mix input and output
2 participants
Some fixes that I catch as I work