Add Automated Simulatability for Attribution Methods (5 bis) - #145
fanny-jourdan wants to merge 17 commits into
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
馃挕 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0a275b63e5
鈩癸笍 About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 馃憤.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| classes_ids = sorted(corresponding_predictions.unique().tolist()) | ||
| classes = {class_id: self.classes[class_id] for class_id in classes_ids} |
There was a problem hiding this comment.
The prompt builder derives classes only from corresponding_predictions, but the contrastive branch later reads classes[gold_index]. When a learning sample is misclassified into a predicted class set that does not include its true label (e.g., a collapsed model that never predicts one class), construct_prompt raises KeyError and the whole experiment fails. This can occur with valid inputs, so the class map should include label-only classes (or be validated before use).
Useful? React with 馃憤聽/ 馃憥.
|
|
||
| # Fallback: look for one explicit class name in the whole response. | ||
| # This keeps scoring tolerant to formats like "Label: positive". | ||
| class_hits = [class_name for class_name in set(model_predictions) if class_name.lower() in raw_prediction] |
There was a problem hiding this comment.
The new fallback scoring path treats any substring occurrence as a class hit, which can mis-score outputs when labels overlap or appear inside unrelated text. For example, with classes like toxic/non-toxic, a correct answer can produce multiple hits and be marked wrong; with short labels like no, incidental text such as "I don't know" can be counted as a prediction. This biases simulatability scores and should use boundary-aware/exact label matching.
Useful? React with 馃憤聽/ 馃憥.
AntoninPoche
left a comment
AntoninPoche
left a comment
There was a problem hiding this comment.
Nice Work, thank you very much @fanny-jourdan !!
I left a few comments on the LLM interface; it is too complex, IMO. I would leave more of this work to the users. We cannot anticipate all the weird model setups.
On AttrSim, I have mainly open questions about how to visualize attributions.
In any case, this is more than enough for what we need on the contrastive part.
| try: | ||
| return ModelWithSplitPoints( | ||
| "hf-internal-testing/tiny-random-bert", | ||
| split_points=["bert.encoder.layer.1.output"], | ||
| automodel=AutoModelForMaskedLM, # type: ignore | ||
| ) | ||
| except OSError as exc: | ||
| pytest.skip(f"Skipping llm_labels tests: unable to load tiny-random-bert ({exc})") |
There was a problem hiding this comment.
We should not skip if we cannot load tiny-random-bert with the ModelWithSplitPoints. It would be a huge problem in the library if it did not work.
Additionally, the fixture is defined in tests/conftest.py and does not need to be created anew.
|
|
||
| To evaluate attribution methods faithfulness, there are the [`Insertion`](https://for-sight-ai.github.io/interpreto/api/attributions/metrics/insertion/) and [`Deletion`](https://for-sight-ai.github.io/interpreto/api/attributions/metrics/deletion/) metrics. | ||
|
|
||
| Attribution methods can also be evaluated globally via [`Automated Simulatability`](https://for-sight-ai.github.io/interpreto/api/attributions/metrics/attrsim/). |
There was a problem hiding this comment.
If you modify the README.md, do not forget the docs/index.md
There was a problem hiding this comment.
it's already done :)
| try: | ||
| self.model = AutoModelForCausalLM.from_pretrained( | ||
| model, | ||
| dtype="auto", | ||
| device_map=device, | ||
| ) | ||
| except TypeError: | ||
| self.model = AutoModelForCausalLM.from_pretrained( | ||
| model, | ||
| torch_dtype="auto", | ||
| device_map=device, | ||
| ) | ||
|
|
||
| if self.tokenizer.pad_token_id is None: | ||
| self.tokenizer.pad_token = self.tokenizer.eos_token | ||
|
|
||
| if device == "auto": | ||
| # model_device = getattr(self.model, "device", None) | ||
| model_device = self._resolve_model_device() | ||
| if model_device is None: | ||
| model_device = next(self.model.parameters()).device | ||
| self._inputs_device = model_device | ||
| else: | ||
| self._inputs_device = torch.device(device) | ||
|
|
||
| def _resolve_model_device(self) -> torch.device | None: | ||
| hf_device_map = getattr(self.model, "hf_device_map", None) | ||
| if isinstance(hf_device_map, dict): | ||
| for target in hf_device_map.values(): | ||
| if isinstance(target, int): | ||
| return torch.device(f"cuda:{target}") | ||
| if isinstance(target, str): | ||
| if target in {"disk", "meta"}: | ||
| continue | ||
| return torch.device(target) | ||
|
|
||
| model_device = getattr(self.model, "device", None) | ||
| if model_device is not None and str(model_device) != "meta": | ||
| return torch.device(model_device) | ||
|
|
||
| return None |
There was a problem hiding this comment.
I do not think we should try such a complex device resolving.
If loading the model each time is a pain, we can just ask the user to provide the model and tokenizer instances. Also, it prevents future bugs and leaves it to the user side.
| tokenizer_max_length = self._compute_tokenizer_max_length(generation_kwargs) | ||
| inputs = self.tokenizer( | ||
| batch_prompts, | ||
| return_tensors="pt", | ||
| padding=True, | ||
| truncation=True, | ||
| ).to(self.device) | ||
| max_length=tokenizer_max_length, | ||
| ).to(self._inputs_device) |
There was a problem hiding this comment.
It seems weird to me to resolve the max length like this.
I suggest we put in parameters tokenizer_kwargs: dict = {} and generation_kwargs: dict = {}.
Then, we update our default parameters with these two dictionaries and pass them to the tokenizer and model.
It gives the user more flexibility, since LLMs can have weird, specific parameters.
Simultaneously, you still put 32 as the default value.
| # if llm_pred.split(" ")[0].lower() == ref_pred.lower(): | ||
| raw_prediction = llm_pred.strip().lower() | ||
| if not raw_prediction: | ||
| continue | ||
|
|
||
| first_token = raw_prediction.split(" ")[0] | ||
| first_token = re.sub(r"^[^a-z0-9_-]+|[^a-z0-9_-]+$", "", first_token) | ||
|
|
||
| if first_token == ref_pred.lower(): | ||
| score += 1 | ||
| continue | ||
|
|
||
| # Fallback: look for one explicit class name in the whole response. | ||
| # This keeps scoring tolerant to formats like "Label: positive". | ||
| class_hits = [class_name for class_name in set(model_predictions) if class_name.lower() in raw_prediction] | ||
| if len(class_hits) == 1 and class_hits[0].lower() == ref_pred.lower(): |
There was a problem hiding this comment.
With my trials and errors, in the contrastive code I use:
pred =text.strip().replace("\n", " ").split(" ")[-1]
We need to take the last token because reasoning models have their chain of thoughts first.
The method was outdated because I do not call it in the contrastive code (maybe I should.)
To what extent did LLM write this function, and how much of it is really necessary? I do not know how to read regex; I tend not to trust them.
There was a problem hiding this comment.
I asked Codex to get the code working because the version you gave me wasn't working. Thanks to that, I didn't have any more issues, so I figured it was necessary, but maybe there was some extra code not necessary usefull
| def construct_prompt( # type: ignore | ||
| self, | ||
| setting: PromptTypes | PromptSetting, | ||
| interesting_samples: list[str], | ||
| corresponding_predictions: torch.Tensor, | ||
| corresponding_labels: torch.Tensor, | ||
| nb_learning_samples: int, | ||
| *, | ||
| corresponding_attribution: list[AttributionOutput], | ||
| ) -> tuple[str, list[str], list[str]]: |
| def _format_attr_vector( | ||
| elements: list[str] | torch.Tensor, | ||
| attr_vector: torch.Tensor, | ||
| top_k: int = 6, | ||
| ) -> str: | ||
| if isinstance(elements, torch.Tensor): | ||
| elements = [str(e.item()) for e in elements] | ||
| else: | ||
| elements = [str(e) for e in elements] | ||
|
|
||
| top_k = min(top_k, attr_vector.shape[-1]) | ||
| top_indices = torch.topk(attr_vector.abs(), k=top_k).indices.tolist() | ||
|
|
||
| pieces = [] | ||
| for idx in top_indices: | ||
| token = elements[idx] if idx < len(elements) else f"tok_{idx}" | ||
| pieces.append(f"{token}: {attr_vector[idx].item():+.3f}") | ||
| return "{" + ", ".join(pieces) + "}" |
There was a problem hiding this comment.
There might be many ways to verbalize attributions; yours works well. But did you experiment or imagine any other form?
Also, you give 3 digits, which is quite arbitrary, attributions might be under one thousandth or really high. So this might not be adapted. Maybe just giving the top k tokens without values is enough.
Final point, you ignore negative values, is it voluntary?
| system_prompt_parts = [ | ||
| "You are a classifier. Predict the class for each evaluation sample.", | ||
| "Use the provided learning examples and attribution explanations to infer the model behavior.", | ||
| "Only return the class name, no additional text.", | ||
| f"The classes are: [{', '.join(list(classes.values()))}]", | ||
| ] |
There was a problem hiding this comment.
With this, you also talk about attributions for the baseline. But do not mention what contrastive explanations correspond to in the case of contrastive.
AntoninPoche
changed the title
AntoninPoche
changed the title
This branch has not been deployed
2 participants
Description
HuggingFaceLLMclassType of Change
Checklist
CODE_OF_CONDUCT.mddocument.CONTRIBUTING.mdguide.make lint.make test.