Make --allow-redefinition mean --allow-redefinition-new

[JukkaL]

Also now --allow-redefinition is the primary flag name for the previous --allow-redefinition-new flag, and the --allow-redefinition-new name is a (soft deprecated) alias.

The original --allow-redefinition behavior is still available via --allow-redefinition-old.

Also removed some now redundant --local-partial-types flags in tests.

[github-actions]

Diff from mypy_primer, showing the effect of this PR on open source code:

spack (https://github.com/spack/spack)
+ lib/spack/spack/llnl/util/filesystem.py:1840: error: Argument 2 to "_log_file_access_issue" has incompatible type "str | Path"; expected "str"  [arg-type]
+ lib/spack/spack/llnl/util/filesystem.py:1844: error: Argument 1 to "appendleft" of "deque" has incompatible type "tuple[int, str | Path]"; expected "tuple[int, str]"  [arg-type]
+ lib/spack/spack/config.py:486: error: Unused "type: ignore" comment  [unused-ignore]
+ lib/spack/spack/util/web.py:83: error: Unused "type: ignore" comment  [unused-ignore]
+ lib/spack/spack/spec.py:4298: error: Item "VariantValue" of "Spec | Any | VariantValue" has no attribute "versions"  [union-attr]
+ lib/spack/spack/spec.py:4311: error: Item "VariantValue" of "Spec | Any | VariantValue" has no attribute "original_spec_format"  [union-attr]
+ lib/spack/spack/url_buildcache.py:1175: error: Need type annotation for "filename_to_mtime"  [var-annotated]
+ lib/spack/spack/solver/requirements.py:320: error: Argument 1 to "_parse_and_expand" of "RequirementParser" has incompatible type "Any | list[str]"; expected "str"  [arg-type]
+ lib/spack/spack/solver/requirements.py:336: error: Argument "message" to "RequirementRule" has incompatible type "Any | list[str] | None"; expected "str | None"  [arg-type]

steam.py (https://github.com/Gobot1234/steam.py)
- steam/id.py:91: error: Incompatible types in assignment (expression has type "int", variable has type "Type | None")  [assignment]
- steam/id.py:92: error: Incompatible types in assignment (expression has type "int", variable has type "Universe | None")  [assignment]
- steam/id.py:95: error: Incompatible types in assignment (expression has type "int", variable has type "Instance | None")  [assignment]
- steam/id.py:97: error: Unsupported operand types for <= ("int" and "None")  [operator]
- steam/id.py:97: note: Right operand is of type "Universe | None"
- steam/id.py:97: error: Unsupported operand types for > ("int" and "None")  [operator]
- steam/id.py:97: note: Left operand is of type "Universe | None"
- steam/id.py:99: error: Unsupported operand types for <= ("int" and "None")  [operator]
- steam/id.py:99: note: Right operand is of type "Type | None"
- steam/id.py:99: error: Unsupported operand types for > ("int" and "None")  [operator]
- steam/id.py:99: note: Left operand is of type "Type | None"
- steam/id.py:101: error: Unsupported operand types for <= ("int" and "None")  [operator]
- steam/id.py:101: note: Right operand is of type "Instance | None"
- steam/id.py:101: error: Unsupported operand types for > ("int" and "None")  [operator]
- steam/id.py:101: note: Left operand is of type "Instance | None"
- steam/id.py:111: error: Unsupported operand types for << ("None" and "int")  [operator]
- steam/id.py:111: note: Left operand is of type "Universe | None"
- steam/id.py:111: note: Left operand is of type "Type | None"
- steam/id.py:111: note: Left operand is of type "Instance | None"
- steam/id.py:482: error: Incompatible types in assignment (expression has type "int", variable has type "Instance")  [assignment]
- steam/id.py:486: error: Incompatible types in assignment (expression has type "int", variable has type "Instance")  [assignment]
- steam/id.py:488: error: Incompatible types in assignment (expression has type "int", variable has type "Instance")  [assignment]
- steam/id.py:490: error: Incompatible types in assignment (expression has type "int", variable has type "Instance")  [assignment]
- steam/id.py:492: error: Incompatible types in assignment (expression has type "int", variable has type "Instance")  [assignment]
+ steam/id.py:494: error: Argument "instance" to "ID" has incompatible type "int"; expected "Instance | None"  [arg-type]
- steam/id.py:558: error: Item "None" of "Any | None" has no attribute "get"  [union-attr]
- steam/protobufs/struct_messages.py:24: error: Incompatible types in assignment (expression has type "tuple[()]", variable has type "dict[str, Any]")  [assignment]
- steam/protobufs/msg.py:229: error: Incompatible types in assignment (expression has type "int", variable has type "EMsg")  [assignment]
+ steam/protobufs/msg.py:241: error: Argument 1 to "__init_subclass__" of "MessageMessageBase" has incompatible type "int"; expected "EMsg"  [arg-type]
- steam/app.py:388: error: Incompatible types in assignment (expression has type "OwnershipTicket", variable has type "AuthenticationTicket")  [assignment]
- steam/app.py:1090: error: Incompatible types in assignment (expression has type "PublishedFile[PartialUser]", variable has type "PublishedFileDetails")  [assignment]
- steam/abc.py:211: error: Incompatible types in assignment (expression has type "steam.comment.Comment[Self, PartialUser]", variable has type "steam.protobufs.community.GetCommentThreadResponse.Comment | None")  [assignment]
- steam/abc.py:668: error: Unsupported operand types for > ("datetime" and "None")  [operator]
- steam/abc.py:668: note: Left operand is of type "datetime | None"
- steam/abc.py:668: error: Unsupported operand types for < ("datetime" and "None")  [operator]
- steam/abc.py:668: note: Right operand is of type "datetime | None"
- steam/abc.py:796: error: Incompatible types in assignment (expression has type "PublishedFile[Self]", variable has type "PublishedFileDetails")  [assignment]
- steam/gateway.py:830: error: No overload variant of "wait_for" of "SteamWebSocket" matches argument types "int", "Callable[[UnifiedMsgT], bool] | Callable[[Any], Any]"  [call-overload]
+ steam/gateway.py:830: error: No overload variant of "wait_for" of "SteamWebSocket" matches argument types "int", "Callable[[UnifiedMsgT], bool]"  [call-overload]
- steam/state.py:464: error: Argument 2 to "pop" of "dict" has incompatible type "None"; expected "Clan"  [arg-type]
- steam/state.py:1038: error: Incompatible types in assignment (expression has type "Event[EventType, Clan]", variable has type "CMsgClientClanStateEvent")  [assignment]
- steam/state.py:1042: error: Incompatible types in assignment (expression has type "Announcement[Clan]", variable has type "CMsgClientClanStateEvent")  [assignment]
- steam/state.py:1208: error: Incompatible types in assignment (expression has type "PartialMember | None", variable has type "GroupMember | ClanMember")  [assignment]
- steam/state.py:1456: error: Argument 2 to "pop" of "dict" has incompatible type "None"; expected "ClanInvite | GroupInvite"  [arg-type]
- steam/state.py:1750: error: Incompatible types in assignment (expression has type "User", variable has type "int")  [assignment]
- steam/state.py:1753: error: Incompatible types in assignment (expression has type "int", variable has type "User")  [assignment]
- steam/state.py:1754: error: Incompatible types in assignment (expression has type "int", variable has type "User")  [assignment]
- steam/state.py:2372: error: Incompatible types in assignment (expression has type "PartialClan", variable has type "PartialUser")  [assignment]
- steam/state.py:2374: error: Incompatible types in assignment (expression has type "Event[EventType, PartialClan]", variable has type "PartialUser")  [assignment]
- steam/state.py:2376: error: Incompatible types in assignment (expression has type "Announcement[PartialClan]", variable has type "PartialUser")  [assignment]
- steam/state.py:2378: error: Incompatible types in assignment (expression has type "PublishedFile[Any | ClientUser | PartialUser] | None", variable has type "PartialUser")  [assignment]
- steam/state.py:2383: error: Incompatible types in assignment (expression has type "Review", variable has type "PartialUser")  [assignment]
- steam/state.py:2385: error: Incompatible types in assignment (expression has type "Post[PartialUser]", variable has type "PartialUser")  [assignment]
- steam/client.py:893: error: Incompatible types in assignment (expression has type "list[GameServer]", variable has type "list[IPsWithSteamIDsResponseServer]")  [assignment]
- steam/client.py:1285: error: Incompatible types in assignment (expression has type "float", variable has type "int")  [assignment]
- steam/ext/commands/commands.py:285: error: Incompatible types in assignment (expression has type "GroupMixin[Any | None] | Group[CogT, [VarArg(Any), KwArg(Any)], Any] | None", variable has type "Self")  [assignment]
+ steam/ext/commands/commands.py:831: error: Incompatible return value type (got "Any | Command[Any, Any, Any]", expected "C")  [return-value]

pwndbg (https://github.com/pwndbg/pwndbg)
+ pwndbg/dbg_mod/gdb/__init__.py: note: In member "cast" of class "GDBValue":
+ pwndbg/dbg_mod/gdb/__init__.py:1556: error: Name "type" already defined on line 1554  [no-redef]
+ pwndbg/dbg_mod/gdb/__init__.py: note: At top level:
+ pwndbg/dintegration/__init__.py: note: In member "connect" of class "IntegrationManager":
+ pwndbg/dintegration/__init__.py:509: error: Argument 1 to "DecompilerConnection" has incompatible type "_Method"; expected "ServerProxy"  [arg-type]
+ pwndbg/aglib/godbg.py:851: error: Argument 1 to "fmt_bytes" of "FormatOpts" has incompatible type "bytes | bytearray"; expected "bytes"  [arg-type]
+ pwndbg/commands/procinfo.py: note: In member "status" of class "Process":
+ pwndbg/commands/procinfo.py:183: error: Argument 1 to "append" of "list" has incompatible type "str"; expected "int"  [arg-type]
+ pwndbg/commands/kmod.py: note: In function "kmod":
+ pwndbg/commands/kmod.py:95: error: Argument 2 to "add_symbol_file" of "Process" has incompatible type "object"; expected "int | None"  [arg-type]
+ pwndbg/hexdump.py: note: In function "load_color_scheme":
+ pwndbg/hexdump.py:57: error: Unsupported target for indexed assignment ("None")  [index]
+ pwndbg/hexdump.py:58: error: Unsupported target for indexed assignment ("None")  [index]
+ pwndbg/hexdump.py:63: error: Unsupported target for indexed assignment ("None")  [index]
+ pwndbg/hexdump.py:64: error: Unsupported target for indexed assignment ("None")  [index]
+ pwndbg/hexdump.py:67: error: Unsupported target for indexed assignment ("None")  [index]
+ pwndbg/hexdump.py:68: error: Unsupported target for indexed assignment ("None")  [index]
+ pwndbg/hexdump.py:70: error: Unsupported target for indexed assignment ("None")  [index]
+ pwndbg/hexdump.py:71: error: Unsupported target for indexed assignment ("None")  [index]
+ pwndbg/hexdump.py: note: In function "hexdump":
+ pwndbg/hexdump.py:143: error: Value of type "None" is not indexable  [index]
+ pwndbg/hexdump.py:145: error: Value of type "None" is not indexable  [index]
+ pwndbg/hexdump.py:152: error: Value of type "None" is not indexable  [index]
+ pwndbg/commands/ptmalloc2.py:1589: error: Unused "type: ignore" comment  [unused-ignore]
+ pwndbg/dbg_mod/lldb/__init__.py: note: In member "cast" of class "LLDBValue":
+ pwndbg/dbg_mod/lldb/__init__.py:784: error: Name "type" already defined on line 782  [no-redef]
+ pwndbg/dbg_mod/lldb/__init__.py: note: In member "vmmap" of class "LLDBProcess":
+ pwndbg/dbg_mod/lldb/__init__.py:1153: error: Name "pages" already defined on line 1143  [no-redef]
+ pwndbg/dbg_mod/lldb/__init__.py: note: In member "create_value" of class "LLDBProcess":
+ pwndbg/dbg_mod/lldb/__init__.py:1480: error: Name "u64" already defined on line 1476  [no-redef]
+ pwndbg/dbg_mod/lldb/__init__.py: note: In member "arch" of class "LLDBProcess":
+ pwndbg/dbg_mod/lldb/__init__.py:1866: error: Argument "name" to "ArchDefinition" has incompatible type "Any | str"; expected "Literal['x86-64', 'i386', 'i8086', 'mips', 'aarch64', 'arm', 'armcm', 'rv32', 'rv64', 'sparc', 'powerpc', 'loongarch64', 's390x', 'hexagon']"  [arg-type]
+ pwndbg/dbg_mod/lldb/repl/__init__.py: note: In function "parse":
+ pwndbg/dbg_mod/lldb/repl/__init__.py:819: error: Need type annotation for "raw"  [var-annotated]
+ pwndbg/dbg_mod/lldb/repl/__init__.py: note: At top level:

egglog-python (https://github.com/egraphs-good/egglog-python)
+ python/egglog/ipython_magic.py:41: error: Item "EGraph" of "Any | EGraph" has no attribute "to_graphviz_string"  [union-attr]
+ python/egglog/egraph.py:471: error: Unused "type: ignore" comment  [unused-ignore]
+ python/egglog/egraph.py:471: error: Item "function" of "Callable[..., Any] | Any" has no attribute "__wrapped__"  [union-attr]
+ python/egglog/egraph.py:471: note: Error code "union-attr" not covered by "type: ignore[attr-defined]" comment

Tanjun (https://github.com/FasterSpeeding/Tanjun)
+ tanjun/permissions.py:231: error: Incompatible types in assignment (expression has type "Mapping[Snowflake, Role] | Cache | None", variable has type "Mapping[Snowflake, Role] | None")  [assignment]
- tanjun/dependencies/limiters.py:812: error: Incompatible types in assignment (expression has type "_Cooldown", variable has type "AbstractCooldownBucket | None")  [assignment]
- tanjun/dependencies/limiters.py:814: error: Item "AbstractCooldownBucket" of "AbstractCooldownBucket | None" has no attribute "check"  [union-attr]
- tanjun/dependencies/limiters.py:814: error: Item "None" of "AbstractCooldownBucket | None" has no attribute "check"  [union-attr]
- tanjun/dependencies/limiters.py:850: error: Incompatible types in assignment (expression has type "_Cooldown | None", variable has type "AbstractCooldownBucket | None")  [assignment]
- tanjun/commands/slash.py:3222: error: Incompatible types in assignment (expression has type "def (AutocompleteContext, float, /, *Any, **Any) -> Coroutine[Any, Any, None] | None", variable has type "def (AutocompleteContext, str, /, *Any, **Any) -> Coroutine[Any, Any, None] | None")  [assignment]
- tanjun/commands/slash.py:3225: error: Incompatible types in assignment (expression has type "def (AutocompleteContext, int, /, *Any, **Any) -> Coroutine[Any, Any, None] | None", variable has type "def (AutocompleteContext, str, /, *Any, **Any) -> Coroutine[Any, Any, None] | None")  [assignment]
- tanjun/commands/slash.py:3235: error: Argument 1 to "call_with_async_di" of "Context" has incompatible type "def (AutocompleteContext, str, /, *Any, **Any) -> Coroutine[Any, Any, None]"; expected "Callable[..., Coroutine[Any, Any, Never] | Never]"  [arg-type]
+ tanjun/commands/slash.py:3235: error: Argument 1 to "call_with_async_di" of "Context" has incompatible type "def (AutocompleteContext, str, /, *Any, **Any) -> Coroutine[Any, Any, None] | def (AutocompleteContext, float, /, *Any, **Any) -> Coroutine[Any, Any, None] | def (AutocompleteContext, int, /, *Any, **Any) -> Coroutine[Any, Any, None]"; expected "Callable[..., Coroutine[Any, Any, Never] | Never]"  [arg-type]

pandera (https://github.com/pandera-dev/pandera)
- pandera/engines/utils.py:73: error: Unused "type: ignore[call-overload]" comment  [unused-ignore]
+ pandera/engines/utils.py:73: error: Unused "type: ignore[union-attr, call-overload]" comment  [unused-ignore]
+ pandera/engines/pandas_engine.py:268: error: Name "pandera_dtype" already defined on line 265  [no-redef]
+ pandera/engines/pandas_engine.py:1457: error: Unused "type: ignore" comment  [unused-ignore]
- pandera/typing/pandas.py:211: error: Unused "type: ignore" comment  [unused-ignore]
+ pandera/typing/pandas.py:238: error: Unused "type: ignore" comment  [unused-ignore]
- pandera/typing/pandas.py:249: error: Unused "type: ignore" comment  [unused-ignore]
+ pandera/io/xarray_io.py:248: error: Need type annotation for "data_vars"  [var-annotated]
+ pandera/backends/pandas/container.py:774: error: Unused "type: ignore" comment  [unused-ignore]
+ pandera/backends/pandas/container.py:776: error: Incompatible return value type (got "T | Any | None", expected "T")  [return-value]
- pandera/backends/pandas/array.py:454: error: Incompatible types in assignment (expression has type "str", variable has type "Series[Any] | DataFrame | None")  [assignment]
- pandera/backends/pandas/array.py:456: error: Incompatible types in assignment (expression has type "str", variable has type "Series[Any] | DataFrame | None")  [assignment]
- pandera/api/pandas/array.py:251: error: Redundant cast to "Series[Any]"  [redundant-cast]
+ pandera/io/pandas_io.py:994: error: Incompatible types in assignment (expression has type "dict[str, str]", target has type "str")  [assignment]
- pandera/backends/pandas/components.py:980: error: Incompatible types in assignment (expression has type "Hashable", variable has type "str")  [assignment]
+ pandera/decorators.py:397: error: Invalid index type "str | int" for "list[Any]"; expected type "SupportsIndex"  [index]
+ pandera/decorators.py:401: error: No return value expected  [return-value]
+ pandera/api/pandas/model.py:196: error: Unused "type: ignore" comment  [unused-ignore]
+ tests/pandas/test_schema_statistics.py:46: error: Unused "type: ignore" comment  [unused-ignore]
+ tests/pandas/test_schema_inference.py:26: error: Unused "type: ignore" comment  [unused-ignore]
+ pandera/strategies/xarray_strategies.py:297: error: Need type annotation for "coord_specs"  [var-annotated]
+ pandera/strategies/xarray_strategies.py:359: error: Need type annotation for "coord_specs"  [var-annotated]
+ pandera/api/polars/model.py:82: error: Unused "type: ignore" comment  [unused-ignore]
+ pandera/api/ibis/model.py:86: error: Unused "type: ignore" comment  [unused-ignore]

[JukkaL]

Looking at the mypy primer output, the results look overall like a net improvement, but supporting None partial types better would reduce the number of deviations significantly, i.e. this pattern:

# mypy: allow-redefinition

x = None

def f() -> None:
    global x
    x = 1  # Error if using --allow-redefinition, works otherwise

cc @ilevkivskyi in case you have ideas, as you worked on partial types recently

[ilevkivskyi]

IIRC currently we simply disable None partial types when --allow-redefinition-new is set, and instead rely on widening. But widening doesn't work in this case either, since we prohibit widening variables from a different scope.

I don't know what was the motivation for disabling partial None types in the first place. If it was just because it felt redundant at the time (because local partial types didn't allow the above example anyway), then the simplest solution my be to just enable them back.

[ilevkivskyi]

Actually after a quick experimentation the other option (allow widening of global names) looks simpler. I mean something like this:

         if name.kind == LDEF:
             # TODO: Consider reference to outer function as a different scope?
             return False
-        elif self.scope.top_level_function() is not None:
+        elif self.scope.top_level_function() is not None and name.kind != GDEF:
             # A non-local reference from within a function must refer to a different scope
             return True
         elif name.kind == GDEF and name.fullname.rpartition(".")[0] != self.tree.fullname:

in refers_to_different_scope(). IIUC this should be safe since such functions are considered part of module top-level anyway (i.e. have def_or_infer_vars = True). But I didn't look deeply into other possible consequences of this change.

[ilevkivskyi]

These are actually both enabled in self-check, so this is a no-op.