flex segfaults in yy_top_state

[nxg]

Consider the program below.

Building and running this segfaults:

% flex -o flex-stack.c flex-stack.lex
% cc -o flex-stack flex-stack.c
% ./flex-stack "one two"
word:<one>
zsh: segmentation fault  ./flex-stack "one two"

Looking in the debugger:

%  lldb ./flex-stack                                                      
(lldb) target create "./flex-stack"
Current executable set to '/checkouts/me/codeberg/beastie/scratch/flex-stack' (arm64).
(lldb) proc lau "one two"
Process 27897 launched: '/checkouts/me/codeberg/beastie/scratch/flex-stack' (arm64)
word:<one>
Process 27897 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0xfffffffffffffffc)
    frame #0: 0x0000000100000f18 flex-stack`yy_top_state at flex-stack.c:1588:13
   1585	}
   1586	
   1587	    static int yy_top_state  (void)
-> 1588	{
   1589	    	return (yy_start_stack)[(yy_start_stack_ptr) - 1];
   1590	}
   1591	
Target 0: (flex-stack) stopped.
(lldb) p yy_start_stack_ptr
(int) 0
(lldb) ^D

Ie, there is a stack underflow when yy_top_state is called when no start-state has been pushed. I would expect yy_top_state to handle this OK, and return 0/INITIAL in this case (if (yy_start_stack_ptr == 0) return 0; perhaps?). In the real case, I was calling yy_top_state in an error handler, hoping to give the caller some suitable feedback.

The same thing happens with %option reentrant, though with yyguts_t involved.

Versions, etc:

% flex -V
flex 2.6.4
% uname -a
Darwin PHAS-MACDESKE207E9 25.6.0 Darwin Kernel Version 25.6.0: Sat Jul 11 15:24:35 PDT 2026; root:xnu-12377.161.13~4/RELEASE_ARM64_T8103 arm64
%  cc --version
Apple clang version 21.0.0 (clang-2100.1.1.101)
Target: arm64-apple-darwin25.6.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin

This is specifically the 2.6.4 built from the sources here, rather than the v2.6.4 that comes with macOS (though it displays the same behaviour).

ALPHABETIC  [a-zA-Z]
WS      [^a-zA-Z!]

%option noyywrap stack

%%

{ALPHABETIC}+   {
    printf("word:<%s>\n", yytext);
    printf("stack %d\n", yy_top_state());  // <----- note
    return 1;
}
{WS}+   {
    return 2;
}

%%
int main(int argc, char** argv)
{
    if (argc != 2) {
        fprintf(stderr, "Usage: %s string\n", argv[0]);
        exit(1);
    }

    yy_scan_string(argv[1]);

    int token;
    while ((token = yylex()) != 0) {
        printf("-> %d\n", token);
    }
}

[Mightyjo]

Basically, you've called yy_top_state() on an empty stack. We consider this a bug in the user-provided scanner, because you know your state machine and whether you pushed before calling yy_top_state() or yy_pop_state(). On the other hand, Flex would have to solve the halting problem to decide whether user-provided code was correct (in general).

I understand your reasons for disagreeing.

We could recognize that %option stack is turned on and push INITIAL at start-up. I don't think that would break any extant code. Anyone have a counterexample?

[nxg]

Thanks for your thoughts.

Basically, you've called yy_top_state() on an empty stack.
We consider this a bug in the user-provided scanner, because you know your state machine and whether you pushed before calling yy_top_state() or yy_pop_state().

Bug or not, crashing is never the right thing to do! An error message and exit() would be reasonable, if necessary, or recovering (for example by returning INITIAL in the case of an empty stack?; or yy_pop_state and/or yy_top_state returning a dummy, testable, value for stack underflow?; I don't think these are great suggestions).

An error handler, or any handler for a state with start-condition <*>, could potentially be called in multiple different start-conditions.

I realise (on reflection) that my mental model was that the ‘current state’ is what is on the top of the stack. That is, I thought that yy_top_state() was equivalent to YY_START (whereas of course the state-stack is the stack of conditions that have been YY_START). As a documentation point, I think the explanation of yy_push_state is ambiguous here, but that for yy_pop_state implies that stack-top and YY_START are necessarily distinct (and thus my mental model was wrong).

This also implies that yy_top_state feels somewhat redundant, since it seems unlikely a user would want to peek at the previous stack state. A more useful function would probably be to return the number of elements in the stack, which would enable code to clear the stack (not sure why that would be useful, but...), which is currently impossible. A ‘stack top’ operation is one of the standard stack primitives, of course – so I see why someone added this function – but it usually (?) can indicate ‘stack empty’ without error.

We could recognize that %option stack is turned on and push INITIAL at start-up.

That would be an attractive design, I think, but I believe it would change the semantics, because it would imply (yes?) that YY_START became equivalent to yy_top_state().

Thus:

1. My underlying problem is solved, since I realise that what I'm actually looking for is YY_START.

2. The code still shouldn't crash in any circumstances, so if there's no sensible way of recovering, it would seem reasonable to add a check for underflow and, for example, produce an error message and exit.

3. Can I also suggest a documentation tweak:

   • yy_push_state ( int new_state ) : pushes the current start condition onto the top of a start condition stack and switches to new_state as though you had used BEGIN new_state (recall that start condition names are also integers). The new start state remains available in YY_START. (comment: it seems both useful, in a be-nice-to-the-reader way, and natural to mention YY_START in this context)
   • Function: void yy_pop_state () : pops the top of the stack and switches to it via BEGIN. It is an error to pop an empty stack, and the program will (XXX) in this case.

4. Consider adding a yy_stack_size so a user can test if they are about to make an erroring call to yy_pop_state, and possibly deprecating yy_top_state.

[nxg]

(Sidenote: It would be useful to add an %option or compiler define to avoid declaring yy_top_state since it's (likely to be) little-used.
As it is, I have to include a if (0) { yy_top_state(); } in my code to avoid an unused-function warning, which if course is just asking for an unreachable-code warning to appear at some point.)

[nxg]

Two final thoughts:

• If yy_pop_state were to return the previous top-of-stack, as is common with pop-stack functions, then a user could get the effect of yy_top_state() with a pop followed by a push. Also, yy_pop_state could document that it returns negative on stack-underflow.
• If it were useful, I'm sure I could put together a patch to offer.