flex `%option noline` outputs a #line directive

[fooofei]

Used flex version

2.6.4
download from https://github.com/westes/flex/releases/download/v2.6.4/flex-2.6.4.tar.gz

Input file

I have a file which filename is test.flex,
the content is

%option noline
%option c++

%%
.* {return 1;}
%%

Process

I used command line flex test.flex to generate file lex.yy.option_noline.cc,

then I remove the %option noline from file test.flex,

I use command line flex --noline test.flex to generate file lex.yy.commandline_noline.cc.

git diff the two generate file, a part of the diff is:

diff --git a/lex.yy.commandline_noline.cc b/lex.yy.option_noline.cc
index 893d434..2678b1e 100755
--- a/lex.yy.commandline_noline.cc
+++ b/lex.yy.option_noline.cc
@@ -386,9 +386,15 @@ static const flex_int16_t yy_chk[8] =
 #define yymore() yymore_used_but_not_detected
 #define YY_MORE_ADJ 0
 #define YY_RESTORE_YY_MORE_OFFSET
+#line 1 "test.flex"
+
+

We can see, if we use %option noline, flex also outputs #line to the generate .cc file ( lex.yy.option_noline.cc ) which was not expected. On the other hand, the command line option flex --noline is not outputs #line.

Found & Research

I found in the flex source code, it use a global variable
int gen_line_dirs
to decide flex is outputs #line or not. but when flex outputs #line the variable gen_line_dirs is 1, then it make outputs.

I debug the source code, the callstack is:

(gdb) bt
#0  add_action (new_text=0x7fffffffd170 "#line 1 \"test.flex\"\n") at misc.c:134
#1  0x000000000040ed93 in line_directive_out (output_file=0x0, do_infile=1) at misc.c:379
#2  0x000000000040d798 in readin () at main.c:1473
#3  0x000000000040ae18 in flex_main (argc=2, argv=0x7fffffffe2f8) at main.c:170
#4  0x000000000040af6d in main (argc=2, argv=0x7fffffffe2f8) at main.c:209

The line number at the end of callstack line maybe not right, I add some fflush(stdout) in the file for debug, it will affect the line number.

Releated issues of this repo

#55 flex --noline outputs a #line directive
#161 filter: Don't emit #line directive to header if noline option is set
#56 %option noline generates and error message

Attachment

the file test.flex and lex.diff:
check_lineno.zip

[jannick0]

Would dec42ee solve this issue?

[Smattr]

Is dec42ee orphaned? As far as I can see it is not an ancestor of any current branch or tag. I just hit the issue @fooofei describes and the commit @jannick0 references certainly looks like a possible fix to me.

[jannick0]

Until the issue is fixed here a suggestion for a workaround: Pipe flex's output through sed instructed to remove all line directives starting at the beginning of the line in flex's output by replacing

flex $(LFLAGS) -o x.c x.l or flex -t $(LFLAGS) x.l > x.c

by

flex -t $(LFLAGS) x.l | sed -e '/^#line/d' > x.c

NB: flex automatically copies verbatim line directives as part of c code in x.l to flex's output which are removed, too, if they start at the beginnig of the line in x.l. Clearly a corner case.

[Smattr]

Yes, it should be straightforward to workaround. It would be nice to fix this in Flex itself though, as #344 seems to achieve. For the curious reader, what led me to trip over this was attempting to make a reproducible build for Debian packaging. Do we have any estimate on when 2.6.6 is likely to ship?

[westes]

I don't do time estimates. Noted about the need and priority, however.

…

On Wednesday, 18 March 2020, 12:06 pm -0700, Matthew Fernandez ***@***.***> wrote: Yes, it should be straightforward to workaround. It would be nice to fix this in Flex itself though, as #344 seems to achieve. For the curious reader, what led me to trip over this was attempting to make a [reproducible build for Debian packaging](https://wiki.debian.org/ReproducibleBuilds). Do we have any estimate on when 2.6.6 is likely to ship? -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: #268 (comment)

-- Will Estes westes575@gmail.com

[Smattr]

I have not yet bisected, but I think this was possibly (silently) fixed in 64cf032, the relevant hunk being:

diff --git src/scan.l src/scan.l
index 099d153..82d57df 100644
--- src/scan.l
+++ src/scan.l
@@ -164,9 +164,13 @@ M4QEND      "]""]"
        ^"%x"{NAME}?    return XSCDECL;
        ^"%{".*{NL}     START_CODEBLOCK(false);
     ^"%top"[[:blank:]]*"{"[[:blank:]]*{NL}    {
+               char trampoline[512];
                 brace_start_line = linenum;
                 ++linenum;
-                buf_linedir( &top_buf, infilename?infilename:"<stdin>", linenum);
+               snprintf(trampoline, sizeof(trampoline),
+                        "M4_HOOK_TRACE_LINE_FORMAT(%d, [[%s]])",
+                        linenum, infilename?infilename:"<stdin>");
+                buf_strappend(&top_buf, trampoline);
                 brace_depth = 1;
                 yy_push_state(CODEBLOCK_MATCH_BRACE);
             }

[Smattr]

I had a chance to bisect this today, and my guess above was apparently wrong. The issue is indeed fixed but it was fixed sometime prior to 24fd055 on 2017-09-04. I.e. this issue was apparently fixed prior to it even being opened, but has not yet made it into a release.

My vague "prior to 24fd055" wording is because I can't bisect beyond that due to #265, prior commits segfault during build. While bisecting I also hit (1) failures to build some commits due to a missing skeleton header, (2) failure to install some commits due to /usr/bin/install failing to find flex.1, and (3) the build modifying checked in files, making an auto-bisect somewhat awkward. I assume all these problems are already known to the maintainers.

Sorry, ignore the above. While the discussion about segfaulting in prior commits is accurate, this bug is not fixed. I guess I was bamboozled by all the cross-cutting issues, but in trying to confirm my results I cannot. The bug is still reproducible on the current head of master, 7ea145a. Can someone with authority please re-open this issue?

[jjmaestro]

@Smattr I think you are right, the issue is still present and I wrote a fix that adds an if-guard in the exact code you mentioned in your comment (64cf032).

This bug was also reported again (#463). I'll make a PR with the fix!