Failing to build javascript backend

[Mai-Lapyst]

When trying to build the javascript backend, the build process exits with following error:

/home/mai/projects/raku/nqp/nqp-m --module-path=/home/mai/projects/raku/nqp/gen/js/stage1 --target=mbc --output=/home/mai/projects/raku/nqp/gen/js/stage1/QAST/Compiler.moarvm /home/mai/projects/raku/nqp/gen/js/stage1/QASTCompiler.nqp
===SORRY!=== Error while compiling /home/mai/projects/raku/nqp/gen/js/stage1/QASTCompiler.nqp
Use of undeclared variable '$T_UINT' at NQP::src/vm/js/Operations.nqp:579, near ", $T_UINT]"
   at NQP::src/HLL/Grammar.nqp:234  (/home/mai/projects/raku/nqp/NQPHLL.moarvm:panic)
 from NQP::src/NQP/Actions.nqp:479  (nqp.moarvm:variable)
 from <unknown>:1  (nqp.moarvm:variable)
 from <unknown>:1  (nqp.moarvm:term:sym<variable>)
 from NQP::src/QRegex/Cursor.nqp:729  (/home/mai/projects/raku/nqp/QRegex.moarvm:!protoregex)
 from <unknown>:1  (/home/mai/projects/raku/nqp/NQPHLL.moarvm:termish)
 from NQP::src/HLL/Grammar.nqp:79  (/home/mai/projects/raku/nqp/NQPHLL.moarvm:nulltermish)
 from NQP::src/HLL/Grammar.nqp:395  (/home/mai/projects/raku/nqp/NQPHLL.moarvm:EXPR)
 from <unknown>:1  (nqp.moarvm:circumfix:sym<[ ]>)
 from NQP::src/QRegex/Cursor.nqp:729  (/home/mai/projects/raku/nqp/QRegex.moarvm:!protoregex)
 from <unknown>:1  (/home/mai/projects/raku/nqp/NQPHLL.moarvm:circumfix)
 from <unknown>:1  (/home/mai/projects/raku/nqp/NQPHLL.moarvm:term:sym<circumfix>)
 from NQP::src/QRegex/Cursor.nqp:729  (/home/mai/projects/raku/nqp/QRegex.moarvm:!protoregex)
 from <unknown>:1  (/home/mai/projects/raku/nqp/NQPHLL.moarvm:termish)
 from NQP::src/HLL/Grammar.nqp:79  (/home/mai/projects/raku/nqp/NQPHLL.moarvm:nulltermish)
 from NQP::src/HLL/Grammar.nqp:395  (/home/mai/projects/raku/nqp/NQPHLL.moarvm:EXPR)
 from <unknown>:1  (nqp.moarvm:arglist)
 from <unknown>:1  (nqp.moarvm:args)
 from <unknown>:1  (nqp.moarvm:term:sym<identifier>)
 from NQP::src/QRegex/Cursor.nqp:729  (/home/mai/projects/raku/nqp/QRegex.moarvm:!protoregex)
 from <unknown>:1  (/home/mai/projects/raku/nqp/NQPHLL.moarvm:termish)
 from NQP::src/HLL/Grammar.nqp:395  (/home/mai/projects/raku/nqp/NQPHLL.moarvm:EXPR)
 from NQP::src/NQP/Grammar.nqp:181  (nqp.moarvm:statement)
 from <unknown>:1  (nqp.moarvm:statementlist)
 from <unknown>:1  (nqp.moarvm:blockoid)
 from NQP::src/NQP/Grammar.nqp:472  (nqp.moarvm:package_def)
 from <unknown>:1  (nqp.moarvm:package_declarator:sym<class>)
 from NQP::src/QRegex/Cursor.nqp:757  (/home/mai/projects/raku/nqp/QRegex.moarvm:!protoregex)
 from <unknown>:1  (nqp.moarvm:package_declarator)
 from <unknown>:1  (nqp.moarvm:term:sym<package_declarator>)
 from NQP::src/QRegex/Cursor.nqp:757  (/home/mai/projects/raku/nqp/QRegex.moarvm:!protoregex)
 from <unknown>:1  (/home/mai/projects/raku/nqp/NQPHLL.moarvm:termish)
 from NQP::src/HLL/Grammar.nqp:418  (/home/mai/projects/raku/nqp/NQPHLL.moarvm:EXPR)
 from NQP::src/NQP/Grammar.nqp:181  (nqp.moarvm:statement)
 from <unknown>:1  (nqp.moarvm:statementlist)
 from NQP::src/NQP/Grammar.nqp:148  (nqp.moarvm:comp_unit)
 from NQP::src/NQP/Grammar.nqp:40  (nqp.moarvm:TOP)
 from NQP::src/QRegex/Cursor.nqp:1505  (/home/mai/projects/raku/nqp/QRegex.moarvm:parse)
 from NQP::src/HLL/Compiler.nqp:550  (/home/mai/projects/raku/nqp/NQPHLL.moarvm:parse)
 from NQP::src/HLL/Compiler.nqp:465  (/home/mai/projects/raku/nqp/NQPHLL.moarvm:execute_stage)
 from NQP::src/HLL/Compiler.nqp:501  (/home/mai/projects/raku/nqp/NQPHLL.moarvm:run)
 from NQP::src/HLL/Compiler.nqp:504  (/home/mai/projects/raku/nqp/NQPHLL.moarvm:)
 from NQP::src/HLL/Compiler.nqp:496  (/home/mai/projects/raku/nqp/NQPHLL.moarvm:compile)
 from NQP::src/HLL/Compiler.nqp:166  (/home/mai/projects/raku/nqp/NQPHLL.moarvm:eval)
 from NQP::src/HLL/Compiler.nqp:441  (/home/mai/projects/raku/nqp/NQPHLL.moarvm:evalfiles)
 from NQP::src/HLL/Compiler.nqp:364  (/home/mai/projects/raku/nqp/NQPHLL.moarvm:command_eval)
 from NQP::src/HLL/Compiler.nqp:289  (/home/mai/projects/raku/nqp/NQPHLL.moarvm:command_line)
 from NQP::src/NQP/Compiler.nqp:37  (nqp.moarvm:MAIN)
 from NQP::src/NQP/Compiler.nqp:35  (nqp.moarvm:<mainline>)
 from <unknown>:1  (nqp.moarvm:<main>)
 from <unknown>:1  (nqp.moarvm:<entry>)
make: *** [Makefile:1191: /home/mai/projects/raku/nqp/gen/js/stage1/QAST/Compiler.moarvm] Fehler 1

Command used: perl Configure.pl --backends=moar,jvm,js --gen-moar

Permalink to line in gh:

nqp/src/vm/js/Operations.nqp

Line 579 in 3af1893

add_simple_op('chown', $T_VOID, [$T_STR, $T_UINT, $T_UINT], :side_effects);

[massa]

Hello! I've been having exactly the same problem here...

[raiph]

Hi @Mai-Lapyst and @massa,

I began writing this comment to try help you. But also -- indeed much more so -- so others could read it as a sort of mini tutorial on digging into problems. Please feel free to skip to the end to see the conclusion that might help you.

(@Mai-Lapyst started the ball rolling by writing a solid initial post/comment. As I followed the link they provided, and traveled onward to see where it went, I decided to write down my steps because that felt like the thing to do. Then that took over and became the purpose -- to write something I could share a link to for anyone who might be interested in how I went from the line with the error into how it got into the project. It's waaay over the top compared to my original purpose, which was to help you two, and not anyone else, but I hope you don't mind!)

---

The following mini tutorial on digging into a problem was written while using a Chrome browser in mid 2024. The UI details I describe have been stable for a good few years so will hopefully work for later readers.

---

I followed the link you provided. That showed a particular line of code in GitHub.

Next, I clicked the Blame button at the top left of the code window next to the Code button. That showed, to the left of the line of code, a "nqp::chown args should be unsigned" link to the commit that was/is to "blame" for the last change (as of mid 2024) to that line of code.

Next, I clicked that link. That showed the commit details: 11 Nov 2022: nqp::chown args should be unsigned. You can see it introduced $T_UINT into the JS backend at the line that is mentioned in the error ("Use of undeclared variable '$T_UINT'").

So I presume that the build has been broken since at least 2022.

The commit also tells us it was MasterDuke17 who committed that code. So now we know who made the change and thus who will be the right person to talk to, or at least to mention if we talk to someone else.

---

I think it's often worth trying to understand roughly what happened before contacting a core dev. So I continued...

Next, I clicked the Browse files button at the top right of the code window. That listed the nqp repo as it was immediately after MasterDuke2017's commit.

Next, I navigated to the folder containing the JS backend code (.../src/vm/js/). I confirmed the commit shown at the top of the files window was "nqp::chown args should be unsigned" and then clicked the History link to the right of that to show the history of commits to that folder (the JS backend). I saw the "nqp::chown args should be unsigned" commit plus other commits made by MasterDuke17 in 2022.

Next, I clicked on the prior two commits to see more of the commit details:

25 Oct 2022: Add nqp::chown() op. Works for the MoarVM and JVM backends, untested for the JS backend. Currently is Linux(Posix?) only, though that isn't checked at this level and will have to be checked in Rakudo.

7 Oct 2022: Make unsigned comparison ops available. The MoarVM and JVM implementations work, but the JS ones are completely untested.

Bingo. So:

• nqp's main branch HEAD has presumably had a broken JS backend since at least 2022.

• MasterDuke17 made at least two commits to the nqp JS backend in 2022 in which they made it clear they'd not tested the changes.

---

I could explore further but I think the foregoing will do as far as nailing down where to next hunt for what went wrong with the nqp JS backend build.

The remaining thing perhaps worth understanding is the status of the JS backend in general. MasterDuke17 made commits without testing the JS backend. What's that about? I'll discuss that in my next comment.

[raiph]

Why did MasterDuke17 make commits without testing the JS back end works? I think the short answer is that they're being sensible. I'm not aware of anyone using the JS back end let alone developing it. Assuming I'm right, why actively maintain it? But maybe I'm painting the wrong picture. This comment represents the most interesting tidbits I found as I tried to assess the situation.

core dev IRC comments on the JS backend

There are many mentions of "js" and "backend" on the #raku IRC channel. They include many comments by Paweł (the original creator of the JS back end) and current core devs: "js backend" [all words / ignore case] search of #raku.

2015 thru 2019

Paweł Murias created the JS back end. Their work on it under a grant was documented in a blog; the first entry was written in 2015, the last in 2019.

Their last post starts with:

Work on the Rakudo.js grant has been completed and now I'm at the stage where community feedback is needed. It would be super grateful for Your feedback. You can provide it in blog comments on this post.

It shows no comments. I don't know, but it's possible Paweł Murias solicited feedback, got none, and was demoralized by that. Regardless, I think I'm right in saying they more or less stopped further developing the JS back end around that time.

In that last blog post they also linked to "The final grant status update". This provides a clear picture of what it could and couldn't do at the end of 2019. I'll quote some key points I want to briefly discuss:

Rakudo.js has been released on NPM as rakudo

We pass our chosen subset of roast tests ... ~61% of total roast tests (the grant was targeting only a subset of the whole roast ... with the biggest omission being multi threading and IO things).

The key thing to get is that, as 2020 was about to begin, a Rakudo with a JS back end and other goodies such as a nice browser hosted "playground" (which is still running), was all nicely packaged up and released as an NPM as a reasonably smoothly building working subset of Raku.

That said, it's also important to recognize that it was (and still is): experimental status; just a subset (in contrast to the full Raku implemented by the MoarVM back end); and unoptimized.

2020 thru 2021

It's possible the JS back end build broke in 2020.

In their answer to their own 2020 SO post "Building multiple backends for Raku fails, @sumanstats said they'd successfully built both MoarVM and JVM back ends, but got blocked on building the JS back end:

So the issue left here is how to npm link while building rakudo.

As far as I know, no one dug into this linking issue, and that's the last SO discussion of trying to build the JS backend.

2022 thru 2024

As noted in my prior comment, I presume the HEAD of the nqp main branch has had a broken JS backend since at least 2022 if not before.

Now? The future?

It's possible that MasterDuke17 would look into fixing their commits. It's not clear that'll be enough, but it might be worthwhile progress. I think it all depends on someone else deciding they're determined to get it building again, and building a good relationship with the relevant core devs, starting with MasterDuke17.

If we're very unlucky, no one will fix the JS back end so it's building again.

If we're very lucky someone will step forward to create a WASM backend. I can imagine that happening in some ideal future; perhaps they'd wrap something around MoarVM to transpile from C to WASM, or build on the RakuAST work to create another back end along the same lines as the current JS back end. (I think that either approach would be able to steal some stuff from the current JS backend.)

Hth.

[MasterDuke17]

Does e7d1bbe3df help?

[Mai-Lapyst]

Thanks for the deep dive into the problem @raiph .

But I got the feeling here is a big misunderstanding. I know very well how to use github (see my profile, I also contributed elsewhere), and also fully know how to disect code via a git blame. However, this is an issue tracker and I here have the role of an reporter. This has nothing to do with fixing the code, that would be an pull request.

It also is very debateable what the information about who exactly "introduced" the bug. I'm not a person to blankly point fingers onto others; and again: this is an report, it reports that something is broken and might be worth looking into. As a reporter I do not have the time nor the responibility to understand your internal team structure and who to contact. If you want that, document it atleast via code owners and/or create labels that an action or bot can then use to assign issues to people, but please do not expect everyone who tries to help you by providing feedback to also have a deeply knowlage about your internal team structure.

I think it's often worth trying to understand roughly what happened before contacting a core dev.

Again: I report (aka giving feedback) here. I really do not understand how this is compareable with directly contacting a core dev, as I didn't intend that in any way or form and thus also pinged or mentioned anyone. If an simple github issue is for you like an personal email, this seems more a problem of misunderstanding how github issues are ment to be used.

MasterDuke17 made at least two commits to the nqp JS backend in 2022 in which they made it clear they'd not tested the changes.

Okay, we successfully know now that somebody is at fault by not testing code propely. Again: I am not that sort of a person to punish people but if you really really want that, fine: "MasterDuke17 is now officially from your "analysis" the one soly responsible for breaking nqp for javascript due to bad craftmanship." Because thats what I'm reading here from your passive-agressive "analysis". Again: i dont point fingers, but aparently you do. This is highly toxic in my eyes, but lets continue.

I think the short answer is that they're being sensible.

Thats a nice take. "Let's just break software capabilities that we market at every corner and be proud of it." It's like breaking the linux kernel by letting it halt everytime someones hits the letter Z just because "its sensible that we've not tested this." Honestly this feels like a troll here, even though I recognize that you propably didnt intend that.

As for marketing features: Raku openly tells everyone in their public ressources that it can be compiled to js (aka using Rakudo.js with building depends on nqp's js backend), thus a untested backend is not sensible. It's just bad craftmanship.

Proof: https://rakudo.org/downloads | https://web.archive.org/web/20240604083814/https://rakudo.org/downloads

I'm not aware of anyone using the JS back end let alone developing it. Assuming I'm right, why actively maintain it?

See above.

Their work on it under a grant was documented in a blog

To the whole grant thing: this is unrelevant for an issue wich reports a bug.

That said, it's also important to recognize that it was (and still is): experimental status.

Well maybe it seems so from someone like you that is deep into the internals of this project that this is indeed the case, but truth is, that the public image is not painted that this is unmaintained nor experimental. See proof above; it states it is useable in modern browsers, which means for the average user that it is supported. The website then should be changed to reflect this state to not cause unneccesary confusion in users.

If we're very unlucky, no one will fix the JS back end so it's building again.

If we're very lucky someone will step forward to create a WASM backend.

I dont know how any of these statements help the issue thats on hand: the backend is broken, I reported it so it can get attention and someone can contribute a fix.

---

This all has nothing to do with the issue on hand, and I would like to ask you to not abuse an issue reporting tool as an (personal) blog, or atleast not in issues I reported. I'm interested in improving the software by reporting bugs, not discussing who went wrong where, who's fault it is, why something is broken bc no money was left (your whole grant thing), or someone gave up bc feedback wasn't provided. Yes it is sad, but the truth is: thats open source software. We work all tirelessly your asses of to improve software, to free people from comercially burden and vendor-lockin, etc. Yes we never get any attention or be rewarded for our work. And personally I dont report bugs or contribute things because I want to earn money. I do it because I want to improve software I use. If anybody else sees it differently and want to make money off of it, it seems like a problem for those people.

And again, this is how oss works, no reward, all the work. It always was that way and always will.

---

I will now continue with the issue of building nqp, if thats alright with you. Thanks.

[Mai-Lapyst]

Does e7d1bbe3df help?

@MasterDuke17 thanks for the change. It seems that atleast the reported issue now has ceased to appear. It now fails at a different location. How do you want to proceed with these issues? Open a new one for each case where something is currently broken or continue here?