Personally I think child processes are the right approach for this. Launch a new process* for each query and it can (if you choose to go that route) dynamically load in compiled code. Exit when you’re done, and the dynamically loaded code is gone. A side benefit of that is memory leaks are contained, since all memory you allocate is about to be removed anyway.

I’d probably be fine with hundreds or thousands of these hanging in memory. I suspect the generated code for a single query would be in hundreds of kilobytes, maybe a megabyte. But yeah, this is one of those technical details I’d worry about.

Honestly, I wonder if you could just use an actual HTTP server for this? They can handle hundreds or even thousands of simultaneous requests. They can handle requests that complete in a fraction of a millisecond or ones that run for several hours. And they have good tools to catch/deal with code that segfaults, hits an endless loop, attempts to allocate terabytes of swap, etc. HTTP also has wonderful tools to load balance across multiple servers if you do need to scale to massive numbers of requests.

Not sure how a HTTP server would solve the CPU bottleneck of scanning terabytes of data per query?

I somehow didn’t think a regular JIT solution might be applicable here, but it is. Thank you! There seems to be a number of projects doing JIT for C++, will look at them.

So far I’ve been following recommendations from this person: https://old.reddit.com/r/NewMaxx/comments/16xhbi5/ssd_guides_resources_ssd_help_post_your_questions/

This plea for help is specifically for non-coding, but still deeply technical work.

Another idea that just occurred to me. Maybe position: absolute; both the real content and the gibberish content with the same top, left, width, and height attributes so that the real content and the gibberish overlap and occupy the same location on the page. Make sure both the real and gibberish content elements have no background so that remains clear. Put the gibberish content in the DOM before the real content. (I think that will ensure that the gibberish appears behind the real content even without setting the z-index.) And then make JS set the color of the text in the gibberish element the same color as the background so humans can’t see it.

Be aware that these techniques can affect accessibility for people using screen readers.

As of May 2023, 65% of the Ukrainian refugees that left Ukraine starting February 2022 and decided to stay in Poland found a job—so, within around a year, as opposed to 5-6 years as in the article. Cultural similarity here is likely making it much, much simpler. For those who want to read more about the situation of Ukrainian refugees in Poland, this report by Polish National Bank (Narodowy Bank Polski, NBP) might be useful: https://nbp.pl/wp-content/uploads/2023/05/Raport_Imigranci_EN.pdf (in English!), there is a lot of interesting details.

Yep, thank you, that’s pretty close to what I imagined!

A lack of planning on your part doesn’t constitute an emergency on mine.

Though I kind of think Japanese grammar cannot express this thought and the closest you can get is Ganbatte!

Good question! I quickly found this table, though this is yearly statistics only: https://www150.statcan.gc.ca/t1/tbl1/en/tv.action?pid=3510019201

Last job, we started writing mixing bits of Kotlin in an otherwise mostly-Java in a monolithic Spring-based service. Good experience.

One reason (among many) is that employment in American companies is less stable than in Europe with strong employment laws. Twitter could not do the same type of layoffs in Europe, with stories like this one being pretty common. But this safety net has a cost, and the cost is a part of total employment cost for employers. Whether the safety net is worth it for employees in IT, that’s another matter—but it can’t not be taken into account because of the law.

BTW, in some European countries there is a strong culture of IT workers doing long-term contractor work exactly to trade off employment laws for (usually quite a lot) higher wage.

Given these criteria, ggplot2 wins by a landslide. The API, thanks to R’s nonstandard evaluation feature, is crazy good compared to whatever is available in Python. Not having to use numpy/pandas as inputs is a bonus as well, somehow pandas managed to duplicate many bad features of R’s data frame and introduce its own inconsistences, without providing many of the good features¹. Styling defaults are decent, definitely much better than matplotlib’s, and it’s much easier to consistently apply custom styling. Future of ggplot2 is defined by downstream libraries, ggplot2 is just the core of the ecosystem, which, at this point, is mature and stable. Matplotlib’s activity is mostly because that lack of nonstandard evaluation makes it more cumbersome to implement flexible APIs, and so it just takes more work. Both have very minimal support for interactive and web, it’s easier to just use shiny/dask to wrap them than to force them alone to do web/interactive stuff. Which, btw, again I’d say shiny » dask if nothing but for R’s nonstandard evaluation feature.

Note though that learning proper R takes time, and if you don’t know it yet, you will underestimate time necessary to get friendly. Nonstandard evaluation alone is so nonstandard that it gives headaches to people who’d otherwise be skilled programmers already. matplotlib would hugely win by flexibility, which you apparently don’t need—but there’s always that one tiny tweak you would wish to be able to do. Also, it’s usually much easier to use the platform’s default, whatever publishing platform you’re going to use.

As for me, if I have choice, I’m picking ggplot2 as a default. So far it was good enough for significant majority of my academic and professional work.

¹ Admitably numpy was not designed for data analysis directly, and pandas has some nice features missing from R’s data frames.

At such scale, a scraper wouldn’t be necessary, that’s easily doable by humans involved in these communities—with a human touch as well.

Yes, many times. And I recall using the technique manually back when I was working with Subversion many, many years ago. No fun stories though, sorry, it’s just another tool in a toolbox.