How isn't that like claiming you're a chef because you have a catering company where you serve microwaved foods, and claim it's silly to try to distinguish degrees of cooking skills.
How do you define software engineering such that it's 20-30 years old? Why isn't it 50 years old? I ask because I've heard the argument that software engineering is a young discipline but it feels like you're really stretching it with those 20-30 years.
>When the S-curve of hardware progress flattens, software engineering will gain in importance.
But a lot of software engineering issues are entirely divorced from the underlying hardware. So I'm not sure how the shape of hardware has any significant bearing to our current situation.
>Engineering is very much about the accumulation of experience
I would argue that we actually know a ton about how to build software. It's just that it's not really being taught or valued or applied. I would argue that's an important distinction because it strengthens the analogy of electrical engineer vs electrician. So it's more about just an accumulation of experience.
> But a lot of software engineering issues are entirely divorced from the underlying hardware. So I'm not sure how the shape of hardware has any significant bearing to our current situation.
To play devil's advocate with the other's position, even just over the time I've been in software since the early/mid 2000s I've seen things shift a number of times as new hardware creates new demands from our software or obsoletes old development struggles. PC RAM capacities rising 100x from 20+ years ago, more operations coming over on a slow and flaky WAN from untrusted devices, fewer people tolerating apps that only work on the tower PC stuck on their desk, more apps expected to sync state across devices because they're cheap enough we can own a bunch of them, use cases we learned to handle with plugged-in tower PCs adapting to power-draw-sensitive mobile devices, mobile devices being 1.5GHz instead of 16HMz, CPUs becoming fast enough that performance is often a footnote concern for whether to implement crypto, dedicated AI microchips changing what's possible on-device...
All of these seem like very reasonable justifications to rethink the way we build software, to change what's standard practice or to invent new tools that solve old problems in a modern context. To some degree the clock on when our industry starts accumulating stable knowledge won't start ticking until these kinds of changes stop happening.
>All of these seem like very reasonable justifications to rethink the way we build software, to change what's standard practice or to invent new tools that solve old problems in a modern context. To some degree the clock on when our industry starts accumulating stable knowledge won't start ticking until these kinds of changes stop happening.
But we already have stable knowledge that's simply not considered valuable. Your list of changes over the decades is of course a change but I don't understand why the attitude is "unless things stay stable over the next 30 years, there's no point in trying".
All those SOLID, agile, TDD, DDD, micro-services memes that are being implemented so superficially and cargo-cultishly are actually grounded in a real understanding that has a context, constraints, and trade-offs. The issue is that superficial and cargo-cult application. "Software engineers" that decide they're going to do micro-services without understanding what it solves, how it solves it and what they're trading off. And a business that doesn't care or know enough to ask for better. Maybe they don't need better! Maybe it's ok to slap together and the product will be equally mediocre whether the devs use cargo-cult pattern X or not. Maybe the quality doesn't matter that much to the business. In which case the devs being hired also don't need to actually know that much software engineering. In which case maybe the idea of electrical engineer vs electrician makes sense here too.
> we have people who create studies to test our efficacy and help inform future course development. We care about proving the efficacy of our approaches and constantly improving."
I'm curious how that works, what the results are. For instance, is the endpoint metric how many more students pass standardized testing? If so, how does that leak into the definition of 'mastery'?
That page also includes studies that talk about course pass rates.
I don't know if our efficacy work necessarily changes "mastery" definitions, but I can imagine (but have no direct involvement with) these studies and smaller-scale studies are used by our pedagogy and course creators to improve how we approach the material.
HPC where you try to wring the last ounce of performance from machines insufficiently powerful for the task. Weather forecasting, weapons simulation, protein folding...
That's interesting. I don't think as optimized memory manipulation via convoluted casting and such as "complex stuff". I think of that as optimization.
I thought what the author meant by complex was a gigantic system that needs to use many design patterns for good reasons; the code models a very complex, convoluted domain.
At the end of the day, it boils down to dependency management, wouldn't you say? That is, Python2 could be running forever if the environment is maintained. In the same vein, a messed-up OS or libs can stop a C app in its tracks.
For concurrency I think Raku has slightly more developed async support than Python, but the bigger advantage, I think, is in parallelism where, aside from the CPython GIL limiting practical parallelism in the main implementation (which is a big deal), Python as a language lacks the parallel iterables produced by the hyper (parallel but constrained to return in order) and race (parallel and not constrained in order) methods on the Iterable role in Raku, or anything like Raku’s hyperoperators that are semantically concurrent and parallelizable at the discretion of the optimizer. (Come to think of it, while also parallel, all those are also high-level concurrency constructs that Python lacks equivalents to.)
Python as a language can support parallelism via threads, and CPython as an implementation can via multiprocessing, but those are both very low level abstractions; Raku has higher level abstractions which allow much more succinct expression of parallel operations.
Sounds like multiprocessing.Pool.imap and imap_unordered? When dealing with io you also have async/await equivalent iterators like asyncio.as_completed().
sub foo ( Int \n where 1..100 ) {
# ___ start is a prefix operator that creates a Promise
# V
start { sleep n.rand; (1..n).pick }
}
my $result = foo(10);
say $result.status; # Planned
say await $result; # 6 (after a delay)
# this is nearly identical to foo above
sub bar ( Int \n where 1..100 ) {
Promise.in( n.rand ).then({ (1..n).pick })
}
(Note that `start` does not need a block unless you are grouping more than one statement as I have above.)
There are combinators like `Promise.allof` and `Promise.anyof`.
You usually don't need to use `Promise.allof`, because you can use `await` on a list.
my @promises = Promise.in(1.rand) xx 10;
my @results = await @promises;
(Note that the left side of `xx` is _thunky_ so there are ten different Promises with ten different random wait times.)
---
You should see the `react`/`supply` and `whenever` feature.
I am not that familiar with Python, but the GIL has long prevented any real in-process concurrency. Perl has concurrency but it's complex, heavy, and poorly supported. Raku's approach to this is built to avoid all these problems (like Elixir).
I agree the GIL is a problem but it's only an issue for CPU-bound problems. Is there really an important amount of CPU-bound work that is written in a scripting language? If it's CPU-bound, wouldn't you want to use something lower level?
If it's entirely CPU bound, you can use multiprocessing to negate most of the GIL issues, and transparently send inputs/outputs between the parent and child processes. If it's I/O bound, then AsyncIO is a great way to express asynchronous workflows. If it's a combination of both I/O and CPU bound workloads, there are ways to mix multiprocessing and AsyncIO to better saturate your cores without losing the simplicity or readability of Python: https://github.com/jreese/aiomultiprocess
Very true, I had not even thought about the multiprocessing package; it's sometimes not as convenient as multithreading but it'll get those other cores working.
Indeed and as a Perl developer I make use of XS/external libraries, cooperative multitasking (event loops/promises), and forking to cover these use cases. It doesn't preclude wanting the additional option to take advantage of threads in a useful way, since they do exist.
Why should first-class concurrency needs be required to script in Elixir? This question seems to imply that Python is somehow a default language and special requirements must be needed to justify writing in something else. Elixir is general-use and pleasant to write scripts in so seems reasonable to me for someone to do so if that's their thing.
lliamander replied that Elixir does too and it's worth a look
To that, 7thaccount replied that Perl6 and Elixir fill different niches.
So far, it seems Perl6 fills a niche that requires scripting and first-class concurrency.
My question then is: what is this niche that requires very solid concurrency but also scripting. In other words, what does Perl6 have in terms of concurrency that Python does not (given they are both scripting languages)?
I don't know of many instances where scripting and concurrency would be needed in the same application. But if you wanted to use single language both for scripting tasks and applications that require concurrency, then Raku or Elixir would work.
One instance I can actually think of, that would be specific to Erlang/Elixir, is if you have a long running service that you sometimes run batch jobs against (like importing data from a CSV). An Elixir script can hook directly into an Elixir/Erlang service and interact with it using message passing. It's a heck of a lot simpler than RMI, and avoids the need to expose new HTTP endpoints specifically for such tasks.
I think so, at least in some ways. I've shipped a project using PowerShell to script Windows Server/Hyper-V, and it was a pretty pleasant experience. Having a scripting language that not only does typical scripting stuff (wrangling text, etc.) and understands your application's objects is excellent.
Some differences:
* You can actually write your whole application in Elixir, whereas I could not see doing that with PowerShell
* In Erlang/Elixir, instead of objects you have processes. Think of your application as a microservice architecture on steroids, using the Erlang RPC protocol as a means for inter-process communication.
Because each process just sends and receives messages, your script can just send messages to any of the processes that make up your application, as if they were your own service. All you have to do is connect to the remote application using the Erlang Distribution Protocol (to connect different runtimes over the network).
It's a trip, and it took me a little while to wrap my head around. However, I now much prefer it to working with other concurrency abstractions (such as callbacks).
More to the point, does Python 3 have operators that transform sequential operations into operations that are both concurrent and parallelizable, and, in the case of iteration, provide control of parallelism parameters and whether results are constrained to be in-order or not.
To which the answer is “not only does Python not have them, but with the GIL CPython couldn't do much with them even if the language had them.”
The `start` statement prefix operator is a bit like Go's `go` keyword.
(In that when I translate Go into Raku, I usually replace `go` with `start`.)
my Channel \input .= new;
my Channel \output .= new;
start for input -> \message { output.send(message) }
start for output -> \message { say message }
input.send(5);
But what he was really talking about is something more like the following.
sub foo () {
sleep 10;
42
}
# ___
# V V
my $delayed-result = start foo();
say 'Hello World'; # Hello World
say $delayed-result.status; # Planned
say await $delayed-result; # 42
I'm embarrased to say I write scripts in node :P I used to write scripts in python but I've been writing so much JS that it's just easier for me in node. Plus, node defaults to locally installed deps so I don't have to deal with virutal environments.
Well just from experience untangling async calls in python is a nightmare and sometimes hard to reason about. The red/blue function problem is real. Meanwhile dispatching concurrent long-running scripting tasks is basically trivial in elixir (Enum.map over Task.async, then Enum.map over Task.await)
I'm guessing because Python's concurrency relies on the Global Interpreter Lock. Although I think concurrent.futures might address that. Haven't worked with python concurrency libraries in a bit.
I agree the GIL is a problem but it's only an issue for CPU-bound problems. Is there really an important amount of CPU-bound work that is written in a scripting language? If it's CPU-bound, wouldn't you want to use something lower level?
tldr; Using a scripting language that allows for native threads or has a strong concurrency model builtin to the core would be beneficial for any CPU bound scripting task...
If your program is CPU bound the GIL will slow you down. I'm sure since the python ecosystem is quite vast there are ways around the GIL... but then you have to worry about every library you use not supporting "real" multi-threading, or (b)locking for no reason and shitting all over your efforts.
As I've posted above, I'm a bit confused by CPU-bound work being processed in a scripting language. If you're planning on doing intense CPU-bound work, maybe use a lower-level language? I'm not saying abandon Python: you can extend Python with C or just use IPC to transfer data between a Python front-end and a computation back-end.
When I have a bit of Raku code that is too slow I complain (send a bug report) and then someone else figures out why the JIT isn't doing a better job and fixes it.
Then bingo-bango it ends up even faster than using NativeCall to call into the C code.
Of course there may be a delay before someone gets around to figuring it out; so in the meantime, NativeCall is actually very easy to use.
---
I would like to note that someone wrote Perl6/Raku code and the equivalent C/C++ code. They reported that the Perl6/Raku code was faster. (It was before Raku was brought up as an alias/name.)
I'm fairly sure that the reason the C/C++ code was slower is that it copied strings around rather than using pointers and indexes into other strings like MoarVM does.
At one time it was an obvious dichotomy that you would not use a scripting language for CPU bound work, but these days it is a much more blurry line. Partly because modern efficient languages are becoming ergonomic enough to work well as scripting languages while still giving you very good performance.
I actually love doing CPU bound work in Groovy which is usually described as a scripting language. But it gets converted straight to java byte code which is JIT'd and ends up as machine code. It only takes a few static typed hints here and there and it runs in the same league as C++ implementations. And it gets Java's excellent concurrency support for free.
I totally you feel you, I guess I thought your question was substantially more surface level than it was. My apologies.
I'm personally with you. I also don't tend to think object boxing is really the performance bottleneck for most applications, and if/when it is, likely the other requirements should've already ruled out using one (a scripting language).
It's like writing Nifs for Elixir, yeah sure you _can_, they have their purpose, but you could also just write another application to do that one thing and like you said, use IPC.
So in summary, we agree with each other, here's to:
How isn't that like claiming you're a chef because you have a catering company where you serve microwaved foods, and claim it's silly to try to distinguish degrees of cooking skills.