There's a difference. First, you're not incurring a large runtime penalty, so this works better in higher performance scenarios.
Second, you're trapping and propagating successful values in a monad, which means your obligating the caller to handle both success and error conditions.
> First, you're not incurring a large runtime penalty
How large, and why? Only when exceptions are thrown, or also during the happy path?
From what I understand, with Monads the code check errors after each expression (like in C/Go), but behind the scene. Isn't it the case?
If so, wouldn't that incur an overhead that is not present with exceptions, where normal code runs without checks and where "throws" perform a sophisticated jump?
With exceptions you have overhead on the happy path and potentially even in code that can never error: you have to keep all your old stack frames around in case you throw, since you never know when you might throw. Doing nice tail calls, or continuations, gets very hard since you have to figure out how your exception handling interacts with them. (Of course F# has to pay a lot of those costs already, since it runs on a VM that was designed for a language with exceptions). Whereas with result types everything is just functions and values and you don't need any special cases.
I consider myself a skilled programmer that is aware enough of low level issues to usually write faster alternatives to any "not quite optimal" code on the first try, without resorting to any micro-optimization. That includes rewriting code that makes use of libraries like STL of which most people will just say "dude, you are insane, you can't be faster than the STL. They put so many man-years of effort into that".
But it rarely matters how much work you put into something to make it faster. What matters is what you can avoid to do at all.
I have pretty much zero knowledge of CPU-level issues and barely know enough x86 assembler to read my compiler's output. I have never cared for optimizations like inlining code, vectorization or tail call elimination. I have tried inlining a few times, and the speed-ups it brought were negligible. (And the slow-downs that too much of inlining brings, by increasing code size and thereby increasing cache pressure, must be very subtle and basically unmeasurable, while being potentially enormous. And the disadvantages from a maintainability standpoint are considerable as well).
If you have to care about these things, then maybe you're doing too much work. (Well, I guess I would use vectorization/SIMD if I had to write a video codec, but I've never done that).
Avoiding Exceptions in .NET isn't some micro-optimization that makes a tiny difference in runtime speed...
Exceptions are pricey pricey things that take time and create some minorly interesting VM/GC behaviour that can impact your application. If at all possible they should be avoided in executed code.
> But it rarely matters how much work you put into something to make it faster. What matters is what you can avoid to do at all.
Using Result<> types to handle error cases is a very good example of doing just that: avoid the big hit of exceptions and poor exception handling in cacadingg situations with sensible return types that let you avoid a buttload of delicate work.
> Exceptions are pricey pricey things that take time and create some minorly interesting VM/GC behaviour that can impact your application.
This is all very vague and handwavey. Not sure about the .NET world but in other systems the costs of exceptions is not significant and they can be used successfully even in the most demanding low-latency applications.
> Using Result<> types to handle error cases is a very good example of doing just tha
Result types don't scale. I suppose it's the sort of thing that programmers have to keep rediscovering for themselves but there's a reason why even Haskell eventually "discovered" exceptions (and I suspect the Rust guys will get there eventually). In the real world where you have lots of deeply nested function calls some of which cross component boundaries good error handling requires three things: contracts (you need to be able to express possible errors at boundaries), context (you need to know exactly what resources have been allocated so they can be cleaned up when an error is caught) and recovery (you need to be able to return the application to well-defined state/point so computation can continue). Frankly, Result types don't give you any of these. What you end up with is a very generic set of errors (necessary to avoid a combinatorial explosion of error types) and a poor-man's implementation of exceptions (see panic/recover in golang). Or I suppose you can write code that doesn't involve deeply nested call-graphs and multiple components and everything is perfectly "flat" and stateless... but this wouldn't be like any real application I've ever seen.
Even so, exceptions are generally not used for error handling in Haskell, yet life seems to go on. You'll see code using `throwError` or `catchError`, but usually these will just be providing a nice bit of sugar on top of an Either or Maybe type. So it seems like the takeaway should be that Result types do scale, with proper library+language support.
We sorta use exceptions for IO-based code, mostly because we love parallel combinators in async and they totally require async exceptions to work right.
What I was trying to say somewhere between the lines: Try structuring your code so you don't run into errors in the first place. Because if error handling performance matters, you're having too many of them.
This is always good advice. One should strive so their code doesn't run into errors in the first place.
However, sometimes the errors are actually part of the fabric of the domain itself. They're expected to occur very frequently. The data it encodes is valuable. The way they're handled is important.
Nevertheless, you're right. More often than not, you're better of using traditional exception handling and preventing errors from occurring in the first place.
Oh, I agree. I'd use results rather than exceptions even if they were slower (and in some implementations they are), because they make the code so much easier to understand and maintain and that's more important the overwhelming majority of the time. But for what it's worth, correctly implemented they can also be faster. And I find that's generally the case: a clear, simple implementation that expresses the essence of the problem will be faster than any number of carefully-tuned edge cases.
> Only when exceptions are thrown, or also during the happy path?
This is where I want to make a key distinction between exceptions and errors ala Railway programming. Performance with exceptions doesn't really matter because exceptions are supposed to be exceptional.
With Railway oriented programmer, the whole point is that you're encoding errors into the logic of some type of system. These types of errors may not only be not exceptional, they may a very regular part of the application.
Like I told another commenter, a friend of mine uses them in an application where they easily would throw thousands of exceptions per second if they used exceptions. Not only would it undermine the performance of their system, it's using the wrong tool for the job.
The performance criticism is a common one, but I don't think most people actually know what they're talking about. I certainly don't, since I don't use exceptions, but I've heard many respected engineers say that they needn't be slow (it depends greatly on implementation).
It shouldn't matter much anyway: You usually don't optimize for exceptional cases, especially if they lead to program termination.
And furthermore, you should not make heavy use of exceptions anyway :-). They lead to non-local control flow that is extremely hard to follow. They are nice for short scripts where they print a stack trace and abort the program. But they are a huge maintenance burden in larger programs.
You are correct that the performance of exceptions isn't typically needed because exceptions aren't meant to be thrown that often. They're exceptional.
Railway oriented programming is better suited when errors are much more common in the ringtone and when their handling is a fundamental part of the rules you're encoding.
One friend uses a Railway oriented approach precisely because throwing that many exceptions per second (I don't know the exact number) would severely undermine the performance of their system.
So I'm not talking about exceptions, I'm talking about talking about errors. You can easily generate thousands of errors per second if you're simply validating and routing large streams of data.
It's nice to get concrete and leave the abstract shit talk (of which I've been guilty many times).
So let's focus on that problem you mention and not argue errors vs exceptions terminology. In the case you mention I would clearly make two streams of data. In the simplest case, two append-only dynamically allocated arrays, one for parsing errors and one for parsed results. This is very likely possible to do - unless one needs an exact ordering, in which case it could be still possible to add positions to each of the streams.
It's intuitively clear to me that this approach will be significantly faster than a Result<>/Either based approach. Because control flow is much simpler. There are two consumers, one for the error stream, and one for the parsed stream. There is no switching between the cases on the consuming side, and complexity is much reduced because each part can do one thing and do it well.
You also get the freedom to process the streams independently, e.g. at different points in time, or not at all, etc. It's much simpler - if you try this route once, you will ask yourself afterwards, how have you ever endured this complex algebraic data types stuff.
Yes, you could still add a consumer of those Result<> values that demultiplexes them in two. But nobody does that. Because, why did we introduce Result values in the first place then?
This is just one of the many examples I encounter daily and that demonstrate how advanced type systems and ideas like OOP or FP encourage bad program structure. They are just not good ideas.
This is not demultiplexing into two different data streams, but handling both clumped together. Exactly what I was speaking of: Avoiding this clumping and processing the cases as separate streams (and independently!) brings huge payoffs.
Just try it; you will realize how much simpler your application gets - so many problems simply go away if you make more homogeneous data streams, and not fewer heterogeneous ones. And so many possibilities magically open up.
Very briefly, here is the difference:
def process_both_cases(stream):
for item in stream:
if case1:
do_complex_stuff_1(item)
elif case2:
do_complex_stuff_2(item)
else:
assert(False)
vs
def process_case1(stream):
for item in stream:
do_complex_stuff_1(item)
def process_case2(stream):
for item in stream:
do_complex_stuff_2(item)
I think it's immediately clear how much better the second version is. It's less complicated: The conditionals are not needed. Processing the cases is decoupled.
You... do realize that it's quite easy to make the former out of the later, and that's how every FP parser combinator library works.
Using a combinator to combine two undecorated funtions into one, you don't need to result to two streams, you can significantly simplify your runtime by only needing one. It's been written for you: http://hackage.haskell.org/package/base-4.11.1.0/docs/Contro...
Please consider that the incredible strength of ADTs is that you can just write generic code to combine those things without resorting to the crabtastic pattern matches that you're proposing, AND avoid the ugly implicitly unpredictably ordered world that you're describing.
Heck, we don't even need to resort to combinators for this. It's just you making up a complex case by saying, "What if I hide half of example 2 by handwaving it?" You still are going to have a case1/case2 decomposition in your second version. It's universally worse.
I don't see a pattern match anywhere in my example. And it's not in the slightest "unpredictably ordered" and has much simpler control flow.
> You still are going to have a case1/case2 decomposition in your second version. It's universally worse.
No. There is simply none. The producer makes them as two, and the two are processed independently. There is no problem if you don't make one. If you had ever tried the approach I propose out you'd know the flexibility that comes with it. (I'm currently working on language tooling, and there it is absolutely the case that it's very easy do without ASTs, and process the cases independently).
There, "forks" does contain an ADT of pointers into s1 and s2. But it contains nothing more, so it's very minimalistic. The advantage from the first case is still there: Most code profits from the homogeneity of s1 and s2, and from the actual, physical decoupling of the cases, and so much pattern matching just goes away.
(Btw. I'm not interested in fancy FP constructs. You will not convince me that you can process ADTs without pattern matches. You might be able to avoid writing them down in the source code, but I care about what actually happens on the hardware.)
(And actually I think avoiding to write down the pattern match in the source code and only generating it magically with higher level constructs is a bad, bad idea. It has poor readability. And it encourages the mindset "let's just do the worst shit on the actual machine. We can avoid writing it in the source code, and we don't care what happens on the machine and how slow it is. And we're too lazy to think about actual solutions that actually avoid real work".)
> Where I need to relate the individual items of s1 and s2 in a common ordering, I've had great success with this approach:
Sorry I misunderstood your use of the phrase "two streams." In my world multiple streams effectively severs causality between the streams. Or did I? You wrote:
def process_case1(stream):
for item in stream:
do_complex_stuff_1(item)
def process_case2(stream):
for item in stream:
do_complex_stuff_2(item)
If this genuinely relates to your example above then the only possible consistent ordering without any async/multithreading is to exhaust stream1, then process your errors. If you don't do that and the implication is that the streams won't close on errors but rather continue to emit, then you really can't guarantee any ordering to the processing.
Given this inconsistency, I'm going to continue the post with your intent of causality being preserved and assuming assuming that process_data() offers the next value (or null) and we're calling these in a loop. Because otherwise you have a whole heap of problems and your example doesn't even faintly resemble any subject matter from the post, or what most folks were talking about here. And if you believe that it doesn't destroy causality, you're either writing a lot of code to recombine your streams in lockstep (in which case, splitting is quite the extreme step to avoid matching) or you just don't understand the implications of stream semantics in this case.
> There is no pattern match there, sorry to disappoint.
However you provided a destructuring pattern match in your code!
s1, s2 = process_data()
Your process data is pattern matched on a generic product there. Sorry to disappoint. The actual type of the value is:
(Either Null S1Type, Either Null S2Type)
(And I assume it must be an `Either Null a` because we're discussing a case where either case1 or case2 will exist, so there's no valid value for one when the other exists.)
This is sort of why folks who know about ADTs roll their eyes at the sizzling invective about their complexity. You often end up using the same solutions, you just don't name them. It seems like the naming itself is objectionable. You've written about 4 posts I can see here about how much you hate ADTs, but then you're happy for an implicit `Either Null a` on every value, no opting out, and `a,b = f(x)` is fine because the name of the ADT is a punctuation mark, I guess?
As for the first example, Pattern Matches as Case statements are related in that a pattern match is a hyperactive case statement. So when you have one, you have the other to a varying degree.
> (Btw. I'm not interested in fancy FP constructs. You will not convince me that you can process ADTs without pattern matches. You might be able to avoid writing them down in the source code, but I care about what actually happens on the hardware.)
You seem to be making a huge deal out of pointer chasing. However, you're also suggesting a detour through the function contexts of process_case1 and process_case2 just to offer a null in one case.
Your plan seems a bit unfair, while we're on the subject. You're arguing that a pattern match is "heavy overhead", but you're completely ignoring the actual use case of "railway programming": to make a single clean codepath obvious and localize error handling at the end. But the thing you've offered doesn't really do that.
You're processing S1 and S2 (which I'll rename E1 for error), but what happens when I want to compose it? Extending the metaphor, your new code might look like:
s1, e1 = process_data()
process_ecase(e1)
// If you don't put an if( s1 != null ) here you're going
// to have to do a ton of function calls AND guarantee
// your handlers check their args for null.
if( s1 != null ) {
s2, e2 = process_case(s1)
process_ecase(e2)
if( s2 != null ) {
// ...
}
}
You could perhaps change the ordering of these ops (or push the null checks into your code which has a bunch of problems). And most of all, your compiler is totally powerless to help you work out if you have an error or not. You might also instead try to embed the dispatch to the next step in the def blocks of each individual step (an approach that could never work with the stream example, as you need to define your stream network (or at least a global namespace for it) before streaming events through it.
> And it encourages the mindset "let's just do the worst shit on the actual machine.
Okay, your example extended above. The Haskell-style of this combinator pattern (which DOES require all errors have the same base type, let me be transparent about that) is:
-- Assuming the steps are: a -> Either Error b
let pipeline = process_data >>= step2 >>= step3
pipeline () `catchError` errorHandler
The errorHandler is just a normal function taking your shared error type. Each step is just a function that returns an Either Error or its value. The values can vary so long as the types between steps line up (and the compiler checks this in F#, ML and Haskell).
It's easy to read, too. It's easy to extend, and it's predictable, and it's made from very simple decorated functions. We can even safely make it interact with exceptions, if we need to, without overly complicating the callers.
> And it encourages the mindset "let's just do the worst shit on the actual machine. We can avoid writing it in the source code
What's completely confusing to me about your claim here is that you're doing dynamic dispatches and perhaps excess function calls, demanding your process functions do null checks, you have no clear looping construct, and your compiler has 0 opportunity for any sort of inlining but the most naive, cache busting type. But you're essentially dunking on what becomes BNE statements at the machine code level. At runtime, all the types are erased in the best case or preserved in exactly the same way as any normal object with a dispatch table would. The actual cost is, in most cases, indistinguishable from your if-statement rails.
I actually think you like lots of elements of the FP style, you just seem to dislike the names. Perhaps if you could hold your nose with a clothespin and play through the stink of math-ish names, you might find more you like?
I think you're assuming I'm less experienced than I actually am!
> If this genuinely relates to your example above then the only possible consistent ordering without any async/multithreading is to exhaust stream1, then process your errors. [...]
Unless we know there are dependencies between the processing of s1 and s2 (and we don't; we're talking very abstract here) I'm totally unconcerned about the processing order, and I even stated explicitly that in the simplest case they're just append-only arrays. And the processing of the items in s1 and s2 can be done in any order, but why not just do it in a single thread, first process "data" completely, then "s1", then "s2". In many cases of course, it will be possible to run in 3 parallel threads, where the additional two process s1 and s2 as they are emitted from the first thread. But that's just a small optimization and a lot of complexity and not relevant here.
> Your process data is pattern matched on a generic product there. Sorry to disappoint. The actual type of the value is:
You're purposefully twisting words here. And no, that's not "the type", because I was (obviously) just using straight forward pseudocode whose semantics I assumed would be intuitively clear to everyone. That's not the kind of pattern matching we'd been talking about. In my code s1 and s2 are simply two known variables holding the streams. That's something different than a single variable holding either of two distinct cases. I assume you know that and just want to sell the FP cool-aid or troll me; please refrain from this style of discussion.
> This is sort of why folks who know about ADTs roll their eyes at the sizzling invective about their complexity. You often end up using the same solutions, you just don't name them. It seems like the naming itself is objectionable. You've written about 4 posts I can see here about how much you hate ADTs, but then you're happy for an implicit `Either Null a` on every value, no opting out, and `a,b = f(x)` is fine because the name of the ADT is a punctuation mark, I guess?
There is so much hair-pulling in here and what you say is simply wrong. I was just returning two friggin' variables. And why are you talking about "Null"? That's of no interest. There is no use in my pseudocode for any kind of thing resembling what's called "Null" in any programming language. As a Haskell enthusiast you should know that there is always a canonical stream/array, and that's the empty one. Why are you haunting me with Nulls, that's not fair, I don't want them, and I can do without them. (And just to be clear, I don't do OOP either; I think it's at least as bad as FP)
> What's completely confusing to me about your claim here is that you're doing dynamic dispatches and perhaps excess function calls,
No I'm frigging not. I don't get it why you're talking of dynamic dispatches now. What I have given is some pseudo code of the simplest kind, that easily translates to any procedural programming language, including assembly code. Why do you make it seem like it was something more complicated? You must be deliberately hair-pulling just to win a stupid argument. I'm not doing any "virtual object-oriented blah blah" or any kind of types craziness.
s1 is a stream (or let's just say (again), an "array", to hopefully prevent any misunderstandings) of homogeneous things. Let's just say all items in s1 have type S1 and all items of s2 have type S2. It's the simplest kind of problem.
> -- Assuming the steps are: [..]
Congrats. I learned Haskell once, too. Until at some point it became clear that it didn't bring anything practical to the table and only complicates program designs. This "uniform error handling" idea (actually it seems to be more of a necessity/accident) just makes for bad code. Don't do that. Just do the first processing step, handle errors accordingly. Then do the second step, and handle errors accordingly. The second step in general requires different error handling than the first, (and often the only thing they have in common is that they might "die"), and there is no need and only disadvantages in restricting them to have uniform error handling.
What's wrong with my procedural pseudo code? Fill in the required error handling between the steps as soon as you have a concrete instance for my abstract example.
> dynamic dispatches
Nope
> excess function calls
What's that?
> demanding your process functions do null checks
No, I don't have any "Nulls" of any kind, since my data is simple and normalized.
> no clear looping construct
I used a pseudo-code for loop. Every 1-month beginner to programming has an immediate grasp of that. I never heard anyone complain that was somehow not "clear". It's plain, procedural code. You're trolling me so hard it hurts.
> I actually think you like lots of elements of the FP style, you just seem to dislike the names. Perhaps if you could hold your nose with a clothespin and play through the stink of math-ish names, you might find more you like?
I've been there and I know to use FP-ish constructs when they make sense (very rarely. Much easier to restrict to procedural altogether).
> Unless we know there are data dependencies between s1 and s2 (and we don't; we're talking very abstract here)
The entire discussion is about railway programming for early exit on error handling. So yeah, there are dependencies there.
> You're purposefully twisting words here.
No. I'm pointing out you pattern matched against a generic 2-tuple in your pseudocode. There's no words to twist, returns in functions on modern hardware are atomic over 1 value in all but the most obscure languages, and it's not a new phenomenon.
If it's valid to say, "I returned 2 values" it's equally valid to say, "I returned one value and restructured it." They represent the same thing.
> And no, that's not "the type", because I was (obviously) just using straight forward pseudocode whose semantics I assumed would be intuitively clear to everyone.
Both statements can be true. I'm simply writing a type that captures your "obviously true" statements. Therefore, both are obviously true.
> There is so much hair-pulling in here and what you say is simply wrong. I was just returning two friggin' variables.
You're the one consistently lambasting folks about paying attention to what happens at the "bare metal" (ha!) level. I encourage you to take your own medicine here.
> Why do you make it seem like it was something more complicated? You must be deliberately hair-pulling just to win a stupid argument. I'm not doing any "virtual object-oriented blah blah" or any kind of types craziness.
Either you wrote something so vague it can be anything you want (e.g., your magical streams that are also not streams). If we remove the nonsensical (or I guess by your admission, non-sequitur) streams bit then you're left with this outcome. I told you exactly what I'd do in my post. You have to recognize the type of the result of each step as either a success or an error.
You've chosen to pretend none of this is about error handling, which is pretty boring IMO. Error handling is what makes code actually hard to write.
> What's wrong with my procedural code? Fill in the error handling between the steps as soon as you have a concrete instance for my abstract example.
I admit I don't know since it seems to mean exactly what you want it to, and no one else is privy to either the viewpoint nor the problem you're actually discussing. You're just here hurling vague insults about how everything other than your "obvious" pseudocode is nonsense.
> Dude, I used a pseudo-code for loop.
Not in any example I reviewed that didn't completely ignore the actual problem to burn down a straw man about streaming constructs. Also, don't call me dude. I'm not.
> I've been there and I know to use FP-ish constructs when they make sense (very rarely. Much easier to restrict to procedural altogether).
Considering you didn't recognize the product pattern match and objected vigorously to the idea that it might be as such; and tuples something nearly every statically typed FP dialect has? I gotta confess that I don't think you know the field or the techniques very well. Fair enough, many modern FP techniques in use today are less than 25 years old. No shame there. But you should probably investigate a bit more before speaking so confidently on the subject.
You really are just advocating for doing what the F# example given was doing, without all that helpful syntax or compile-time verification of correctness & totality. I dunno why you'd want that. Given that you are accidentally reusing FP pattern matches, they can't be that complicated now can they?
> The entire discussion is about railway programming for early exit on error handling. So yeah, there are dependencies there.
"Early" does not have anything to do with "dependency". You can do a million unrelated things in an order of your choice and abort at the first error. There need not be any dependencies.
> returns in functions on modern hardware are atomic over 1 value in all but the most obscure languages, and it's not a new phenomenon.
This is true and totally unrelated to my point.
> No. I'm pointing out you pattern matched against a generic 2-tuple in your pseudocode.
Again, it's pseudo-code, so no, it need not be a "generic 2-tuple", not even needs to be a "tuple" (whatever that means in a given PL). In a real implementation the types are known precisely (Just like the types of the alternatives in an ADT are known precisely). There is no kind of "matching" (there are no conditionals) whatsoever. s1 and s2 are simply two variables (for example pointers, in C) of precisely known type. And the process_data() function returns two values that get assigned to the variables. It's the simplest kind of code.
The whole point of the discussion was whether to design the output as a single stream of ADT values, or as multiple streams of precisely known values. In this context it's totally clear why my (single) assignment to s1 and s2 is not the "matching" that you need to do on the (many) values of the single stream in the ADT version. Really I don't care how hard you want this assignment to s1 and s2 to be a matching/destructuring of a "tuple"; It doesn't change anything and is totally unrelated to my point.
Sorry to call you "dude". I had already decided to remove that word before reading your answer.
> Again, it's pseudo-code, so no, it need not be a "generic 2-tuple", not even needs to be a "tuple" (whatever that means in a given PL). In a real implementation the types are known precisely (Just like the types of the alternatives in an ADT are known precisely). There is no kind of "matching" whatsoever. s1 and s2 are simply two variables (for example pointers, in C) of precisely known type. It's the simplest kind of code.
That's not different syntactically or even mechanically from pattern matching over a generic 2-tuple. So if you want to do this as a social convention, that's fine. I'm not sure why other people would adhere to such a convention.
> The whole point of the discussion was whether to design the output as a single stream of ADT values, or as multiple streams of precisely known values.
No. The whole point of the talk we're discussing is that a single stream of chained operations that have failure states is difficult to cleanly assemble without ADTs. You've backed away from the error contingency in your argument. To the uncharitable onlooker, this appears to be because ADTs solve the problem elegantly and procedural programming turns into am error prone, one-off pile of if-then-else spaghetti. See also: every complaint about Golang error handling.
Ignoring the Success-or-Failure nature of the talk is to fundamentally ignore the problem it's trying to solve. If you DID have two streams of values that had no contingency, we wouldn't write code this way in any runtime with async primitives.
> Sorry to call you "dude". I had already decided to remove that word before reading your answer.
> The whole point of the talk we're discussing is that a single stream of chained operations that have failure states is difficult to cleanly assemble without ADTs
And I suggested that this is overly complicated and procedural approaches are better. In procedural code we don't "assemble chained operations". We do them one-by-one and handle errors appropriately.
And we can typically arrange the data and control so that error handling is much easier. That was my point.
> one-off pile of if-then-else spaghetti.
ADTs and FP techniques provide only syntactic sugar to avoid this spaghetti in the source code. But due to that they just encourage having the worst mess of control actually happen on the machine.
On the other hand, I demonstrated how to avoid this mess by decomposing data into individually homogeneous bins/streams/arrays/whatever. My example does not have a single if statement (yes, you might argue there are if statements hidden in the for loops).
I clearly stated that it's often possible to choose this approach.
> And I suggested that this is overly complicated and procedural approaches are better. In procedural code we don't "assemble chained operations". We do them one-by-one and handle errors appropriately.
Go does this. It's widely and I think rightly criticized for this. It's "easy" to do but not "simple" because it's prone to human error.
> And we can typically arrange the data and control so that error handling is much easier. That was my point.
You can do more work to validate in a separate pass if we're discussing data transforms. It's harder to do this for a series of chained imperative steps. The talk details an example making a series of API queries. It's fantastic to handle a CRUD loop with the ADT method because it lets you write a simple specification and collect errors out in the end, while reserving exception handlers for truly exceptional cases like network faults. F# is incredibly good at this kind of flow.
> ADTs and FP techniques provide only syntactic sugar to avoid this spaghetti in the source code. But due to that they just encourage having the worst mess of control actually happen on the machine.
You've said this multiple times, but it's just not true. The F# code is just a function that calls functions. F#, ML and Haskell compilers all aggressively inline these calls. You pay the exact same costs for validating an error exists in the Golang style, in practice. Indeed, lots of people generate Go from liskov-style templates JUST to get ADTs to make error handling less painful. It's gaining popularity in Swift and Rust as well. Folks there aren't trying to impress people with their FP knowledge, they're cribbing a technique that works and scales well.
> My example does not have a single if statement (yes, you might argue there are if statements hidden in the for loops). I clearly stated that it's often possible to choose this approach.
Your example, by your own admission, doesn't address error handling flows. You need conditional logic to work with operations that don't always succeed. You say, "Structure it so you never have them." I think a lot of FP people would agree with this (in fact, there is a really cool functional pearl about doing exactly this at scale: https://github.com/matt-noonan/gdp-paper/releases/download/j...) but they'd think about it less as a function of assurances and "thinking very hard" and more about using methods to make inconsistent code provably false at compile time.
Very few languages can actually encode formal notions of correctness, right now. Recently languages like Idris, Agda, Coq, and TLA come up as tools for formal correctness, where you can actually have your compiler prove your invariants and their implications.
I'm sorry, I ignored the second part of your question. I was responding to:
> What is an example of exception that happens many times per second?
I made a distinction between exceptions and errors because exceptions are a specific mechanism for propagating errors. I wasn't trying to be pedantic, I was trying to point out that the answer to your first question is more obvious if you subsitute out the word exception for error.
The answer is simply: Because you're probably the one generating the errors and you're going to generate a lot of them.
With regards to the second question:
> And where it could not be easily avoided to step on the error path at all?
I'm not sure how to interpret this question, but I'll throw out an answer anyway. You don't need to take a Railway Oriented Approach. There are other ways. Your reply was an example.
For a streaming scenario, applying a function with the signature ('a -> Result<'b,'e>) to a Stream.map is pretty trivial. It may not be the most performant or scalable, but it's easy to do and easy to reason about. However, one could also easily create two streams as you suggested with a partition type function with the signature: (Stream<'a> -> ('a -> Result<'b,'e>) -> (Stream<'b> * Stream<'e>)
You're free to use the Result<> type as much or as little as it makes sense. How one should best solve the problem depends on the scope of the problem and what you need to do downstream, right?
> I wasn't trying to be pedantic, I was trying to point out...
I agree!
> but it's easy to do and easy to reason about
I agree it's an obvious/intuitive approach to take, just like OOP. However, from a computational and software maintainability standpoint, I think it rarely pays out.
> How one should best solve the problem depends on the scope of the problem and what you need to do downstream
That's right, however ADTs are very rarely occurring naturally in what I do (I've dabbled in quite a few areas).
I've found them to occur naturally in parsers (because languages have forks in their syntaxes), but even there I've had great gains by avoiding ASTs altogether, and making multiple streams, supported by a ADT stream of simple data that does not contain real data but only "joins" the other ones (it contains only a type-flag and a pointer into one of the other streams).
I think ADTs are the right choice whenever the ordering between a set of things with heterogeneous types is very important. If that's not the case, code can be vastly simplified by sorting the data by their types and processing them as separate bins - separately and independently.
There is no performance penalty associated with try/catch. At least in JVM languages, try is free (its a compile time phenomenon) and throw is also free (memjmps), its only the creation of the exception that is expensive and this can be avoided by pre-allocating an exception (at the cost of no stacktrace info).
Developers are better off in the end with proper exceptions because these sorts of Result objects don't scale. Haskell ultimately embraces proper exceptions and I suspect Rust will get there too eventually. F# actually has proper exceptions which makes you wonder why the author is reinventing the wheel badly.
There are different ways to implement exceptions with different performance trade-offs.
Mechanism used by CPython (and IIRC many C++ implementations) involves having per-thread flag "exception occurred" which is checked after every function return (this makes sense performance-wise when essentially every function contains "implicit try/finally" in form of bunch of Py_DECREF or destructors of locally scoped variables).
The same idea can be implemented by returning instances of special "ExceptionThrown" class and checking for that, which you in fact can do by doing "if argument is ExceptionThrown return argument" in prologue of every function, which is exactly the thing described in the article. (early versions of dfsch did exactly this)
Only when you know that in most cases you don't need to do significant cleanup on stack-unwind (ie. you have tracing GC that directly scans stack or you don't allocate that much) it starts to make sense to do stack unwinding by doing non-local jumps.
Typical implementation of that has some linked-list of currently active try/catch/finally sites that has to be maintained at runtime (typically as part of activation records on control stack). Extension of this idea is that such sites do not necessarilly have to unwind-stack before running the recovery code (i.e. Windows' SEH). And then that you can have two such lists, one for handling errors and other for cleanup during stack unwind (i.e. CL-style condition system)
So the author isn't reinventing the wheel. I should have explained that Railway programming isn't supposed to replace doing try/catch, it's typically best suited for scenarios where errors are not exceptional. It works best when errors are occurring very regularly, are part of the logic of the system, and where you want to ensure the programmer takes advantage of F#'s exhaustive pattern matching to ensure developers account for all the scenarios.
In this scenario, performance matters because you want the creation of the error (these aren't exceptions) to be relatively cheap because it is a regular part of the system.
In these scenarios, using exceptions would be abusing exceptions, because exceptions are typically supposed to be exceptional.
For a web server under high load, exceptions are extremely expensive to build the stack frame as mentioned below. The rule of thumb is to reserve Exceptions for Exceptional Circumstances (oom, unable to connect to db etc) and to use a pattern like in this article for un-exceptional exceptions such as required field validation errors. Having done this clean up a few times, it makes a huge difference.
I've had a guy come in on an F# project that used exceptions in a relatively data intensive workflow and rewrite it to use Results like this.
The amount of boxing and unboxing as you go through the various switches actually can ultimately make this much less efficient than the exception based flow. The code was ultimately something like 8 times as slow, and topped out CPU on beastly boxes which was surprising to me.
Performance myths like this are really interesting. Usage of features is usually not inherently responsible for bad performance, but the way they are used. Usage of these features can amplify the effects of bad software architecture. Some features might just be elected as the culprit while there is really a different problem.
If a little boxing and unboxing in a functional language takes 7 times as long as the rest of the code, something must be very very wrong. It's hard to believe that.
In this concrete case: a data intensive workflow should really have no or only few error situations. The performance of exceptions should not matter at all. I would guess that reworking the code to avoid exceptions also included reworking the program structure.
I don't believe the boxing/unboxing performance claim either. I built a large project in this style (in F# specificially) and the efficiency is not a problem. It's possible said person built his custom implementation that had issues but if you use ILSpy to decompile how union types end up compiled you'll see it comes down to a boolean check against an integer followed by a pointer dereference.
OCaml has very fast exception throwing/handling mechanism which apparently is why some of the applications built in OCaml make heavy use of exceptions for control flow (which as been mentioned by others, if you push too far is a nightmare to to reason about). The appropriate strategy to port such code to F# for example (which is a highly similar language to OCaml) is to switch to unions in the return type.
The approach advocated by Scott is a general pattern with which you build an entire application around. It does marvel when building a system surrounded by others that you cannot trust and database with brittle consistency (as most 15+ years line of business database eventually become).
As for some of the other suggestion advocated in the comments, you have to note that composability becomes a factor. If you are dealing with a collection of entries each of which might fail or succeed, the railway oriented programming approach scales to handling collection of errors/successes gracefully which so many error handling strategies fail to meet.
I have tried designing an application around Haskell's error monads in the past, and it was an absolute disaster. There was so much typing. The types became really hard to understand and the program structure became really rigid. I do not know that it can't be done better, but I've heard prominent Haskell guys say that combining errors (or any monads at at all) sucks in practice.
And I've got zero problems with just handling errors procedurally. I realize that errors simply combine very badly: If there are many possible error kinds the best you can do in the end is usually to just quit. That's not what they promised on the tin :-)
So in the end I just write procedural code and I'm careful not to get into situations where many different kinds of errors can happen. I select a few error cases that I care enough about to handle. Because handling even a single kind of error means a lot of complexity on top of a software project.
Yes I agree it can become unwieldy if it is used throughout an application. I think the sweet spot is too keep the pattern at the application boundary. You can use business objects the way Scott describes them for the internals without threading Either/Results monads throughout your application. Moving calls to services and databases to the application boundary also helps keeps the internals (where most of the business logic will reside) clean of Either/Results. For the few things left that could fail in the internals, stick to exceptions mechanism and avoid catching. I'm a big believer bugs should make the application crash.
You're describing a highly sub-optimal solution if there was performance degredation, and lots boxing/unboxing is a code smell.
The biggest Exception penalty is only paid once when the exception code needs to be JITed along with more minor performance issues. Dollars for donuts, any CLR language will be faster using not-Exceptions, so a refactoring to model the exception path would have to do a fair bit work in unrelated areas to be slower.
I think you misunderstood his arguments. He's stating that favouring a Result-type monad (as opposed to using exceptions that may never occurs) leads to boxing of values inside a tag of a union-type. Threading exceptions handling where exceptions are never thrown in practice typically little cost.
However this is really different from boxing of value-types in a situation where you don't have access to generic collections for instance and as such we're talking about a different cost. Unless you're entire application is passing integers or floats around (and not proper objects/records) it's highly unlikely the pattern is causing the situation he described.
This just isn't true at all. Exceptions use a totally different mechanism from railway-oriented programming, have very different constraints, and different performance characteristics.
Please don't entertain these silly posts. At best, it only grants legitimacy to a crassly presented idea.
I'm wondering: why not make every returned value implicitly of type Either with Left some generic type?
In most cases, I'm not interested in the types of exception that can be thrown. And when I need the type of an exception, a runtime check suffices in almost all cases.
Let's not make exception handling more cumbersome than it needs to be, and only invoke verbose mechanisms when necessary.
> I'm wondering: why not make every returned value implicitly of type Either with Left some generic type?
Because then you lose the ability to have code that doesn't error. If you explicitly mark those parts of your code that can error, you can then start to decouple those parts that can error from those that will never error. E.g. you can separate the business logic which figures out which database query you need to run (which can't error) from the code that actually performs database queries (which contains timeouts, retry logic etc.), and then you can test those two things separately and make sure you've covered all the cases. Whereas with pervasive/implicit exceptions, every line of your business logic might throw an exception (or might not throw an exception now, but later be refactored to throw one), so you need to test a combinatorial number of cases (or, more likely, your tests won't cover some code paths).
Wrote a reply to a deleted comment, figured it was worth keeping to expand on my point:
> - what if the tables / schema / etc doesn't exist?
Then figuring out which query to run won't fail, executing the query will fail; the whole point was to separate the two.
> - what if the logic which selects the DB query returns no query?
You make that impossible, if that's not a valid answer. (If that is a valid answer then you use a return type that represents that - Maybe/Option - and then you're forced to handle that porperly). Most programming languages don't even permit a function to "return no value" so that's already not a problem. (Some programming languages permit a function to return "null" or loop forever; don't use those languages).
> - what if the logic attempts to pull a value from an out of bounds array (or similar fail)? Sure you would check the bounds before querying the array but why did you get to a situation where you tried to pull an out of bounds index in the first place?
So you don't get into that situation: you only use array indices that are valid by construction, you don't give yourself any way to construct an invalid index. (Ensuring that indices into a specific array are a different type from indices into any other array is a useful first step. Of course, in practice you probably avoid using arrays and indices at all; if you use higher-level operations like map and reduce then there aren't any indices to worry about).
(Very occasionally there may be cases where you can't avoid having to do something that might error, in which case what you want to do is fail-fast, erlang style. But if you're doing what I'm advocating, this will be your only option: the point of all the above is that we make invalid states unrepresentable, so when we get into an invalid state it's simply impossible to continue. And really the vast majority of the time - every time actually, in my experience - you can find a way to write the code so that it can't error, if you actually try).
Realistically, almost all but the simplest code can produce an error, starting from out-of-memory errors.
You can break up code according to many criteria, but doing it based on whether the code throws an error or not seems not very practical, since you can break up code only once.
Further you could have smart test tools that can figure out what exceptions can be thrown in what circumstances. You don't need to bother the programmer with them.
> Realistically, almost all but the simplest code can produce an error, starting from out-of-memory errors.
If you care about handling out-of-memory errors then you absolutely do want to distinguish between code that allocates and code that does not. The vast majority of application code is content to treat out-of-memory as an impossible condition and fail-fast in the case where it does happen. So sure, technically the distinction is between code that can produce errors that you want to handle and code that cannot produce errors that you want to handle, but the point goes through all the same.
> You can break up code according to many criteria, but doing it based on whether the code throws an error or not seems not very practical, since you can break up code only once.
Nonsense. You can, and should, decompose your code along multiple axes, including any effects, of which error handling is indeed only one.
> Further you could have smart test tools that can figure out what exceptions can be thrown in what circumstances. You don't need to bother the programmer with them.
"smart test tools" would amount to an ad-hoc, informally specified, bug-ridden implementation of half a type system. The distinction between code that can error and code that cannot should be as lightweight as possible, but it mustn't be completely elided, because the programmer needs to be able to see it when they're working on the code. The "=" versus "<-" distinction in Haskell-style "do notation" is the best compromise I've seen: it's minimally intrusive, but it is visible.
Yes, I understand. But by "implicitly" I really meant that. You don't see the type, only the compiler does. The programmer only sees the non-error part of the type (the "Right" part of Either).
> Either is a sum type. What if I don't want that.
Product types work as well, if you want. But unless your return type is a monoid (or like, a semilattice maybe?) then you're going to end up with Sum Types somewhere.
The vast, vast majority of software engineers and students work with sum types every day and they're totally fine with them. Most languages implicitly sum many values with null.
These sounds like exceptions rather than errors to me. Difference being, exceptions truly are exceptional and not anticipated; like running out of memory. How do you even handle that? Would probably let it crash and restart (Erlang model). Errors is something you anticipate can happen, such as getting 4xx-5xx back from the server and know what to do in those cases.
Then it's fair to say that you definitely can't get an error by adding two numbers.
