- 1 Chans & goroutines vs Reactive Extension
- 2 Broadcasting on a set of channels (golang-nuts)
- 3 A proposal for generic in go
- 4 golang-nuts: using channels to gather bundles from inputs for batch processing
- 5 golang-nuts: concurrency safety
New Oct2015, updated 27Nov2020
I think it’s fair to copy my own comments at Go Forum in this note. I would also do newer golang-nuts and golang-dev matters that I have taken part in. The latter forums I won’t backlog.
You will have to go there to read the whole threads. These are basically my words.
Disclaimer: this blog not would basically be about matters I want to know more about!
Newest at the bottom:
Chans & goroutines vs Reactive Extension
(15Oct2015) I created this topic, see Chans & goroutines vs Reactive Extensions. My starting point for this was while writing an update in A reactive manifest. This again was triggered by a meeting I attended: Funksjonell reaktiv programmering i Swift, med Clojure backend (Trond H. Bordewich et al.) at www.digs.no here in Trondheim.
Interesting references in the Go Furum post are:
“I’d love to see a comparison of Golang’s goroutines/channels vs async”
“GoFlow – Dataflow and Flow-based programming library for Go (golang).”
“This is quite a minimalistic implementation of Flow-based programming and several other concurrent models in Go programming language that aims at designing applications as graphs of components which react to data that flows through the graph.”
“NSQ is a realtime distributed messaging platform designed to operate at scale, handling billions of messages per day.”
“Regarding NSQ, Go channels (not to be confused with NSQ channels) and the language’s built in concurrency features are a perfect fit for the internal workings of nsqd. We leverage buffered channels to manage our in memory message queues and seamlessly write overflow to disk.” More here:
“Also, all of our C apps are built on libevent – i.e. a single-threaded event loop with callbacks that enables us to efficiently handle many thousands of simultaneous open connections. We use the same technique in Python via tornado, but the amount of work a single Python process can do is limited largely by runtime overhead. Additionally, this style of code can be hard to read and debug when you’re not used to it. Go’s lightweight concurrency model allows you to write in an imperative style while providing a built-in way to leverage the multi-core architecture of the computers we operate on.”
Some of my comments
Of course I understand the difference between a programming language and a pattern, between bricks and a house. It still puzzles me to see all these frameworks evolving from use with a non-concurrent languages (like Swift) that don’t have any notion of process (like gorotines) and communication (like channels) and synchronisation (like zero-buffered or full channels) ported and embraced on concurrent platforms (like go). Whether it’s push or pull with channels is kind of orthogonal to the problem, isn’t it? There can’t be any reception before production anyhow.
And it puzzles me to read things like the below (quote from ycombinator in my initial question):
(reubenbond at ycombinator) Channels in Go don’t compose: I can’t easily take one channel, mutate the values as they arrive, and create another channel from the mutated values. With Rx, that’s just var uppers = keyPresses.Select(key => key.ToUpper()) and now I’ve got a new stream of data. If keyPresses completes, so does uppers. If it fails, that failure is propagated through uppers. This isn’t easy in Go.
goroutines don’t have any reasonable kind of monitoring. I can’t say “this routine has completed/failed”, I have to implement that notion every single time using a WaitGroup or something, and that only handles completion, not failure.
How correct are the answers? It’s not excactly the syntax “var uppers = keyPresses.Select(key => key.ToUpper())” on go, but it’s not much different, is it? Once channels have been connected between goroutines. “Composition: A Way to Make Proofs Harder” by Leslie Lamport certainly rules CSP to be compositional, and any goroutine (or PROC in occam) can spawn a new process. And with mobile channels (and go channels) you can spawn any channel you like at any time, except you’d have a way to connect that new channel (like sending it over an existing channel). Isn’t this enough?
Would a web server framework written in go for go from the onset look different than the others? It looks like GoFlow is one such? How does it differ from how a Reactive Extension for Go would have been?
(As you understand, I see this from an embedded programmer’s bench, but being curious about this)
Gorilla web toolkit
This is perhaps not a reactive framework as such, but then.. is it?
From https://groups.google.com/forum/#!topic/golang-nuts/m5U2Bkb88Dk I read “For session handling, roll your own or use Gorilla’s ”
This is perhaps what I’m after: “roll your own or use Gorilla’s”. Saying the same in a Swift or C++ world, I assume would be much more consequential. It’s not as easy to roll once own in those languages as in Go? So, those frameworks are mostly not driven by the Go society? “Package gorilla/sessions provides cookie and filesystem sessions and infrastructure for custom session backends”, see http://www.gorillatoolkit.org/pkg/sessions
Broadcasting on a set of channels (golang-nuts)
See goolan-nuts (here). I was late in the discussion with this comment (here). The thread established the idiomatic way to do broadcast on a set of channels early (by Rob “Commander” Pike). Case closed…
But then the theme switched into sending pointers over channels, and it was said that this was ok. Hmm. I posted this question (less typos):
occam pi has/had mobile channels, where data pointed to by the pointer that’s implicitly sent over the channel will fall out of scope on the sender side. I don’t even think that data is converted to a const, the data won’t exist anymore. The language takes care of this.
Sending a pointer across without such a mechanism is rather risky, isn’t it – if one doesn’t introduce a “taken” return message and all parties respect it? If not, is it avoidable to insert data state and eventual polling?
A proposal for generic in go
See goolan-nuts (here). As always I am late (29June2016 at 15:42):
The suggestions of generics discussed here and in the referenced documentation, will it be possible to compile the “Go-with-generics” language into some idiomatic Go and compile it with the standard compiler? (I guess what idiomatic means is the real question here..). Or would it break idiomatic Go?
If it won’t break it, wouldn’t it be better to add some of this instead of leaving some of these important matters out? (I have read a guy mention concurrency and Haskell; the language is so flexible that it may be added any way. And it has. So there is no standard way for it)
Also, would it be possible to add generics around channels to (limit their functionality I assume (like not legal to address both sides of a channel in the same goroutine (I know it’s sometimes nice)))?
And make formal analysis (like transformation from Go-with-generics to CSPm) easier?
After some comments I added this:
I assume the scope of the discussion about introducing generics is not how Go-generics might hypothetically be transformed to idiomatic Go. I find no reference to “unsafe” in neither of these (already mentioned here):
- “Proposal: Go should have generics” by Ian Lance Taylor https://github.com/golang/
- “proposal: generic programming facilities” by Andrew Gerrand https://github.com/golang/go/
issues/15292Neither do I find any mention of “reflect”, which I assume might be relevant.From this I infer that adding generics to Go is a rather large affair. It also looks like that in the pages above.More interesting, I don’t see formal modeling verification mentioned in any of the generics-documents above, neither in this recent thread:
- “formal verification in Go? someday perhaps?” at https://groups.google.com/
forum/#!topic/golang-nuts/ MVITBF3TcOEI know that Rob Pike was involved with formal verification since long (“Bell Labs and CSP Threads”, by Russ Cox, see https://swtch.com/~rsc/thread/ ) and that this background was colouring much of how Go ended up. Has generics been discussed along this line: that Go-generics might be more/less suitable for formal verification?
There is an interesting comment after this.
golang-nuts: using channels to gather bundles from inputs for batch processing
This guy uses buffered channels for his problem. I posted a discussion on that theme, and mentioned some alternative patterns like the Overflow buffer pattern, the Knock-come pattern and the Go pattern shown in XCHAN – shown in the Appendix: Mixing Input and Output in the Go select Statement. Also, whether an output into a zero-buffered channel in a select with a timeout might do.
Here are my references from that posting:
 Rob Pike: “Go concurrency patterns”: https://www.youtube.com/watch?
Disclaimer: there are no ads, no gifts, no incoming anything with my blog notes, just fun and expenses:
golang-nuts: concurrency safety
At golang-nuts (here) 26Nov2020. My comment the day after:
Thread safety, concurrency safety, goroutines safety. The same stuff to me.
It’s important that concurrent goroutines don’t interfere with each other in such a way that internal state is compromised by other goroutines. Typically through access to internal data, through pointers. If only go channels are used to communicate between goroutines, than that case is closed. That’s one of the reasons they exist – also in other similar languages, with that aspect often based on CSP.
But then, goroutines may communicate in a circle, the smallest would be two sending to each other and awaiting answer from each other at the “same” time. A system would easily freeze from this point and out. This would cause deadlock, which is pathological. Using deadlock free patterns may solve that problem from the beginning, provided such a pattern is used correctly.
Finally(?), one goroutine could be busy by itself, or by communicating with others, so that it is not able to read vital messages from other goroutines, ever. This would be a livelock.
I think go comes with a tool  that would help for some of these types of problems. I have not tested it myself, since I use another, embedded concurrent language. But I do read these mail lists every time they come, so I try to learn what this community is up to.
There would be ways whereby one could model concurrent programs as well. Lots of modelling languages exist. Myself I have to some extent used CSPm (with the FDR tool), FSP (with the LTSA tool) and Promela (with the Spin tool). https://golang.org/doc/articles/race_detector.html
There are some good comments also around my comment. Like this, by B Candler:
Here’s a video explaining how channels can replace various concurrency primitives: it’s worth getting your head around: https://www.youtube.com/watch?v=5zXAHh5tJqQ – GopherCon 2018: Bryan C. Mills – Rethinking Classical Concurrency Patterns.
Some main points from Mill‘s excellent lecture:
- An asynchronous API return to the caller before its result is ready
- Problems with futures and async/await pattern
- In go: Asynchronous ≡ synchronous. In go, synchronous and asynchronous APIs are interchangeable
- Concurrency is not asynchronousity
- The downsides of condition variables (monitors)
- A buffered channel can be used as a semaphore
- Share data by communication the data
- Share completion by completing communication
- Events can be completions
- Share a thing by communicating the thing
- Worker pool pattern (=thread pool)
- Start goroutines when you have concurrent work to do now