This year past I had the privilege of speaking at Clojure Conj in Washington DC. This was a blast.
I also had the opportunity to speak at YOW LambdaJam in Brisbane. Lots of great people to talk to here.
I also helped lead ClojureBridge in Sydney.
I also gave four talks at clj-syd:
I’ve also been working on a book: Clojure Recipes.
I did a talk at Clojure Conj 2014 on applying the paradigms of core.async in ClojureScript by porting Clojure Ants to ClojureScript.
This post is about providing a link to a demo of the application that is easy to run.
We ran Clojurebridge in Sydney today. I was surprised by how well it turned out. I’m writing to capture the essence of this experience.
I’ve experienced several training courses paid by for by work. There is kind of a vibe on these that whilst you’re keen to learn, you’re also having a “day off” work – so there is lack of intensity about the day.
I’ve also been to a couple of training courses that I paid for myself. For these you’re generally using your own annual leave, and have a sense of doubt (and/or suspicion) about whether the trainer is really up to scratch.
In both these situations – there is an sense of self-interest that just drags the day down.
Clojurebridge today was different to this because everyone came out of interest in the subject matter, and in contributing to the day. It was a Saturday which had a sense of ease about it. Attendees had given up their personal time because they wanted to be there, but it lacked the commercial pressure of a paid-for course.
In addition the course materials and programme have a strong theme of inclusion, newbie-friendliness and gentle pace.
Clojurebridge aims to be inclusive of women and minority groups. I honestly had absolutely no idea who would rock up.
I chatted with a merchant banker who was changing careers. There were uni-students keen to get into what functional programming was about. There were financial services types who had done a decade of OO on legacy platforms and were looking for something different. There were several people who had never done programming before and just had a computer and a web browser.
The most surprising thing was the couple-pattern. We had six couples rock up and spend the day programming together.
Everyone was really positive, willing to chat, share experiences and ideas.
I rocked up to Thoughtworks in Sydney last night having worked through the training materials to do the ‘night-before installfest’. (We forked the training materials to fix some typos, and also provide some explanation of the materials for people who only knew metric units and not imperial, and also to adapt the example questions from the US tax systems to the Australian GST system).
We welcomed everyone with cupcakes. These were consumed eagerly, there was no lazy semantics.
We spent the evening eating pizza, installing the JDK, Leiningen and Lighttable. (Interestingly enough the Windows machines proved the most problematic, and I am totally going to do a pull request to fix this). This was all done with a drink in the hand.
I helped install Leiningen on two Windows machines with a Polish language setting. It turns out if you squint until the writing goes fuzzy, then you can just rely on muscle memory and it all works the same!
The following morning we welcomed people with muffins and coffee and got into it. The pattern of the material was to introduce a topic for 10-15 minutes, and then leave 40 mins for an exercise, and start again with the next topic. The mood was relaxed, with time for conversation.
Late morning we wandered out as a group down to Circular Quay and grabbed a coffee and it retained this relaxed social fun vibe.
We finished the day having a drink and reflecting on the day.
I left early for dinner with my family, but it sounds like the party rocked on into the night.
I won’t say that if you rock up to a ClojureBridge as a trainer or attendee, you can kick-back and it will be all roses. It works because of what you bring to contribute, your curiosity, your energy, enthusiasm and willingness to learn. As a trainer it works because of your patience, your passion for the topic, your empathy, listening skills or just calm critical-thinking-skills working under pressure.
You should do this! It’s so much fun.
(Did I mention I’m writing a book called Clojure Recipes?)
Introduction
These are my reflections on the talks. I’m trying to be open-minded and take a balanced view of the positive and negative opinions that were exchanged. If that’s not your thing then this probably isn’t for you.
The Venue
The Warner theatre was amazing. It reminded me of the State Theatre back home in Sydney.
Inside the place had a pleasantly baroque feel, and everyone seemed to be relaxed and comfortable:
Day 1
Unlocking data-driven systems
[Video] [Slides] [Twitter] [Blog] [Pre-Conj Prep] [Github]
Paul deGrandis opened with a story about his experiences on a Cognitect client. He kicked off this truism that was echoed throughout the conference:
Data > Functions > Macros
Now the reason why this was important was this snippet of detail:
“These 26 lines [of #Clojure] replaced 2000 lines of JS and 1000 lines of HTML”
Now there wasn’t a lot of detail on that or how it worked. Indeed, some criticised this saying it was an advertisement for Cognitect about proprietary technology.
To some the tone of this critique seemed a little Statler and Waldorf.
The vibe I took from this was the Paul Graham programming ‘bottom-up’ theme. Paul says in a Lisp language you write a DSL for your problem domain and then solve the business problem in the DSL. I took it this was what Paul deGrandis was driving at. If so – this is quite cool.
Now the great thing about Paul D is that he is now the Pedestal guy. If you’ve worked with Pedestal – you’ll know it embodies this philosophy (particularly in the deprecated versions of the front-end).
Cursive: a different type of IDE Om nom nom nom
[Video] [Slides] [Twitter] [Blog] [Pre-Conj Prep] [Pre-Conj Interview] [Github]
Anna Pawlicka has come from London and working with Bruce Durling at MastodonC. She was demonstrating some of the visualisations you can do with Om. She used d3.js and dimple.js to produce charts quickly.
Anna did enunciate the benefits of Om extremely clearly, stating that Om relieves us from querying the DOM – we can just use clojure data structures and diff using reference equality check – and present from there.
Anna demonstrated using http-kit and sente with core.asyc to update components, and these were coordinated with go blocks on the server and client.
Anna described reference cursors in Om, using them for enabling and disabling transitions, and seeing information in a part of the application state.
Anna also followed up the conference by creating an Om-Cookbook – something we’ve been needing for a while.
Cursive: a different type of IDE
[Video] [Twitter Cursive] [Twitter Colin] [Pre-Conj Prep] [Pre-conj Interview] [Github]
Colin Fleming is a charming New Zealander who has picked up the old La Clojure plugin for IntelliJ and is busy turning it into a commercial-grade product.
Colin touched on a lot of developer-expectations that go with Clojure entering the mainstream. Most mainstream developers do expect context-sensitive, ast-aware tooling. The broader theme was an ‘IDEs vs Editors’.
What was fun about this talk was the ribbing he gave Emacs. (Which Bozhidar very humorously returned later in the day.) The theme of this was mainly around it being a ‘text-only’ tool.
He also made a comment on people’s critique of Intellij, saying they could type faster in emacs. His response to that was ‘lack of typing is a feature not a bug’.
He touched on the infrastructure of Intellij – providing a large index in which you can feed in the program elements and AST. This menas you can discriminate based on fully-qualified symbol – for searches and suggestions. You want to be able to do a ‘find usages’ and do a ‘smart find and rename’ [not text based].
He did some refactorings to demonstrate the tool that got the crowd excited. I saw the people around me kicking back in their chairs and banging their fists with excitement.
Generating Generators [Video] [Slides] [Twitter] [Blog] [Pre-Conj Prep] [Pre-conj Interview] [Github]
Steve Miner gave a great introduction to using test.check in Clojure and how you set up generators to work on it. His big idea was that the libraries around this could be better – and so he has built the Herbert library. This makes generating parameters (eg that match regex filters) easier
One of the great things about this talk was the warm-up on generators for Ashton’s talk on Saturday. Steve mentioned there were plenty of opportunities for performance improvement. One trap to stay away from was trying to co-join to regexes, as this was unlikely to succeed in the current implement. Steve was entertaining, the star-trek theme was fun, and he is just a great guy to talk to.
[Video] [Twitter] [Pre-Conj Prep] [Github]
Ghadi Shayban got an intro from Stuart Sierra, “Ghadi is also a classical pianist… which is what I guess you need these days to use emacs…”. (bah-bom tish). [Laughter and boos from the crowd.] Stuart’s acerbic sense of humour cracks me up. (I think this got edited out of the youtube clip.)
Ghadhi spent a fair bit of time on language internals – including relating development with invokedynamic on Clojure to what had already been done on the Nashorn JavaScript engine for the JVM.
Ghadi also talked about the Graal and Truffle projects. (Graal is a general performance project to improve the performance of the JVM relative to Native Code – basically the hotspot infrastructure has been moved from C++ to Java. Truffle is a language framework on top of Graal that handles the AST). To prove the power of Truffle someone had implemented C in Truffle (TruffleC).
Ghadi finished up by saying that he could see some potential improvements with local vars (and the invokedynamic framework). He was hoping for optimisation with control flow, object manipulation, value manipulation, invocation, stack traversal.
The evolution of the Emacs tooling for Clojure [Video] [Slides] [Twitter] [Blog] [Pre-Conj Prep] [Pre-conj interview] [Github]
Bozhidar Batsov is an effervescent guy all the way from Bulgaria. I got to meet him the day before at the Scheme workshop – it seemed we had both flown roughly the same number of hours to be there (about 20 hours in transit).
He continued his charm on stage, with a very confident, polished and witty delivery. (The photo of him here is terrible). He introduced himself as “a night in the order of emacs”. He then proceeded to gentle return the “editor ribbing” given to him by Colin earlier in the day. (I saw them having breakfast together in the hotel, exchanging ideas the morning after – it was all good fun).
Bozhidar started with a discussion of the history of emacs and Clojure, and all the plugins people had written over the years. He also talked a little about the State of Clojure Survey, saying that Cider had gone from 48% to 42% usage, but was still dominant.
One the most interesting points Bozhidar stated, “In 30 years, no one will have heard of Cursive, but they will still be using Emacs.”
Developing Music Systems on the JVM with Pink and Score[Video] [Slides] [Twitter] [Blog] [Pre-Conj Prep][Pre-Conj Interview] [Github]
Steven Yi is a musician and Phd student. His talk revolved around music theory and its computational implementation. You may be familiar with other Lisp-like music systems like Sam Aaron’s work on Overtone, or Andrew Sorenson’s work on Extempore. As you may be aware, Overtone runs on a system called SuperCollider (which does music generation from the algorithmic instruction from Overtone). I believe (although I could be wrong) that Extempore uses SuperCollider as well.
Steven’s talk was more fundamental than these, rather than focus on algorithmic coordination of music, his focus was on the generation of the music (ie replacing SuperCollider). Steven was implementing a library called Blue which runs on CSound to do this.
Steven made the point that digital audio is a sequence of numbers, where time and space are linked. Steven is a great guy to talk to.
Inside Transducers
[Video] [Twitter] [Pre-Conj Prep] [Github]
Rich Hickey was the reason that most people I spoke to came to the conference. He is an extraordinary speaker who speaks with a casual authority, and who for a lot of people changed the way they think about computing problems.
Rich Hickey started off with a heartfelt thanks to his wife who was in the audience, who had been there for him all the time he had invested in Clojure. I found it special because it touched on the tension that is often felt with hobby projects and family.
Rich has made popular the study of Etymology amongst Clojurians – and this was no exception. He took the time to review the latin origins of the word-components of transducers.
This talk went deeper into stateful transducers. It also felt like Rich alluded to some blog posts by Peter Frankel and Christophe Grande.
Rich mentioned future projects in the transducer pipeline (under consideration): parallel transducers, supportive implementation macros, primitives, multi-arity, kv transduce, functional stateful transducers. He also talked about future work into core.async.
This one is a definite must-watch (but you’ve already figured that out).
Day 2
Helping voters with Pedestal, Datomic, Om and core.async [Video] [Pre-Conj Prep] [Github NH] [Github CS]
Nathan Herzing & Chris Shea opened day 2 with a killer demonstration of live-coding in Clojure with Datomic and Om.
Their talk was about building tools to update voters about their mail-in ballots at Turbovote. There was some stuff in here about integrating with legacy systems that matched my own experiences.
This is worth it alone for the inspiration it will give you to get better at your tooling (and perhaps live-coding) skills.
Persistent Data Structures for Special Occasions
[Video] [Pre-Conj Prep] [Github]
Michał Marczyk was introduced as having been specially selected by Rich Hickey. He dropped into the detail of data structure implementation and talked about the benefits of FlexVec, an improvement on Clojure’s vectors. He then explained in detail why in many cases they perform better.
This was one of the most dense talks in the whole Conj. Good for a brain-stretch.
Applying the paradigms of core.async in ClojureScript
[Video] [Slides] [Twitter] [Blog] [Pre-Conj Prep] [Pre-Conj Interview] [Github]
Julian Gamble is the guy writing this post! I talked about core.async syntax and green threads in Clojure and ClojureScript with some visual demos. I’m just going to include some things other people said.
Eric Normand also had some interesting commentary. (Eric is a friend – and so this is well-taken). I found this feedback really freeing – and will allow myself to level-up.
I was thrilled to be a part of Clojure Conj. I learned so much from this experience, and had a blast.
Variants are Not Unions
[Video] [Twitter] [Blog] [Pre-Conj Prep] [Pre-conj Interview] [Github]
Jeanine Adkisson is this super-intelligent compiler-writer who is fun and opinionated. From the start this talk was delivered with animations in vim – and it stayed information-dense from there.
There was a dinner with a bunch of people after and I had the chance to Jeanine about LR2 parsers and monadic parsers. She is great to talk to.
[Video] [Slides] [Twitter EW] [Twitter LC] [Blog LC] [Pre-Conj Prep] [Github EW][Github LC]
Lucas Cavalcanti & Edward Wible presented on the 4 (really 9) superpowers of datomic. You could see that Datomic has massive benefits in the financial services space.
What was most interesting to me about this talk was Edward’s background (Princeton Computer Science, then MBA, Private Equity, then Boston Consulting). He had moved to San Paulo to be the CTO of a bank that was starting from scratch. He was writing the code by hand with the team. He was there to promote this company and invite people to join him. What incredible vision and execution! (If I didn’t have family I was tempted to take six months to join him) This guy is one to watch.
Making Games at Runtime with Clojure
[Video] [Site] [Pre-Conj Prep] [Github]
Zach Oakes delivered one of the must-see presentations at Conj 2014.
This was the first time I had been to a conference talk that had everyone around me was laughing so hard they were crying.
Zach talked about his love for writing games, playing games, teaching programming through writing games. He took special joy in seeing how people used his library.
I hope I haven’t built up your expectations too much – but this was extremely memorable.
Cló: The Algorithms of TeX in Clojure
[Video] [Twitter] [Blog] [Pre-Conj Prep] [Github]
Glenn Vanderburg is an experienced presenter and it shows. Now who in the Clojure community has ever tried to scratch an itch with a hobby project? Glenn set out to rewrite the Tex editor produced by Donald Knuth in Pascal, into Clojure.
This was the premise for a journey into the tools from 30 years ago, the constraints of the systems of that time and a deep appreciation for how good things are now:
The conclusion of this project is interesting, both in its reflection of the miracles Knuth worked at that time, and the power and constraints of the Clojure ecosystem. Glenn is such good fun to watch.
Day 3
[Video] [Twitter] [Blog] [Pre-Conj Prep] [Github]
Zach Tellman started his talk talking about sierpinski triangles and waxed lyrical from there. (Zach used to maintain a 3d library in Clojure called Penumbra).
Zach touched on the truism from the start of the conference “prefer data over functions over macros”. Zach said it was great the spectrum of options we have, but it wasn’t a perfect model. He said it was similar to when the periodic table came out, that it had some gaps, which indicated perhaps there were some other things to look for.
Zach touched on transducers as a specific sort of composition. His concern was for people new to Clojure, that they would require a lot of foundational explanation.
Zach then talked about regular expressions as a composition mechanism, and how it had lead him to write the automat libary. He talked about the richness of automata theory as a set of tools. He said it was quite hard to accomplish anything.
Zach then talked about queuing with core.async and how backpressure was an emergent property. He said the causality of the problem leads to code complexity, and the only tool we had at the moment to work with this was a macro. (The Go macro). He definitely wasn’t saying it was bad. (Zach is the author of lamina which came out before core.async and has some similarities for queuing without the green threads macro).
Zach’s talk was driving at the catch-phrase “prefer composition over X” and talked about all the approaches and pitfalls that this entails. He said that the problem with “just composing something” is that it is inextricably tied up in what you’re trying to accomplish. He was trying to capture all the different forces at play.
This is quite philosophical, but really worth watching.
Building a Data Pipeline with Clojure and Kafka
[Video] [Twitter] [Blog] [Pre-Conj Prep] [Github]
David Pick delivered a real-world experience using Clojure, Kakfa and redshift. David wanted to keep the data warehouse in sync with his primary database, but PostgreSQL didn’t prove helpful in the approaches he tried. He needed a highly resilient system.
The benefit of Kakfa was atomic operations, persistence and the ability to replay the steam to an external sink.
The original web client had been written in Ruby, but they found that the clients for Kafka were JVM based, the and concurrency operations for when Kafka blocks were useful.
They also needed a specialised way to shut the infrastructure down and step through each part. Actors turned out to be a good fit for this, and kept the same semantics that goroutines held for them. Actors also helped avoid race conditions. These semantics were provided by pulsar/quasar.
The other thing he liked about Kakfa was that the data scientists could integrate into it directly for real-time fraud monitoring. They also use it for web-hooks to send async events, and report generation (some massive reports) which can happen in real-time.
[Video] [Twitter] [Blog] [Pre-Conj Prep] [Github]
Ashton Kemerling explained his ‘real-world’ experiences using generative testing. Building on Steve’s explanation of Generative testing to describe the shape of the data and run multiple cases, he talked about how he would let the machine hammer the possibilities and find the edge-cases. He was using Selenium to hammer a website, which even he was a a bit surprised about.
The main downside was that duration is complicated, you can’t tell how long it was going to take to run. The benefit was that in finding a failure scenario that others described with 10 steps, he could narrow it down to 2 steps. He also said it was good for missed errors.
His technique involved getting a snapshot of database state into memory, and then load it each time the browser is loaded, with a browser cache flush.
This was quite an inspiring talk, and convinced me that test.check is ready for real-world usage.
Stewardship: the Sobering Parts
[Video] [Twitter] [Blog] [Pre-Conj Prep]
Brian Goetz delivers one of the must-watch talks for the conference. You can see that Brian has attended a conference or two in his lifetime and he is a very experienced speaker. Some have said this was a celebration for Brian:
His premise was that he was a language designer, and seeing COBOL jump the shark with the addition of the alter command, he was kept up at night worrying about that happening to Java.
Brian took the time to touch on Clojure cliches like etymology and ponies, and clearly had a fun time doing it.
Brian talked about project Valhalla, which is a set of performance improvements in Java 10 for specialised generics, value types and var handles. Brian also talked about Project Panama, (which may come into Java 9) which is bringing ffi, data layout control and autolayout.
This is a must-watch.
Must-Watch Videos
These are the ones that stuck out in my mind:
(I’d really appreciate it if you watched mine – but totally understand if you don’t have time).
Personal Reflections
Almost none of the big names from the Clojure Community bar Rich were on the stage. This seemed strange at first, until I realised it was about community-building. In speaking to people, they were coming from their local Clojure meetups and seeing the most-polished, most-entertaining, most-insightful presentations from across the country and the world. Most of those presentations (and speakers) had been polished with the involvement of their local group. This was a celebration of all the little Clojure communities coming together.
This year Clojure moved in the ‘adopt’ sphere of the Thoughtworks Radar. Some have commented jokingly on what this really means. I do remember reading that the original Conj was an intimate group of less than 100 people. Now with 500 people, and half of them new to the Clojure Conj, in a sense it felt like the Clojure community has come of age. In some ways I feel a sense of loss, that this edgy small community is changing into something different. On the other hand it’s quite exciting to think what will happen if we can pull ourselves together and solve big problems in software development.
(Did I mention I’m also writing a book called Clojure Recipes?) 🙂
Just a quick note – I’m in the process of publishing a book, Clojure Recipes. You can see it for presale on Amazon here.
This post shows you how to setup Datomic running on Cassandra using using Vagrant and Docker from scratch. This is non-trivial and if everything goes well, will probably take a hour.
Note that this are my notes from going through this process, so there will definitely be room for improvement, either a simpler way to do things or a better way to explain what is going on.
Send licence key and under Downloads – and also get the latest version of Datomic Pro.This has the following steps:
Note that steps A-J are here.
K. Setting Up Datomic
L. Running Datomic
Note that steps 1-24 are here:
25. Obtain a license for Datomic pro through the Datomic Website
26. Download Datomic Pro from here (ie not the free version of Datomic)
27. Expand the datomic download
$ unzip datomic-pro-0.9.4384.zip -d datomic-pro-0.9.4384
cd datomic-pro-0.9.4384
28. Copy the cassandra transactor into the config directory (mac terminal)
$ cp config/samples/cassandra-transactor-template.properties config
29. Drop in your license key for datomic pro from the email into the cassandra-transactor-template.properties file (mac terminal)
$ nano config/cassandra-transactor-template.properties
under
license-key=
Also Set the following property entries
cassandra-host=localhost
Also Ensure the following is uncommented:
cassandra-user=datomic
cassandra-password=datomic
30. Provision a keyspace and table for datomic from the datomic directory (mac terminal):
cd apache-cassandra-2.0.8/
./bin/cqlsh -f ../datomic-pro-0.9.4815/datomic-pro-0.9.4815/bin/cql/cassandra-keyspace.cql -u cassandra -p cassandra
./bin/cqlsh -f ../datomic-pro-0.9.4815/datomic-pro-0.9.4815/bin/cql/cassandra-table.cql -u cassandra -p cassandra
./bin/cqlsh -f ../datomic-pro-0.9.4815/datomic-pro-0.9.4815/bin/cql/cassandra-user.cql -u cassandra -p cassandra
Note that with the third one – if you have trouble – log in with cqlsh and run it directly – ie
./bin/cqlsh -f -u cassandra -p cassandra
CREATE USER datomic WITH PASSWORD ‘datomic’ NOSUPERUSER;
GRANT ALL ON KEYSPACE datomic TO datomic;
31. Start the datomic transactor with the new cassandra template properties we just created:
cd datomic-pro-0.9.4815
bin/transactor config/cassandra-template.properties
Ensure you get a result like:
System started datomic:cass://localhost:9042/datomic.datomic/?ssl=&password=datomic&user=datomic
So you know it was successful.
Note this down – this will be the URI to connect.
32. In a new command prompt in the datomic-pro-0.9.4815 directory run:
bin/shell
Ensure you get the prompt
Datomic Java Shell
33. Run through the following commands from the datomic tutorial. Create the database: http://docs.datomic.com/tutorial.html
% String uri = "datomic:cass://localhost:9042/datomic.datomic/seattle;e;e?user=datomic&password=datomic";
% Peer.createDatabase(uri);
Expect the result:
true
34. Connect to the database
% conn = Peer.connect(uri);
Expect result similar to:
<{:db-id "seattle;e;e-bdffc04f-19b0-4c5b-8ff8-97393d14b303", :index-rev 0, :basis-t 63, :next-t 1000, :unsent-updates-queue 0, :pending-txes 0}>
35. Load up the schema
schema_rdr = new FileReader("samples/seattle/seattle-schema.edn");
schema_tx = Util.readAll(schema_rdr).get(0);
txResult = conn.transact(schema_tx).get();
36. Add some seed data
data_rdr = new FileReader("samples/seattle/seattle-data0.edn");
data_tx = Util.readAll(data_rdr).get(0);
txResult = conn.transact(data_tx).get();
37. Query the database
results = Peer.q("[:find ?c :where [?c :community/name]]", conn.db());
db = conn.db();
for (Object result : results) {
entity = db.entity(result.get(0));
System.out.println(entity.get(":community/name"));
}
You should get a list of communities that looks like this:
Downtown Dispatch
Discover SLU
Discover SLU
Discover SLU
Delridge Produce Cooperative
Delridge Neighborhoods Development Association
Delridge Grassroots Leadership
Crown Hill Neighbors
Community Harvest of Southwest Seattle
...
Voila – you have a Datomic database running on Cassandra across three nodes!
Just a quick note – I’m in the process of publishing a book, Clojure Recipes. You can see it for presale on Amazon here.
This post shows you how to setup a Cassandra Cluster from scratch on MacOS Mavericks using Docker running on Vagrant. This is non-trivial and if everything goes well, will probably take an hour.
Note that this are my notes from going through this process, so there will definitely be room for improvement, either a simpler way to do things or a better way to explain what is going on.
This has the following steps:
A. Install Homebrew
B. Install brew-cask
C. Install Vagrant
D. Set up a Vagrant instance running Ubuntu
E. Install Docker on Vagrant
F. Clone docker-cassandra on Vagrant
G. Customise the docker-cassandra instance
H. Start the Cassandra cluster
I. Test from your mac
J. Installing Cassandra on the mac
1. Run the following command to install Homebrew if you don’t already have it installed:
$ ruby -e "$(curl -fsSL https://raw.github.com/Homebrew/homebrew/go/install)"
2. Run the following commands to install brew-cask
$ brew tap caskroom/cask
$ brew install brew-cask
3. Run the following command to install vagrant:
brew cask install vagrant
4. To select a Ubuntu Image for Vagrant – run the following:
vagrant box add ubuntu-server-14.04 https://oss-binaries.phusionpassenger.com/vagrant/boxes/latest/ubuntu-14.04-amd64-vbox.box
vagrant init ubuntu-server-14.04
5. To configure networking for Vagrant – modify the file in /home/users/ to add at the end (after the End):
Vagrant::Config.run do |config|
config.vm.forward_port 9160, 9160
config.vm.forward_port 9042, 9042
end
(Note that docker-cassandra only supports forwarded ports – it doesn’t support vagrant bridges)
6. Start the Vagrant Ubuntu instance:
vagrant up
7. Test the vagrant instance by connecting:
vagrant ssh
You are now shelled into the vagrant terminal.
8. Run the following to install docker
sudo apt-get update
sudo apt-get install docker.io
sudo ln -sf /usr/bin/docker.io /usr/local/bin/docker
sudo sed -i '$acomplete -F _docker docker' /etc/bash_completion.d/docker.io
9. Test docker works with
sudo docker run -i -t ubuntu /bin/bash
You are now shelled into the docker terminal
10. Exit from docker to vagrant
exit
11. Install git on the vagrant instance (Vagrant terminal)
apt-get install git
12. Install nano on the vagrant instance (Vagrant terminal)
apt-get install nano
13. Clone docker-cassandra:
git clone https://github.com/nicolasff/docker-cassandra.git
14. Modify the docker install scripts to allow external password authentication
nano docker-cassandra/install/bin/install-cassandra
15. Add the lines
sed -i -e 's/authenticator: AllowAllAuthenticator/authenticator: PasswordAuthenticator/g' $CONFIG
sed -i -e 's/authorizer: AllowAllAuthorizer/authorizer: CassandraAuthorizer/g' $CONFIG
16. Create the docker image (vagrant terminal)
make image VERSION=2.0.3
17. Confirm the image has been created with
./list-images.sh
You should get
2.0.3
18. Start 3 nodes on the cluster with: (vagrant terminal)
./start-cluster.sh 2.0.3 3
19. List the nodes running: (vagrant terminal)
sudo docker ps
You should get a result similar to:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
1b8062746694 cassandra:2.0.3 /usr/bin/start-cassa 11 minutes ago Up 11 minutes 9042/tcp, 9160/tcp sleepy_darwin
365c4df8fc65 cassandra:2.0.3 /usr/bin/start-cassa 11 minutes ago Up 11 minutes 9042/tcp, 9160/tcp angry_fermat
c36f05d87f9b cassandra:2.0.3 /usr/bin/start-cassa 11 minutes ago Up 11 minutes 0.0.0.0:9042->9042/tcp, 0.0.0.0:9160->9160/tcp determined_franklin
20. Check the setup of the nodes:
./client.sh 2.0.3 nodetool -h cass1 status
You should get a result similar to:
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 192.168.100.3 125.64 KB 256 33.4% 07d65d03-9e95-402c-8ebc-aad77fc22aef rack1
UN 192.168.100.2 116.96 KB 256 34.3% 1e525775-1393-49af-8e95-c7527a6b0e73 rack1
UN 192.168.100.1 114.91 KB 256 32.2% 5c5f239c-60b0-4960-8a89-21a3aafb13d7 rack1
21. On a mac terminal run:
$ telnet 192.168.1.17 9160
This should give:
Trying 192.168.1.17...
Connected to 192.168.1.17.
Escape character is '^]'.
Then control-C to escape
^CConnection closed by foreign host.
This showed it worked.
22. Ensure you have Java installed by running:
java -version
This should give the result similar to:
java version "1.7.0_40"
Java(TM) SE Runtime Environment (build 1.7.0_40-b40)
Java HotSpot(TM) 64-Bit Server VM (build 24.0-b55, mixed mode)
If you don’t have Java installed – you can get it from here.
23. Download Cassandra from here to your Downloads directory.
24. From a terminal in the apache-cassandra-2.0.8 directory – run
csqlsh -u cassandra -p cassandra
You should get:
Connected to Test Cluster at localhost:9160.
[cqlsh 4.1.1 | Cassandra 2.0.3 | CQL spec 3.1.1 | Thrift protocol 19.38.0]
Use HELP for help.
cqlsh>
Congratulations – you have installed a Cassandra Cluster on your Mac using Vangrant and Docker.
I had the opportunity to run a workshop at LambdaJam Brisbane, and it was a blast. The conference is refreshing because it attracts all the really deep thinkers from the software development community in Australia, and it throws in a couple of big name international speakers as well.
They’ve just released the photos here. I’m sharing a few below.
It was great to meet the guys from the Melbourne and Brisbane functional programming communities, and continue the conversations over dinner.
Hilights from the talks included:
My workshop was “Applying the paradigms of core.async in Clojure and ClojureScript”. The slides and code are available.
To the organisers – Dave Thomas and Craig Smith for organising a really fun conference.
Thanks to Liam McLennan and Craig Smith for the photos.
Just a quick note – I’m in the process of publishing a book, Clojure Recipes. You can see it for presale on Amazon here.
Just a quick note – I’m in the process of publishing a book, Clojure Recipes. You can see it for presale on Amazon here.
This post shows you how to setup a Cassandra Cluster from scratch on MacOS Mavericks using Virtualbox VMs. This is non-trivial and if everything goes well, will probably take a couple of hours.
Note that this are my notes from going through this process, so there will definitely be room for improvement, either a simpler way to do things or a better way to explain what is going on.
This has the following steps:
A. Downloads
B. Installing VirtualBox on a Mac
C. Getting Started with VirtualBox
D. Setting Up Ubuntu
E. Setting Up Java
F. Installing Cassandra
G. Run and Check the Cassandra Server
H. Duplicate the Virtual Machine for the Cluster
I. Setup Cluster Networking
J. Setting Up Cassandra
1. Download VirtualBox – https://www.virtualbox.org/wiki/Downloads
2. Download The VirtualBox Extension Pack (Same URL as above)
3. Download a VirtualBox instance for Ubuntu 12.10
http://virtualboxes.org/images/ubuntu/#ubuntu1304
4. Install VirtualBox (VirtualBox-4.3.6-91406-OSX.dmg)
5. Run VirtualBox to ensure it installed correctly, then exit the application.
6. Double-click the VirtualBox extension pack you downloaded earlier – follow the prompts to install the pack in Virtual Box
7. Expand the Ubuntu VirtualBox instance you downloaded earlier
8. In VirtualBox start a new Machine with Machine -> Add and select the VirtualBox instance you downloaded earlier
9. Start the machine
10. If prompted for the Virtual Media Manager – remove the two mounted .iso virtual drives as you no longer have access
11. When logging in – use the password reverse for the ubuntu login, unless otherwise instructed at step 3
12. Select the large cog in the bottom left hand corner for System Settings. Select Keyboard layout (not Keyboard). Use the plus button to add your keyboard, and the minus button to deselect Italian.
13. From the cog in the top right hand corner – select ‘about this computer’ – then wait for the ‘checking for updates’ button to change to ‘install updates’ – then install the updates. This may require a restart and re-login of the virtual machine
14. From the top left hand corner select Dash Home then Terminal
15. Update the keyboard from Italian to your local setting at the server level using the command
sudo dpkg-reconfigure keyboard-configuration
16. Install the virtual box guest additions on the ubuntu guest virtual instance
sudo apt-get install virtualbox-guest-additions
17. In your web browser, log onto the datomic site at http://my.datomic.com/ – request your keys under Account and download Datomic Pro from the Download tab. Also download the license key.
18. Install Java with the following commands:
sudo add-apt-repository ppa:webupd8team/java
sudo apt-get update
sudo apt-get install oracle-java7-installer
19. Check the installation with
java -version
(Ensure that you get Java 1.7 and not an OpenJDK variant)
20. Put JAVA_HOME into the environment using the following commands:
sudo nano /etc/environment
add the line:
JAVA_HOME=/usr/lib/jvm/java-7-oracle
21. Restart the machine, login and open a new terminal
sudo shutdown -r now
22. run the command
echo $JAVA_HOME
Ensure you get
/usr/lib/jvm/java-7-oracle
as a response.
23. Locate the URL of the latest (post 2.0) install of Cassandra here http://cassandra.apache.org/download/
24. Create a Cassandra directory
sudo mkdir /home/cassandra
25. In a terminal window run
wget http://mirror.ventraip.net.au/apache/cassandra/2.0.3/apache-cassandra-2.0.3-bin.tar.gz
tar -xvzf apache-cassandra-2.0.3-bin.tar.gz
mv apache-casandra-2.0.3 /home/cassandra
26. Set up the folders:
sudo mkdir /var/lib/cassandra
sudo mkdir /var/log/cassandra
sudo chown -R $USER:$GROUP /var/lib/cassandra
sudo chown -R $USER:$GROUP /var/log/cassandra
27. Modify the environment
sudo nano /etc/environment
CASSANDRA_HOME=/home/cassandra/apache-cassandra-2.0.3
#add this to the end of the existing path variable – don’t add a new path entry.
PATH=…:$CASSANDRA_HOME/bin
28. Restart the virtual machine, login and start a terminal
sudo shutdown -r now
29. Check the settings with
echo %CASSANDRA_HOME
This should point to the directory just created.
30. Run
sudo sh /home/cassandra/apache-cassandra-2.0.3/bin/cassandra
31. Start a new terminal and run
sudo sh /home/cassandra/apache-cassandra-2.0.3/bin/cassandra-cli
Ensure that you get Connected to “Test Cluster” in the output
32. Exit the cli by running
exit;
33. In the other window, stop Cassandra by running
CTRL-C
If that doesn’t work – try this.
34. Clear the Cassandra Data
sudo rm -rf /var/lib/cassandra/*
35. Close and save the Virtual machine in VirtualBox
36. Right click on the virtual machine in the VirtualBox Manager and select Clone. Then ensure the checkbox Reinitialize the MAC address is checked. After this check Full Clone. You have now created the second node in the cluster.
37. Repeat this for the third node in the cluster.
38. In Virtualbox select the menu VirtualBox -> Preferences -> Network -> Host Only Networks and select the plus button to create a new host-only network (take the default network name – probably vboxnet0)
39. Edit this setting and take note of of the values for
(a) Adapter-IPv4 Address
(b) DHCP Server – Server Address
(c) DHCP – Server Mask
(d) Lower address bound
(e) upper address bound
40. In the VirtualBox manager screen, control-click the second node and select Settings -> Network -> Adapter 1. Then change the drop down list value to Host Only Adapter. Then ensure that the name matches the network name set before (vboxnet0 for example).
41. In the terminal – run
ifconfig
and take note of the following values
(f) inet addr
(g) bcast
(h) mask
42. Now using terminal and nano – edit the /etc/network/interfaces file
sudo nano /etc/network/interfaces
and add the following:
iface eth0 inet static
address 192.168.56.2
# i.e. below the dhcp range
netmask 255.255.255.0
network 192.168.56.1
broadcast 192.168.56.255
gateway 192.168.56.0
Ensure that you leave the entries for lo loopback
43. Now edit the file
sudo nano /etc/udev/rules.d/70-persistent-net.rules
to comment out all rows but the last one (and its comment) and change the eth# (eg eth1) to eth0 (at the end of the row)
44. Now restart the virtual machine:
sudo shutdown -r now
45. Confirm this has worked by ensuring your IP address you just set up comes up under eth0 when you run
ifconfig
If there is an eth1 entry – something has gone wrong – redo the steps above.
46. Note the following Node Numbers
Node 0: 0
Node 1: 3074457345618258602
Node 2: 6148914691236517205
These were generated from here https://dl.dropboxusercontent.com/u/30184176/digitalocean/tokenCalc/index.html
47. On node 0 – edit the Cassandra settings file
nano /home/cassandra/apache-cassandra-2.0.3/conf/cassandra.yaml
and set the following settings:
cluster_name: ‘VirtualBox Cluster’
initial_token: 0
seed_provider:
- seeds: "192.168.56.2"
listen_address: 192.168.56.2
rpc_address: 0.0.0.0
endpoint_snitch: RackInferringSnitch
48. On node 1 – edit the Cassandra settings file
nano /home/cassandra/apache-cassandra-2.0.3/conf/cassandra.yaml
and set the following settings:
cluster_name: ‘VirtualBox Cluster’
initial_token: 3074457345618258602
seed_provider:
- seeds: "192.168.56.2"
listen_address: 192.168.56.3
rpc_address: 0.0.0.0
endpoint_snitch: RackInferringSnitch
49. On node 2 – edit the Cassandra settings file
nano /home/cassandra/apache-cassandra-2.0.3/conf/cassandra.yaml
and set the following settings:
cluster_name: ‘VirtualBox Cluster’
initial_token: 6148914691236517205
seed_provider:
- seeds: "192.168.56.4"
listen_address: 192.168.56.4
rpc_address: 0.0.0.0
endpoint_snitch: RackInferringSnitch
50. Now start Cassandra on the first node (the seed node) using:
sudo sh /home/cassandra/apache-cassandra-2.0.3/bin/cassandra
(note that this will run as a background process – so if you hit enter after waiting for 30 seconds, you’ll get your command prompt back)
51. Now repeat this on each of the two secondary servers
sudo sh /home/cassandra/apache-cassandra-2.0.3/bin/cassandra
52. Fire up cqlsh and run a test:
/home/cassandra/apache-cassandra-2.0.3/bin/cqlsh -u cassandra -p cassandra
53. Run through the CQL commands here
Just a quick note – I’m in the process of publishing a book, Clojure Recipes. You can see it for presale on Amazon here.
This post shows you how to setup Datomic running on Cassandra using Virtualbox VMs from scratch. This is non-trivial and if everything goes well, will probably take a couple of hours.
Note that this are my notes from going through this process, so there will definitely be room for improvement, either a simpler way to do things or a better way to explain what is going on.
Send licence key and under Downloads – and also get the latest version of Datomic Pro.This has the following steps:
Note that steps A-J are here.
K. Setting Up Datomic
L. Running Datomic
Note that steps 1-53 are here:
54. Expand the datomic download
unzip datomic-pro-0.9.4384.zip -d datomic-pro-0.9.4384
cd datomic-pro-0.9.4384
55. Provision a keyspace and table for datomic from the datomic directory:
/home/cassandra/apache-cassandra-2.0.3/bin/cqlsh -f bin/cql/cassandra-keyspace.cql -u cassandra -p cassandra
56. Copy the cassandra transactor sample into the config directory
cd config/samples
cp cassandra-template.properties ..
cd ..
57. Modify the cassandra template to setup the properties of our local instance.
nano cassandra-template.properties
cd ..
58. Copy the licence key you downloaded (at step 16) into the file under the property entry
licence-key=
59. Set the following property entries
cassandra-host=localhost
60. Start the datomic transactor with the new cassandra template properties we just created:
bin/transactor config/cassandra-template.properties
Ensure you get a result like:
System started datomic:case://localhost:9042/datomic.datomic/<DB-NAME>
Note this down – this will be the URI to connect.
61. In a new command prompt in the datomic-pro directory run:
bin/shell
Ensure you get the prompt
Datomic Java Shell
62. Run through the following commands from the datomic tutorial: http://docs.datomic.com/tutorial.html
String uri = "datomic:cass://localhost:9042/datomic.datomic/seattle";
Peer.createDatabase(uri);
conn = Peer.connect(uri);
63. Load up the schema
schema_rdr = new FileReader("samples/seattle/seattle-schema.dtm");
schema_tx = Util.readAll(schema_rdr).get(0);
txResult = conn.transact(schema_tx).get();
64. Add some seed data
data_rdr = new FileReader("samples/seattle/seattle-data0.dtm");
data_tx = Util.readAll(data_rdr).get(0);
txResult = conn.transact(data_tx).get();
65. Query the database
db = conn.db();
for (Object result : results) {
entity = db.entity(result.get(0));
System.out.println(entity.get(":community/name"));
}
Voila – you have a Datomic database running on Cassandra across three nodes!
My book Clojure Recipes, is to be published for sale on October, 2015.
This is what the book is about:
Developers are discovering the immense power of Clojure’s functional programming model to quickly solve problems in domains ranging from social networking to Big Data. Targeting the Java Virtual Machine, Clojure also leverages the Java platform’s maturity and enormous ecosystem. Clojure Recipes is a “code recipe book” for this increasingly popular language.
Julian Gamble focuses on practical and complete examples that illuminate Clojure’s key features and show step-by-step how to solve real-world problems with it. Clojure Recipes provides a series of “learn by doing” step-by-step projects, you’ll learn how to:
Here is a little about me:
Julian Gamble (Sydney, Australia) is a software engineer who has worked in the financial services industry for more than a decade. When he’s not enabling billions of dollars to orbit the globe, he writes and presents on all things software related.
FAQ
What about the cover?
We haven’t got that done yet. We’ll update it as soon as we know
Isn’t there another book with a similar title?
We think it’s great that the Clojure community has grown beyond ‘Introduction to Clojure’ books. We’re both seeking to grow the Clojure community. They have a fantastic team working on it and we wish them all the best.
When did you start with this?
I signed a Contract last December with Pearson publishing. Since then I’ve had my head down getting it ready.