Author: I wanted to consolidate some of the steps I did to get the basic functionality for Common Lisp web applications (CRUD, User Authentication, Domains, etc.) as its a big pain to keep revisiting and troubleshooting which step I did wrong.
So I saved these notes in guide form, hopefully its useful for others. The trick is to extend a lot of this, all the hard parts are here - the rest is just basic Lisp, HTML, JavaScript, building upon this platform.
Highly recommend Lisp for Web Development, its *much, much better* than any of the other frameworks out there! So easy to use. Do learn HTML, CSS & JavaScript though. Esp. JavaScript as you would want to offload some of the computation to client-side to reduce your server costs.
Big Thanks to u/linnilbobo on Reddit for his code on user authentication in Hunchentoot - that was one of the missing pieces for me.
There are many views on this (so please don’t take my word as gospel), but from my experience, Lisp gives the following benefits:
- A fantastic and easy to use open source web server in Hunchentoot
- Very easy integration into databases like PostgreSQL. Just looking at my guide shows how easy all this is (assuming you know lisp, but if you look at the number of lines of code - it’s not that much)
- Live debugging - I.e. I can recompile certain functions and don’t need to restart my web server! That’s a huge win in my opinion
- It’s hard to explain, but I find Lisp very transparent. With knowledge of JavaScript, HTML, CSS and Lisp, I am very comfortable building any web app. Whereas when I use various web frameworks like react, angular, etc, I have to spend so much learning it’s idiosyncrasies (which to me is a waste of time since these frameworks keep changing), and then usually I hit a brick wall somewhere. Other frameworks in my opinion obfuscate too much - they solve for easy wins (which they do), but anything more complex and you are fighting against the language, spending hours on Stack Overflow for solutions to what should be simple problems. In my opinion
I spent 1-2 weeks setting up a CRUD app with Angular and still don’t know the best way to use it. With Lisp, I did the same in one day. It was that experience that pushed to me to learning and using Lisp (I was a complete beginner at the time). For me that was enough proof
- Stuff like NPM I don’t like because there are so many dependencies (which break from time to time) and I’m worried somebody could be putting in bugs / trackers in these packages. I don’t like having many dependencies
- Ability to write DSLs (Domain Specific Language) where I can easily write macros to generate JavaScript and HTML for me from Lisp, increasing the cohesiveness of the whole program
All that said, Lisp by itself is not enough. HTML/CSS/JavaScript are much more important - Lisp is the magic server side plug for all of this, and in my opinion, better than server side JavaScript because Lisp is a very well thought and well featured language (JavaScript is as well, but I think Lisp, at least to me, is slightly more enjoyable to write in).
I find the issue with Lisp is that it’s not used much, so outside of some very good packages, there are many parts which are easier to do in other languages / frameworks with much bigger ecosystems. For me, web development is the exception, because all you need is HTML/CSS/JS and a good server side language.
Other languages are also very good, so this is just a personal opinion. If you like to invest your time in one core language, and not have to keep learning new web frameworks, I recommend Lisp.
It’s very good for solo programmers or small teams, but I do think it’s not suitable in larger teams as I do find lisp code can be a bit harder to read and also most people don’t use lisp.
So in my uneducated view - if you have spare time, if you like working from first principles and have a scientific / mathematical mindset, and if you are working alone or in small teams, I highly recommend Lisp for web dev.
If you are more business minded and want to scale your project quickly and hire more developers, then using an off the shelf package makes more sense.
If you are time poor, stick to what you know - the differences aren’t THAT great, and it’s better to build something now than to spend most of your time learning a new language (unless you have time to devote to it - in which I highly recommend learning Lisp :) It really makes me smile every time I code in Lisp :)
This is hand-wavy and bordering on woo, also not specific to web development.
In my experience, Lisp has this sort of brain amplification power. Whatever weird idea I have about how to think about a problem, I can kinda smush lisp into shape so it solves that problem. with language X, I have to think about how to I say what I'm thinking in X. It's kinda like spending some time working in C then jumping to python. It's far less hassle to get the idea out there. It's not exactly like that, but it has that flavor.
Two sort of related points.
Lisp is pretty transparent. If you need to think about pointers and stuff, you can do that and it'll mostly stay out of your way. But if you don't, you don't.
Lisp will let you represent whatever crazy notion you have in your head. You may understand why the code works, but explaining to other people can be tough, because your internal abstractions are probably not what other people intuitively think about when looking at the same problem.
Not saying any of the above is good or bad. It is different. With java or go or whatever, I tend to think a lot about how do I represent this in java or go or whatever. With lisp it's the closest I've ever gotten tot my idea out there, naked. That can be powerful, or scary, or both.
You might take a look at "Paradigms of AI Programming: Case Studies in Common Lisp". That shows a lot of tricky problems tackled in a pretty breezy way. Want to build a computer algebra system? Not that tough to get something working. Work through Norvig's approach and you can get something pretty good working pretty quick.
Nice points. Going through SICP and solving problems using scheme, I did experience the “brain amplification” sensation.
Once you flesh out the idea in lisp, it’s just a matter of implementing it in the language of your platform. It’s like doing ‘architecture’.. Lisp is about aesthetics and beauty
One major Lisp thing I've been musing about lately...
C programs don't understand other C programs. Of course the C compiler itself is an exception but in general you don't expect to be able to feed your compiled C .exe another C source file and do anything useful with it. That's crazy talk. Likewise with Java or C++ or Fortran or...
But Lisp programs understand Lisp programs. The compiler is always available even at runtime. And not just the compiler but the lexical analyzer, the parser, and the evaluator too. Output is also easy: You can write a Lisp program that builds Lisp data structures dynamically in memory and then prints them to a text file as Lisp source code. Which you can later read back in and execute as a program.
This kind of thing is so natural in Lisp that it opens up whole new ways to think about code. So why is this useful?
One example: In many programs you need configuration files. So you write configuration files for C programs in XML or yaml or JSON and then use a library function to parse them. Why don't we write configuration files for C programs in C syntax? Well, because C syntax is rather inconvenient for this purpose and because there aren't many C libraries that parse C syntax. Besides, how do you handle includes, compiled libraries, macros, linking, etc? Again, crazy talk.
Not so in Lisp. Lisp syntax is very convenient for configuration files (because it's s-expressions) and the parser is built-in. So you write configuration files in Lisp and parse them at runtime! If you change a setting at runtime, it takes not much more than one line of code to automatically print a new configuration file. Which you can then tweak with a text editor if you so desire.
Another example: Macros. Macros use this same principle but there's no external file involved. Macros change Lisp's syntax so you can write more advanced programming constructs and translate them to basic Lisp at compile time. It's the same idea as Typescript-- Typescript is a Javascript program that converts a type-safe version of Javascript into basic Javascript. But Typescript was a one-off major project--like building a compiler. In Lisp the toolkit for source-to-source translation is built-in and Lisp programmers use it every day.
The issue with my code is that somebody can put in arbitrary SQL queries within the data field (I think - not sure if it is 100% possible because their data would be enclosed within the programs SQL query, which may break their attempts)
Best way around it is to put in checks to prevent the code from being anything nefarious. I’ll have a think and figure out what’s the best way of doing this, but I assume not constructing the sql strings from scratch but rather using postmodern’s API (postmodern is Common Lisp’s excellent interface to PostgreSQL) makes sense, or the alternate solution recommended in the other reply to your post.
Does anyone here use Common Lisp for their day job? I'd like to try it out some time (and am currently working through Paradigms of AI Programming by Norvig) but it feels like the language is so bendable that it'd be a minefield for a large team.
I use it in my day job and have done so on and off for a little over thirty years. I've used it in team settings, as well as solo. It's a large force multipler solo and on small teams, but it can also be productive for larger teams.
You want at least one experienced Lisper who can mentor the less experienced and build guard rails where needed. The good news is that it's unusually easy to build guard rails with Lisp because of that same "bendable" quality you mentioned.
Another thing that is unusually easy with Lisp is building code-analysis tools. The source code is made of standard Lisp data structures that you can operate on with standard Lisp functions, and you can do it all interaactively in the live environment. That makes it much easier to build linters and other analysis tools than it is in most other languages.
You need someone on the team who is experienced enough to know how to do these things, and team-oriented enough to want to do it.
You also need to be aware that you probably don't know what guard rails and analysis tools you need when you're starting out. You'll discover them along the way. So you need project management with enough foresight to invest in prevention up front in order to save on cures down the road.
Out of curiosity, how does hiring go at these teams? I'm really leaning towards CL for my next product but the thought of having to hunt down CL devs to hire is giving me pause. This is a boodstrapped product, so not able to afford a bunch of devs and need to scale up fast, just need to be able to find 2 or 3 CL devs.
I don't think I can generalize usefully about hiring. I can generally find Lispers to help me, but that's at least partly because I've been one for a long time and as a consequence I have a handful of contacts I can turn to who will either sign up or help me find someone.
(If you want me to see if I can help, send me mail at mikel@evins.net and we can talk it over.)
In most of the cases on my list of past projects, the people who organized the projects were Lispers themselves or had contacts in the Lisp community, and recruiting from that community was part of the early project setup.
The bad news is that the community of Lispers is small compared to those of more popular languages. The good news is that Lispers are generally eager to work in Lisp, which can be helpful in recruiting. For example, I prioritize Lisp opportunities over others, and I'll work for less money if I like the people and I get to work in Lisp.
I have been part of hiring and mentoring some programmers that were not experienced with Lisp. In all but one case, they were interested in learning it, and made quick progress.
I've worked on a few teams of mixed Lispers and non-Lispers, ranging in size from about three to about sixty. I've found that you want one or more experienced Lispers who are willing and able to take on a mentorship role. With mentors, good programmers can pick up Lisp and its style and idioms pretty quickly (assuming they want to). Without mentors it can be rough going.
Wow - that’s awesome! We would all be really grateful for a blog of your experiences, if you have time one day. Which field do you use it in? I have a stereotype that it’s mostly used in scientific / quantitative fields
I've used it for a long time, so forgive me if this reply gets long.
I'm not the only or primary author of all of these; only a few of them.
- Panels [Apple, circa 1988]: an experimental graphical UI framework with a built-in constraint solver and extensible event-handling system
- GATE [Apple, circa 1989]: a knowledge-based, distributed, automated testing system used to discover and report thousands of application-compatibility bugs in Mac system software before release
- bauhaus [Apple, circa 1991-1994]: an experimental Lisp OS for Newton. The Newton OS (except the microkernel) was written in Dylan back when it was basically Scheme+CLOS; the Dylan compiler and development tools were written in Common Lisp.
- SK8 [Apple, circa 1995-1997]: rapid-application-development system sometimes referred to as "HyperCard on Steroids".
- Reactivity Application Firewall [Reactivity, circa 1998-2004; later Cisco]: Reactivity's product was a complex application-in-a-box that provided rule-based application-level security and integrity for networks. Cisco bought our company and folded it into their product line. I used Common Lisp for several code-analysis and documentation tools used in development of the product.
- Clozure Common Lisp [Clozure Associates, circa 2006-present]: I wrote documentation and code for Clozure Common Lisp and some associated tools
- Secure Outcomes LS1100 [Secure Outcomes, circa 2010]: I wrote the bulk of the control system and GUI for the Secure Outcomes fingerprint livescanner product, using Lispworks.
- Delectus [my own product, circa 2010-present]: Delectus 1, a simple personal data manager, was written mostly in Scheme. Delectus 2, a more capable, flexible, and portable version, is still under development and is written entirely in Common Lisp.
- Confidential project [Confidential client; 2020-present]: a knowledge-based, reasoning-based neurosymbolic machine-control system. As I mentioned, it's confidential, but I can say that it's designed to monitor and orchestrate a variety of different kinds of robots with human supervision in real time.
I've omitted several free libraries and other open-source Lisp software that I've written over the years.
I don't find SBCL or Lispworks superior overall. Each one is better than the other in specific, narrow ways. For example, you can't beat SBCL for availability of compatible open-source libraries. On the other hand, you can't beat Lispworks for a portable GUI framework.
I use both.
As for tips, there are probably a lot, but I have to take a break from HN right now to attend a work meeting. :-)
SBCL is better in the sense that almost everyone who writes an open-source library for Quicklisp distribution is going to test it with SBCL, so most of the time Quicklisp libraries just work with SBCL.
Testing with other Lisps, including Lispworks, is not quite so automatic, so there's a little greater likelihood that something might fail to work.
That being said, most libraries I've tried with Lispworks have worked without any trouble.
That's good to know that the familiar ql libraries will be available if / when venturing into using Lispworks, potential compatibility issues aside ...
Thanks. I forgot a few things, like Alpaca, a programmable, Emacs-like WYSIWYG word processor I wrote with CCL in around 2002, and AllegroGraph--I did some work on it in around 2010.
I also left out a bunch of library projects that live on Github.
I used to work at an AI firm that used it. Large Lisp codebases are a minefield with near impossible security and debugging issues. I still love it as a programming language tho.
I would love to hear more on the security challenges and debugging issues, if you have time. It’s not used as much in industry, so any insights are really very valuable.
Also so cool that you worked in AI! That’s my dream field :)
I saw a fun debugging issue the other day on r/lisp that almost made me want to walk away from lisp for good.
You have a function
(defun my-val () "a string")
Referentially transparent, easy to reason about right? Wrong. Lisp strings are mutable, and everything is a reference. So if you do something like this
(replace (my-val) "b" :start1 0 :end1 1)
(my-val) => "b string"
I appreciate that CL doesn't force everything to be immutable, but it seems like at least strings would benefit from being treated as immutable values.
Destructive modification of literal objects is one of those things you're not supposed to do because the consequences are undefined [0]. Some CL implementations might let you get away with it; others might not. I agree that it would be better if immutability was required for literals.
Nice link. I'd be happy if an implementation just turned that into a compiler error. I can see use cases for reaching in and mutating like that, but also seems like dynamic scope is a better solution for when you want that kind of action at a distance.
Interesting example. Yes, it’s not a perfect language by any means :-) Some things are better in Lisp, some things are better in other languages as well for sure!
There are a number of large companies that use CL quite effectively for safety critial purposes. Can you please give more details about your experience?
siscog in portugal is one example but there are others, historical and current. it is also used in quantum computing and financial markets. gramarly and google flights are perhaps the most famous current users of the software.
edit: i adjusted my answer because i thought i was answering to parent
Thanks! I’ll have a look. I find Certbot a bit fragile to use, but not sure if that’s just the problem space (since you have to verify domain control before issuing SSL certificates). Perhaps this will be a more robust approach
This looks interesting. I started learning Scheme but felt that CL has some interesting pieces and this book seems quite inviting. I wish I could order a paperback for this book
You are most welcome :) Yes, this hopefully is a complete guide from start to finish and without any gaps (which are not easy for us to solve without having a reasonable amount of background knowledge!)
Would you be able to expand on this? I’m very basic when it comes to debugging in Lisp :)
So you are running a second server to test issues away from production? I thought this is possible in other languages / frameworks, so I must be missing something
I read the cook book but still didn’t get the main benefits of it. However the quote was very cool:
That’s how Ron Garret debugged the Deep Space 1 spacecraft from the earth in 1999:
We were able to debug and fix a race condition that had not shown up during ground testing. (Debugging a program running on a $100M piece of hardware that is 100 million miles away is an interesting experience. Having a read-eval-print loop running on the spacecraft proved invaluable in finding and fixing the problem.
You have your normal application, accessible on some port serving HTTP requests (or whatever). Then you start a thread (with `bt`, bordeaux-threads, a very common multi-threading library) and run a Swank server listening on some other port on it. Swank is the backend used by SLIME, it allows manipulation of the Lisp environment. A remote server is still the same Lisp environment as when you run it on your machine, so you can use Emacs to connect to it (M-x slime-connect) and debug live.
Swank doesn't allow remote connections though, otherwise anyone could connect and mess with your server. You must connect through the loopback interface. Easiest way is to set up an ssh tunnel, like: ssh abc@example.com -L 4200:localhost:4200
Whether you can call the Swank server a second server or not is a somewhat philosophical question I guess. There's one application serving different kinds of requests on different ports.
Does that mean the swank thread still gets all the inputs of the live process and you can debug in isolation there and then push changes to the main thread? Sorry if I’m stupid (no need to reply), I need to read into this more.
You don't need to "push changes to the main thread", you can mutate the state of the whole process during debugging, and in Lisp that includes compiling and loading new code that gets used immediately by whatever application you're trying to debug.
It's probably worth mentioning that you usually don't want to make changes in a live production environment, and even when you do, you probably want to capture those changes in a file as well, so that you have a record of what you did. It's possible to make arbitrary changes to a live, running system in Lisp with no other record of what changes you made, but it's not usually a good idea to do that.
> So you are running a second server to test issues away from production? I thought this is possible in other languages / frameworks, so I must be missing something
The Swank connection runs on a different thread, not on a different server.
Ah that might be able to help bridge the gap for me. I still don’t quite follow, but I guess having multiple threads allows for easier testing of the interface with other components, perhaps? Sorry I am out of my depth on this one. Will Google more tonight
Argh! Fixed now - sorry about that. Pressed the wrong button when publishing on Medium (ps I really like Medium, sometimes it gets a bit of a rap, but it’s very easy to write and they are quite user friendly in my opinion at least :)
To be picky, Medium may be writer-friendly but never was really reader-friendly. The page are heavy with banners blocking the view, no possibilities to know in advance which article is paywalled or not. I do not have a problem with authors publishing on Medium, just Medium itself.
I agree with you, it’s not perfect for readers and until I caved and signed up as a member, I also was annoyed at hitting my monthly limit of articles.
They have amazing SEO so inevitably many articles I need while researching a solution led me to Medium.com
It’s just a very hard market to serve (long form reading / journalism etc). That’s why I’m not that bullish on substack because it’s hard to make it good for readers and profitable for writers / the startup.
Medium paid me $10, and I have about 10,000 reads in the last year. I didn’t do it for the money, but that’s an indication for those who do want to write for a side income - it’s a hard business.
For writing, its very good - to the extent, I use it as one of my main word processors lol. Then just copy it to something else for formatting. Very distraction free and the font is nice to make you want to write more.
It would be great if a new competitor did what Medium did, but made it better for readers. IMO the business model should be like Wordpress but for writers, together with an overall platform with high quality SEO. Make the money selling the software and a bit with memberships on the main site.
I feel for Medium and it’s writers - there’s so little money in it, that to some extent they have to resort to click bait titles, bad UI for readers, and high amount of not amazing quality content.
So I saved these notes in guide form, hopefully its useful for others. The trick is to extend a lot of this, all the hard parts are here - the rest is just basic Lisp, HTML, JavaScript, building upon this platform.
Highly recommend Lisp for Web Development, its *much, much better* than any of the other frameworks out there! So easy to use. Do learn HTML, CSS & JavaScript though. Esp. JavaScript as you would want to offload some of the computation to client-side to reduce your server costs.
Big Thanks to u/linnilbobo on Reddit for his code on user authentication in Hunchentoot - that was one of the missing pieces for me.