Problems with Web Application Development Today
I want to separate "web app" development from general "web" development, simply because most websites aren't single-page applications, or web apps in the sense we think of gmail, or hey.com. It's put most concisely in "How the Web is Really Built". Here, though, I'll be talking about that subset of the web which can be categorized more as an application than simply a web site.
Building a web app is always going to be a complex endeavour. SPA view libraries like React have emerged in the past five or so years as an awesome way to handle that complexity on the client side, allowing developers to build interactive user-interfaces in a composable way, re-using components easily in different parts of the site.
Duplication of Concerns
Generally, when building a React app, you get any dynamic data through a loosely coupled JSON API (in my personal preference, probably written with Django REST Framework). If you're using a web framework like Django or Ruby on Rails, then you're going to have a lot of complexity and business logic on both the frontend and backend. Whenever you add a feature, you need to ask yourself where it should be added. The loose coupling between parts that may very well be written in different languages means that it's really difficult to share code, and even if you're doing something as simple as validating a form, you'll probably end up with a decent amount of duplicated business logic, a big no-no if you follow the DRY principle!
Data Fetching Boilerplate
Loosely-coupled APIs present another challenge when building web apps: the boilerplate that comes along with getting the data out of your database and displayed in a browser. The general process for an SPA follows this outline:
[User] Navigates to a page, submits a form, or clicks on a resource.
[Browser] Send a request, along with any route parameters and arguments, to a backend API endpoint.
[Server] API endpoint is called, fetches data from the database using the parameters in the request.
[Server] After retrieving data from the database, marshal that data into JSON to be sent back to the client.
[Browser] Receive JSON from request, un-marshal the JSON into data structures that the web app can understand. For a React app, this would include passing the data as props to each component that uses the data to for render itself.
And that's only the happy path! In the case of a validation error, for example, the server would return a 400 response with some JSON details, and the client again would need to catch that error, and render it properly. It really is a lot of boilerplate for any given request.
There are frontend libraries like SWR which factor out as much boilerplate as possible and encourage declarative patterns for your own data fetching, but "thick client" web apps will always need to marshal data to JSON on the server, and un-marshal to some view-able format on the client.
Managing SPA Complexity in 2020
There's a bunch of different solutions to these problems for those who still want to provide an SPA experience, and they fall broadly into two separate categories.
Frontend-First: The JAMStack
In recent years, there's been a big push towards the JAMStack: pushing all your complexity and business logic onto your frontend, and relying mostly on commodified API services for your backend work. JSON is still the lingua franca here, but reliance on pre-built backends as a service for different functionalities, along with query languages like GraphQL, means that there isn't any backend boilerplate handled by developers. All the complexity and business logic is pushed to the client-side, which alleviates worries about duplication of concerns.
JAMStack and associated "micro-backends" like Auth0, or "backends-as-a-service" like Supabase and Google Firebase allow people who haven't done too much backend work in the past to build truly full-stack apps on their own. m3o is even building a constellation of JAMStack-oriented "micro-services" to provide the batteries to power most web apps. Hmm… "batteries included"? Where have we heard that before…
Backend-First: Re-discovering the batteries
MVC Frameworks like Django and Rails came about during the heyday of Web 2.0 to abstract away a lot of the boilerplate associated with building CRUD web applications, the exact issues that we spoke about above. Back in 2006, all those interactions were through normal browser HTTP requests. SPAs, built with anything from Backbone or Ember to React and Vue, were more responsive. These web frameworks became frameworks for JSON API servers, and for many web app developers, functionality like Django's templates and forms and the battle-tested abstractions for linking them together became vestiges of an earlier age. Django's
Form classes can render validation errors in templates with virtually no boilerplate written by developers. As soon as you want to put that form action over a JSON API, any responses from your server, which were previously just the HTML that the browser displayed, now have to be un-marshalled from JSON on the client and handled specifically. How much was Django really a "batteries included" framework if you needed to pull in REST Framework and OAuth Toolkit whenever you wanted to work with a "modern" frontend?
Many people, myself included, enjoy modeling business logic in the ways Django and its ilk allow for. Backend-first fullstack frameworks have begun to proliferate built on top of these existing frameworks. Phoenix LiveView and Laravel Livewire are two that come to mind immediately, and have been around for a year or more.
These fullstack frameworks go a long way towards solving both of the concerns with traditional SPAs listed above. Separation of concerns is no longer an issue, since there is no separate, loosely coupled frontend codebase. Data fetching is drastically simpler in this paradigm. No longer does all your data need to be serialized to JSON before being converted into HTML; your data-fetching flow looks a lot more like this:
[User] Navigates to a page, submits a form, etc.
[Server] Backend route is called, fetches/stores appropriate data from the database based on the request.
[Server] Data is used to populate an HTML template, which is sent to the client and rendered with the help of the framework.
Your backend business logic renders HTML directly, completely replacing steps 3, 4 and 5 above with a single step: map your data into its visual representation in HTML. The two server steps remaining are the actual business logic in your application: the full-stack framework handles the smooth transitions, without the developer having to worry about serializing and de-serializing their own data. The logical flow of your application becomes a lot simpler for a single developer to follow and to handle.
Even React's Server Components fit this new paradigm: Data fetching no longer happens in AJAX requests, but by declaring a child server component which fetches the data and displays it in its own DOM tree without the developer having to serialize to/from JSON themselves. After the component renders on the backend, its virtual DOM gets sent to the frontend by React itself for display. The developer's interface into this whole process remains high-level and declarative.
As the Pendulum Swings
We started the decade with frameworks like Meteor.js with extremely tight couplings between the client and server, and after a long time wandering in the wilderness of duplicated compelxity across loosely-coupled frontend and backend, it seems like we're entering the twenties with a renewed push towards a more monolithic approach to web development. When even a frontend framework like React is beginning to bridge the gap with the backend, you know it's an interesting idea to explore right now.
So what is Hotwire?
At Hotwire's core is Turbo, a new library that takes HTML from AJAX requests and dynamically modifies the currrent page. It comes out of an existing library called Turbolinks, now called "Turbo Drive" as of today, which is a utility that intercepts all click events on anchor tags, loads the resources over AJAX, and swaps out the
<body> tags, all while handling browser history.
Turbo Frames, one of the new components, is pretty intriguing. Turbo Drive will still AJAX-ify form submissions and link clicks behind the scenes, but instead of swapping out the entire webpage each time, Turbo will look for matching
<turbo-frame> tags on the current page and in the new page's content. If there's a match, it'll dynamically replace that section of the page. Basically, you can compose webpages together, using
<turbo-frame> to delineate template partials as scoped components, similarly to how you'd think of components in a React app. The benefit here being that all the logic is handled on the server-side rather than split between two code-bases.
Trying out Hotwire with Django
What's special about Turbo when compared to Phoenix LiveView and Laravel Livewire is that Turbo is completely backend-agnostic: Drop the JS bundle into a
<script> tag in your page's
<head>, and Turbo Drive works its magic without any co-operation from the server. Turbo Frames can be adapted by wrapping
<turbo-frame> tags around template partials in any backend framework. Turbo Streams, the solution for incremental data updates, can also be used in the context of HTTP requests without any co-operation from the server beyond modifying your template partials. It's only if you want to use Turbo Streams over WebSockets where you'll need some custom code for your specific backend framework.
Since the push behind Hotwire came from DHH and Basecamp, it makes sense that their examples are with Ruby on Rails, and that's where they've made their supporting libraries. I decided to take a shot at building a demo app similar to what's shown in Hotwire's demo video using Django rather than Rails. It really didn't take long! I got my start in Web Development with a JQuery app with a REST API, and even after moving on to Django, I always used Django REST Framework. I never really took advantage of the templating functionality, or the super-useful built-in CRUD operations with Django forms. It was an interesting experience working with
DetailView rather than
ModelViewSet, and I'll be excited to keep exploring this going forward.
After an hour or so more of experimentation and digging into the turbo-rails codebase, I got a working prototype of a Turbo Streams
Broadcastable mixin for Django! I'm working on something similar for Django REST Framework right now, which definitely helped in hitting the ground running. I'll probably look to clean up the code and make sure it works for the other actions, and split it out into its own pypi package.
I'll have to look at how the Rails integration handles authorization – right now, anyone would be able to subscribe to any stream for a given model, which is obviously not ideal for actual production applications.
I was really surprised at how easy it was to set up Turbo to work effectively with a Django backend. The chat app that I built was really simple, but it also was just not many lines of code: not having to worry about moving data around from the frontend to the backend really decreased the amount of time spent on the implementation. Turbo though is in a pretty early beta, and the one main thing I'd like to see be addressed would be a good fallback mechanism for Turbo Streams over websockets. Right now, if you want to broadcast updates over websockets, then you can't also send Turbo Streams in HTTP responses to form actions without getting duplicate data appended. The solution in the Hotwire demo video is simply to not send updates over HTTP, and only stream over websockets. This doesn't seem particularly robust, however, in the case that a websocket connection fails or a client simply doesn't support it. In addition to the five actions, there should probably be an
append-or-replace action that looks for an element with a matching
id, performs a
replace action if one is found, and otherwise performs an
append action. The duplicate updates from the HTTP response and the websocket stream wouldn't conflict in that case, since one will append, and the other will replace with identical data.
This is definitely an exciting time for frontend development! I'm hoping to do some more experimentation in the coming weeks, and I'm glad that framework authors accross the board are putting effort into thinking about how to move the web app developer experience to the next level.