Since its initial release, Kamal (formerly mrsk) has emerged as the long-awaited facelift to Capistrano, while also making the us of Docker seem like a walk in the park. At its heart, Kamal taps into Docker, steering us away from the oftentimes bewildering saga of Ruby on Rails deployment. It transitions us from wrangling a mishmash of infrastructures—spanning development, testing, staging, to production across varied machines and operating systems with their own quirks in versions and package management—to the blissfully straightforward narrative of deploying with a single, neatly packaged container image.
At first glance, tools like Kamal are akin to a silver bullet for the solo developer, allowing for a breezy journey from code to production. This is crucial because it means you can focus on what truly matters: crafting solutions to business challenges with your code.
Post-deployment, the ‘state’ saga begins, ranging from the health of the operating system and network to the dynamics of application data.
However, what often goes unmentioned, similar to discussions about cloud computing: it’s all still running on someone else’s hardware. Beneath the surface, there lurks the ever-present need for handling ‘state, state, state’. And state is what comes after the deploy: starting from the state of the operating system, the network state, all the way up to the state of the application data.
So, after you deploy and then redeploy, it’s time to ponder over your next moves:
- How do I maintain and update the operating system and essential services?
- How can I monitor the current state of the operating system, services, and my application?
- What strategies should I use for managing data storage, whether in short-term storage like Redis or long-term solutions like PostgreSQL and file systems?
- How do I manage state stored in third-party applications, and apply the above considerations to them?
Server Options
When choosing a server, you would probably go for a virtual unmanaged instance from popular provider. They’re budget-friendly, and even the smaller offerings can comfortably support an app, a worker, a Redis instance, and a PostgreSQL instance. At the time of writing, Hetzner’s CX11, at 4.55 Euros, will fit the bill nicely. For a tad more, at 5.22 Euros, you get a boost with an extra thread and double the storage. Over at Digital Ocean, a comparable setup will cost you about 18 USD per month. AWS, with its notoriously cryptic pricing, offers something similar at a ballpark cost. Linode’s in the mix too, with options that might set you back somewhere between 12 to 24 USD.
What do you mean, unmanaged?
Mind you, all of these units are unmanaged, which means all responsibility lies with you as the contract partner to the hosting service. A server left to its own devices can quickly turn into a digital wasteland. Public-facing services are like open invitations for anyone with mischief in mind. Once compromised, it’s essential hot garbage. Best to scrap the server and start fresh.
Even if it’s “just a side project” or “not a top priority,” vigilance is non-negotiable. So, here are some high-level tips on how I keep things under control.
Tackling State
To keep the operating system and essential services in tip-top shape, I use Ansible playbooks, categorizing them by setup, service roles, and specific configurations for my setups — often CapRover on Docker Swarm or vanilla Ubuntu Server.
I organize tasks around the server’s life cycle—think “system upgrades,” “Docker cleanup,” “backup preparations,” “user and SSH key updates,” and so on.
For business processes, programmatic runbooks provide a blend of manual steps, scripts, and API requests (i.e. dns providers, git management, etc.), allowing for scalable and efficient project management. Their greatest strength lies in blending various layers of complexity: Beginning with a straightforward, step-by-step to-do list, you can incrementally incorporate external command calls or integrate comprehensive scripts to streamline and automate processes.
Gaining Insights
Gaining insight into the performance of your operating system, services, and applications is a crucial maintenance activity. There’s no one-size-fits-all solution; it’s all about the scope of your project and your personal preferences.
For a comprehensive overview at various levels, starting with an application monitoring tool like Honeybadger is wise for tracking errors, uptime, and job performance. It’s an excellent first step. As you become more acquainted with your application, developing custom metrics endpoints for specific features can be a beneficial next move.
Diving deeper, the management of services and the operating system necessitates centralized, indexed logging capable of generating actionable metrics. For a solo host, beginning with journald to capture all system logs is a practical choice. Given the diversity of the Linux logging landscape, setting this up can be complex. Selecting software or services that offer more than basic stdout or logfile outputs is crucial, though integrating such logs with journald can add complexity. The ability to live-tail journald, viewing all log outputs in real-time, greatly aids in understanding how different services interact on a server. For environments beyond a single host, integrating tools like Promtail, Loki, and Grafana can address journald’s limitations, especially for developing alerting rules based on incident experiences to improve oversight.
Monitoring the actual health of your host is also vital. Hosting providers may not alert you to issues like maxed-out threads, choked I/O, or full disks. Tools like Netdata as a starter, and the trifecta of Node Exporter, Prometheus, and Alertmanager later on are invaluable for these deeper diagnostics. When selecting tools or third-party services, consider the type of data access they provide and the flexibility for configuring custom metrics and alerts.
Backup Strategies
Many hosting providers offer daily snapshots of your system as part of their service, often costing you 10-20% of your monthly server fees. Generally, this seems adequate until a snapshot coincides with a crucial database transaction or file write, potentially leading to data inconsistencies or corruption. The process for creating snapshots isn’t always transparent, and the common advice to shut down your server for a manual snapshot highlights its limitations.
My approach prioritizes data over the operating system’s state, focusing on database content and important application files — like media or client products — which are irreplaceable and costly to recreate. For everything else, tools like Ansible can rebuild the system from scratch.
Restic in conjunction with Autorestic is my preferred backup solution. It supports a wide range of storage backends and secures backups with encryption, offering differential backups and the option for full backups at specific intervals. The downside is its recovery process, which requires navigating through Restic, but the trade-off for secure, manageable backups is worth it. Adhering to the 3-2-1 backup strategy — three total backups on two different media types, with one stored offsite — provides a robust safety net. Thus, combining provider snapshots with Restic backups across various platforms, like B2 Backblaze and Hetzner storage boxes, ensures comprehensive coverage. While daily backups are standard, timing them to capture the day’s significant work can further safeguard your data.
Conclusion
Managing an application’s lifecycle is a complex task that can’t be fully covered in just one article. I plan to dive deeper into the topics discussed here, sharing more of my experiences and insights. This will allow me to explore these concepts in a more practical, detailed manner, moving beyond theoretical discussions to real-world applications.
Therefore, feel free to leverage Kamal for streamlining your deployment process, but remember, deployment is just one phase in the broader journey of an application’s life.
References
Kamal
Hosting Providers
