For a long time, Pinboard ran off of two dedicated servers. Each had 4GB of memory, a copy of the database, and all the code needed to run the site. The servers lived in separate datacenters, so that in case of a major outage at least one would still be available. The databases kept up with each other in a master-master configuration.

We spread the workload around like this:

Server 1: website, search engine, tag clouds, api

Server 2: RSS feeds, import from outside services, per-user stats, bookmark-by-mail

In addition, both servers ran a crawler that would download and store bookmarks for users who had signed up for the archival service.

This setup worked well for many months, but had two big drawbacks.

The first was a lack of redundancy. When one of the machines went down, the site had to run entirely off the remaining server. If something went wrong with the remaining database during this interval, we risked losing some of the bookmarks created during the outage. While the risk was small, it went against our biggest priority for the site - never losing anyone's data.

The second drawback was that the main website had to share hardware with a bunch of other services. The web server was never free to spread its wings and soar. Instead, it had to contend with whatever else was going on on the machine. For example, once each day the search indexer would wrestle with mysql over memory while it did a full index rebuild. While we tried to time this kind of bickering for slack hours late at night, the situation was like having too many roommates in too small an apartment.

While our median page load times always stayed low, there were some far outliers - cases where a page could take multiple seconds to load. Since our second priority after data integrity was to be the fastest bookmarking website around, I found these dips in peformance deeply frustrating.

Last week the heavens opened and we were finally able to find a high-memory dedicated server that was within our budget - a 16GB machine that rents for about $200/month, which is a rarity in the memory-stingy world of dedicated boxes. Peter and I have just completed moving the main website to this roomy new machine, where the database fits entirely in RAM, and the results have been remarkable. In our stats, we're seeing median page service times that have improved by a factor of ten or more (and the site was not slow to begin with). I invite you to click around and see if things feel faster to you as well.

Having a third server also means having a third copy of the database, which means if one of the machines goes offline we don't lose redundancy. While I hope we never have to benefit from this feature, it's very good to have it.

Coming from a background of working on big websites, where you can just throw servers at any problem, it's been a fun challenge to run a site that has to get the maximum value out of its infrastructure, an experience a little bit like trying to live out of a suitcase. A third server opens up vast new possibilities for features, particularly when it comes to expanding our API. In the meantime, though, I hope you enjoy the peppier service, and please tell me if you spot anything we messed up during the migration. ]]> maciej@ceglowski.com (Maciej Ceglowski) Sat, 07 Aug 2010 00:34:19 +0000 http://pinboard.in/blog fd8e71dbd0de0387cead8ae4b6d436467b4d4c9b We had about thirty minutes of downtime yesterday afternoon when our main server became unreachable. I have not received any satisfying explanation as to what happened. Our hosting provider (alchemy.net) says they found our server powered off, but could not provide any further details.

Curiously, not all our users saw the site go down at the same time. I got two reports that the site was down while it was still responding normally for me, with the usual activity showing up in the site logs. Within an hour of those first reports, the site had disappeared for everyone. At that point I repointed our domain name at the backup server and started opening tickets.

The partial failure, the powered-down server, and some of the network errors I saw while trying to diagnose the problem make me suspect a problem in the datacenter that hosts our machine. I will continue trying to gather information on the outage to see if there's anything we can do to prevent a repeat.

Meanwhile, we seem to be back up and both servers are back to normal. I apologize for the downtime and any inconvenience we caused our users. ]]> maciej@ceglowski.com (Maciej Ceglowski) Tue, 20 Jul 2010 20:32:09 +0000 http://pinboard.in/blog ce742689b51093a04b38ef1d2e275ba90208e6f4

A year ago today this site left the comfortable womb of beta testing and started charging for new accounts.

Here are some of our vital signs, one year in:

• 3.5 million bookmarks
• 11.2 million tags
• 2.5 million urls
• 187 GB of archived content
• 99.91% uptime (6 hours offline)

And here are some of the features we've added to the site in our first twelve months:

• bookmark archiving (paid feature)
• API
• view bookmarks filtered by source
• WordPress plugin
• plain-text notes
• auto sync with instapaper, twitter, google reader, read it later, delicious
• twitter archive
• post bookmarks by email
• downloadable bookmarks for offline viewing
• mobile site
• click history page
• bulk editing widget
• tag clouds and widgets
• a quick 'organize' interface for mass tagging
• bookmarks for geographical locations

Over the past year, we've had about six hours of cumulative downtime, when the website was unreachable. Both these episodes were due to hardware failure. This year, we'll work to make the site a little more resilient against hardware trouble.

We also have a long list of bugs to squash and new features to roll out in an effort to make sure Pinboard remains the best value for your bookmarking dollar. I've updated the site roadmap to reflect our development priorities.

If you don't see a feature you want, or if you have specific questions about our plans for the site, please don't hesitate to ask on the Google group.

A site like this is a real pleasure to work on. I would like to offer a slice of virtual birthday cake to all our users, and thank everyone who submitted bugs, gave us feature suggestions, and took a chance on us in our first year. (Please, limit one slice per customer.)

I also owe a big birthday thank you to honorary co-founder Peter, who has helped me keep the site running while keeping a sharp eye out for customer requests. When you want the quality of customer care that only two Eastern European introverts can provide, choose Pinboard.in. ]]> maciej@ceglowski.com (Maciej Ceglowski) Thu, 08 Jul 2010 15:07:23 +0000 http://pinboard.in/blog 5997c6b96d1dba6d454d93a93923bac048a8754d You can now download sets of bookmarks in a format suitable for offline viewing. Our hope is this will be a handy feature when you need to hop on a train, plane, or just want to read somewhere where there is no wifi available.

To try out the feature, go to your settings page and turn on 'enable downloads'. You should now see two new links at the top of your homepage:

When you click the "download" link, Pinboard will prepare a download with local copies of the 25 most recent bookmarks on that page. You'll be shown a list of the bookmarks in the download along with a status box.

When the download is ready (typically within a minute), the status box will change to a download link:

The downloaded file will open into a folder containing a 'files' directory and a web page called "manifest.htm". Open this file in your browser and you'll see the list of downloaded bookmarks:

We include a clickable version of the original URL under each bookmark in case you want to visit the original site.

Since the download link only grabs the most recent 25 bookmarks from any page, a good way to have better control over what you download is to use the 'starred' page as a kind of staging area for your downloads.

You can see a list of all your available downloads by clicking the 'downloads' link on your user page. You can queue up as many downloads as you want, but there are limits on how much you can download in any given week.

Regular Pinboard users are limited to 50 MB / week. Archival users can download 500 MB. We'll revisit these caps once we see how heavily the feature is used.

As the feature matures, I'll be adding 'download' links to the search and popular pages as well.

There's no way right now to download links for offline viewing to your iPhone or iPad, but this is something we're working on.

As ever, your comments, suggestions and questions are welcome on the Google group. Please send bug reports directly to Peter or me at support@pinboard.in. ]]> maciej@ceglowski.com (Maciej Ceglowski) Thu, 10 Jun 2010 12:34:19 +0000 http://pinboard.in/blog 08fc883aab9c4690ecc2a645fe4df3f06ebf37c8 I've added an experimental feature today that makes it possible to bookmark a geographic location. Placemarks work just like bookmarks, except that in addition to a title, description and tags they include a Google map:

You can turn on this feature on your settings page. When you click 'add place', you'll be taken to a map form that will default to your current location if you are using a geo-enabled browser like Firefox or Safari on the iPhone.

The URL format of a placemark is 'http://pinboard.in/place/u:username/geo:xxxxx/zoom:yy/, where 'xxxx' is the geohash of the location, and yy is the zoom level (as determined by Google maps).

For the moment, I have two use cases in mind for placemarks:

1. You are somewhere with a geo-enabled mobile device and want to remember that location.
2. You want to save a Google map view with a description and tags.

There are some obvious additional features in support of placemarking (KML, map view of all user placemarks, map search) that I will hold back on building for a few weeks. I would first like to see whether the feature is useful, and how people will end up using it.

I will be releasing two additional changes this week - the first is the ability to email a "pinned" location to Pinboard from the iPhone maps app and have it parsed into a placemark.

The second is special behavior when bookmarking a Google maps page (the popup will give you the option of saving the URL as a placemark).

Please let me know your thoughts, suggestions and ideas on the Google discussion group! ]]> maciej@ceglowski.com (Maciej Ceglowski) Thu, 27 May 2010 18:07:13 +0000 http://pinboard.in/blog 825aa0956c281f8838a697a86d3987191638dc33 I've enabled the https:// protocol site-wide for anyone who wants more security/privacy when using Pinboard. The main effect of using the site in secure mode will be that outgoing clicks won't show that you came from a Pinboard page (in other words, the referer field will be empty). ]]> maciej@ceglowski.com (Maciej Ceglowski) Tue, 25 May 2010 10:45:34 +0000 http://pinboard.in/blog 6af16fdd176f063f696d516a0b2b28587965bd86 A number of users have asked for individual control over whether items from outside sources (like Instapaper or Twitter) get added as unread or regular bookmarks.

So I've updated the settings page so that you can configure this behavior for each outside service however you like. Enjoy and please keep the great suggestions coming!

Only you can see your click history. I added the feature because I sometimes find myself visiting the same set of links repeatedly, and the page makes for a convenient shortcut. It's also useful for those times when I can remember clicking an interesting link on the recent page, but don't remember enough about it to find it via search.

As ever, your feedback is most welcome on Twitter, IRC, or the discussion group.

The symptoms of the outage were identical to what happened last week. The main filesystem on our web server suddenly went into read-only mode, crashing the database. Once again I moved all services to the backup machine while the main server went through its long disk check.

In order to prevent further downtime, I've switched the main and secondary servers so that if the problem recurs, it will not disrupt the main site again.

The rest of this post will be fairly technical, in hopes that one of our users might recognize the problem and be able to offer suggestions.

The server in question is a dedicated ServerBeach machine running Ubuntu 8.04.3. It has two SATA drives, one of which is used for backup. Both have SMART diagnostics which show no errors. There are no clues as to the problem in the logs, presumably because the machine is logging to the same drive that goes into read-only mode.

I've set up syslog to write a copy of the log files to the second drive, in the hopes of capturing log output if the problem recurs. Interestingly, both times the error took place around the same time of day, shortly after 13:00 GMT. A look at cron.d and the various cron jobs doesn't show anything scheduled for that time that might be to blame.

Our three working theories are 1) bad sectors on disk, 2) a hardware problem on the SATA bus, 3) a kernel bug in the distro. We'd welcome any leads, especially from people who have experienced similar issues.

To all our users, we sincerely apologize for the outage and will be glad to answer any questions about the measures we're taking to avoid future downtime.