7603
Twitter
Discovery is the biggest thing missing from Mastodon (apart from you the people). I don't need Japanese tweets post…
from twitter_favs
yesterday
Twitter
PM Lee: There is one fundamental reason why HDB leases are for 99 yrs - we need to be fair to future generations. I…
from twitter_favs
yesterday
Twitter
PM Lee: Singapore’s housing policies have been uniquely successful – we are the only major city in the world where…
from twitter_favs
yesterday
Twitter
PM Lee: Many Singaporeans are also concerned about housing – but house prices affect people differently. When you w…
from twitter_favs
yesterday
Twitter
PM Lee: These are some of the major changes over the last few yrs so that all Singaporeans have access to affordabl…
from twitter_favs
yesterday
RFC 7946 - The GeoJSON Format
I really like the GeoJSON RFC - succinct, readable, clear and has good examples
from twitter_favs
2 days ago
Twitter
We checked the four comparison sites promotes and none were this up to date, 2 did offer VDSL2 'fibre' servi…
from twitter_favs
2 days ago
Twitter
I thought exam grading was done on a percentile basis, ie top X% get A and so on. So why is it news that more than…
from twitter_favs
2 days ago
Twitter
The incredible (Remy Sharp) is joining Vue.js London as an Advice Lounge Expert.

You can come to talk to him…
from twitter_favs
3 days ago
Twitter
Pioneer batch of 9 Smart Nation Scholarship awardees unveiled today - we're attracting a strong pool of engineering…
from twitter_favs
3 days ago
Twitter
Save £40 on the Befree Advanced in Blue. Now Only £134.95 - Online exclusive, while sto…
from twitter_favs
6 days ago
Twitter
“A good morning window looks out on some kind of constant object or growing thing, which reflects the changes of se…
from twitter_favs
7 days ago
Twitter
So takes a giant R56bn step closer to a full-blown sovereign credit ratings downgrade to junk statu…
from twitter_favs
8 days ago
Twitter
Another one down. At least 20 more before I can properly seal my Lambdas away inside a VPC with no internet access
from twitter_favs
9 days ago
Twitter
When plastic degrades, it releases gasses that contribute to global warming. Not only doesn't plastic go away, but…
from twitter_favs
9 days ago
Twitter
Also, mailing lists and IRC channels.
from twitter_favs
10 days ago
Home Volume Control: Go+VueJS = R&R – ITNEXT
🎚 Using Go and Vue.js to Create a Volume Control App for Home Devices:
from twitter_favs
10 days ago
Twitter
📹👨‍💻 Just finished recording a 1 hour live coding session:

🛠️Moving an existing React.js app to Gatsby.js

…can'…
from twitter_favs
12 days ago
Twitter
Doesn't look like it's supported out of the box: but th…
from twitter_favs
12 days ago
Twitter
This appears to be for *downloads*. Is there a fetch streams equivalent for uploads?…
from twitter_favs
13 days ago
Twitter
If there was a video of me shooting a firearm in a public space I would be in prison right now awaiting my bail hea…
from twitter_favs
13 days ago
Twitter
Built 👀 "Your Application is being Viewed" notifications for

Trying to solve an old problem:

1.…
from twitter_favs
13 days ago
Twitter
Chrome 69 has AV1 video decoding.

Behind a flag, more information here:

(AV1 is an open o…
from twitter_favs
14 days ago
Audio/Video Updates in Chrome 69  |  Web  |  Google Developers
Chrome 69 has AV1 video decoding.

Behind a flag, more information here:

(AV1 is an open o…
from twitter_favs
14 days ago
Twitter
Wow, whoever built this app really need to share/open up the data!
from twitter_favs
14 days ago
Twitter
Hey Insta, long time no see! I'm alive, I promise!
Had some nice pictures taken in Port Meadow today, and figured…
from twitter_favs
14 days ago
Twitter
hosting a podcast or live stream on the go? this seems insanely useful: SC6-L Mobile Interview Kit,…
from twitter_favs
14 days ago
Twitter
On other news: I do now offer ICO services. I have served two happy customers already. It’s simple: The more you pa…
from twitter_favs
16 days ago
Twitter
Yeah, I don't know how to do a repeating footer. :( There's the margin boxes which will let you do basic…
from twitter_favs
16 days ago
Twitter
Check out - I don’t often recommend partner products, but they are awesome at no…
from twitter_favs
17 days ago
In Pursuit of Production Minimalism — Brandur Leach
In Pursuit of Production Minimalism
While working at Lockheed during the cold war, Kelly Johnson was reported to have coined KISS (“keep it simple, stupid”); a principle that suggests glibly that systems should be designed to be as simple as possible.
While complexity is never a conscious design goal of any project, it arises inherently as new features are pursued or new components are introduced. KISS encourages designers to actively counteract this force by making simplicity an objective in itself, and thus produce products that are more maintainable, more reliable, and more flexible. In the case of jet fighters, that might mean a plane that can be repaired in the field with few tools and under the stressful conditions of combat.
During his tenure, Lockheed’s Skunk Works would produce planes like the U-2 and SR-71; so notable for their engineering excellence that they’ve left a legacy that we reflect on even today.
The famous SR-71, one of the flag ships of Lockheed's Skunk Works. Very fast even if not particularly simple.
Many of us pursue work in the engineering field because we’re intellectually curious. Technology is cool, and new technology is even better. We want to be using what everyone’s talking about.
Our news sources, meetups, conferences, and even conversations bias towards shiny new tech that’s either under active development or being energetically promoted. Older components that sit quietly and do their job well disappear into the background.
Over time, technologies are added, but are rarely removed. Left unchecked, production stacks that have been around long enough become sprawling patchworks combining everything under the sun. This effect is dangerous:
More parts means more cognitive complexity. If a system becomes too difficult to understand then the risk of bugs or operational mishaps increases as developers make changes without understanding all the intertwined concerns.
Nothing operates flawlessly once it hits production. Every component in the stack is a candidate for failure, and with sufficient scale, something will be failing all the time.
With more technologies engineers will tend to be come jacks of all trades, but masters of none. If a particularly nefarious problem comes along, it may be harder to diagnose and repair because there are few specialists around who are able to dig deeply.
Even knowing this, the instinct to expand our tools is hard to suppress. Oftentimes persuasion is a core competency of our jobs, and we can use that same power to convince ourselves and our peers that it’s critical to get new technologies into our stack right now. That Go-based HA key/value store will take our uptime and fault resilience to new highs. That real-time event stream will enable immutable ledger that will become foundational keystone for the entire platform. That sexy new container orchestration system that will take ease of deployment and scaling to new levels. In many cases, a step back and a moment of dispassionate thought would reveal that their use could be withheld until a time when they’re known to be well vetted, and it’s well understood how they’ll fit into the current architecture (and what they’ll replace).
In his book Nine Chains to the Moon (published 1938), inventor R. Buckminster Fuller described the idea of ephemeralization:
Do more and more with less and less until eventually you can do everything with nothing.
It suggests improving increasing productive output by continually improving the efficiency of a system even while keeping input the same. I project this onto technology to mean building a stack that scales to more users and more activity while the people and infrastructure supporting it stay fixed. This is accomplished by building systems that are more robust, more automatic, and less prone to problems because the tendency to grow in complexity that’s inherent to them has been understood, harnessed, and reversed.
For a long time we had a very big and very aspirational goal of ephemeralization at Heroku. The normal app platform that we all know was referred to as “user space” while the internal infrastructure that supported it was called “kernel space”. We want to break up the kernel in the kernel and move it piece by piece to run inside the user space that it supported, in effect rebuilding Heroku so that it itself ran on Heroku. In the ultimate manifestation of ephemeralization, the kernel would diminish in size until it vanished completely. The specialized components that it contained would be retired, and we’d be left a single perfectly uniform stack.
Realistic? Probably not. Useful? Yes. Even falling short of an incredibly ambitious goal tends to leave you somewhere good.
Here are a few examples of minimalism and ephemeralization in practice from Heroku’s history:
The core database that tracked all apps, users, releases, configuration, etc. used to be its own special snowflake hosted on a custom-built AWS instance. It was eventually folded into Heroku Postgres, and became just one more node to be managed along with every other customer DB.
Entire products were retired where possible. For example, the ssl:ip add-on (providing SSL/TLS terminate for an app), which used to be provisioned and run on its own dedicated servers, was end-of-lifed completely when a better (and cheaper) option for terminating SSL was available through Amazon. With SNI support now widespread, ssl:endpoint will eventually follow suit.
All non-ephemeral data was moved out of Redis so that the only data store handling persistent data for internal apps was Postgres. This had the added advantage of stacks being able to tolerate a downed Redis and stay online.
After a misguided foray into production polyglotism, the last component written in Scala was retired. Fewer programming languages in use meant that the entire system became easier to operate, and by more engineers.
The component that handled Heroku orgs was originally run as its own microservice. It eventually became obvious that there had been a time when our microservice expansion had been a little overzealous, so to simplify operation, we folded a few services back into the hub.
To recognize the effort that went into tearing down or replacing old technology, we created a ritual where we symbolically fed dead components to a flame called a burn party. The time and energy spent on some of these projects would in some cases be as great, or even greater, as it would for shipping a new product.
At Heroku, we'd hold regular "burn parties" to recognize the effort that went into deprecating old products and technology.
Practicing minimalism in production is mostly about recognizing that the problem exists. After achieving that, mitigations are straightforward:
Retire old technology. Is something new being introduced? Look for opportunities to retire older technology that’s roughly equivalent. If you’re about to put Kafka in, maybe you can get away with retiring Rabbit or NSQ.
Build common service conventions. Standardize on one database, one language/runtime, one job queue, one web server, one reverse proxy, etc. If not one, then standardize on as few as possible.
Favor simplicity and reduce moving parts. Try to keep the total number of things in a system small so that it stays easy to understand and easy to operate. In some cases this will be a compromise because a technology that’s slight less suited to a job may have to be re-used even if there’s a new one that would technically be a better fit.
Don’t use new technology the day, or even the year, that it’s initially released. Save yourself time and energy by letting others vet it, find bugs, and do the work to stabilize it. Avoid it permanently if it doesn’t pick up a significant community that will help support it well into the future.
Avoid custom technology. Software that you write is software that you have to maintain. Forever. Don’t succumb to NIH when there’s a well supported public solution that fits just as well (or even almost as well).
Use services. Software that you install is software that you have to operate. From the moment it’s activated, someone will be taking regular time out of their schedule to perform maintenance, troubleshoot problems, and install upgrades. Don’t succumb to NHH (not hosted here) when there’s a public service available that will do the job better.
It’s not that new technology should never be introduced, but it should be done with rational defensiveness, and with a critical eye in how it’ll fit into an evolving (and hopefully ever-improving) architecture.
Antoine de Saint Exupéry, a French poet and pioneering aviator, had this to say on the subject:
It seems that perfection is reached not when there is nothing left to add, but when there is nothing left to take away.
Nothing left to add. Nothing left to take away.
Most of us can benefit from architecture that’s a little simpler, a little more conservative, and a little more directed. Only by concertedly building a minimal stack that’s stable and nearly perfectly operable can we maximize our ability to push forward with new products and ideas.
In Pursuit of Production Minimalism was published on May 10, 2017 from San Francisco.
Find me on Twitter at @brandur.
Did I make a mistake? Please consider sending a pull request.
Photographs by Ben Harrington (SR-71), Robyn Jay (embers of a burning fire), and Md. Al Amin (boat and sky). Licensed under Creative Commons BY-NC-ND 2.0, BY-SA 2.0, and CC BY 2.0 respectively.
Sent  from  my  iPhone  from iphone
17 days ago
Twitter
I wanted to express my gratitude for your contribution to YouTube and the very interesting and eclectic…
from twitter_favs
17 days ago
Twitter
So technically I’m the wrong recipient, Paynow can’t allow undo the transaction for the sender, original recipient…
from twitter_favs
18 days ago
Twitter
Hmm login monitoring is probably easy to do with CloudTrail. Amazon Macie () will…
from twitter_favs
18 days ago
Marta
I really dig Marta for macOS! Blazing fast!
from twitter_favs
18 days ago
Twitter
I really dig Marta for macOS! Blazing fast!
from twitter_favs
18 days ago
Twitter
It explains AWS to humans.
from twitter_favs
18 days ago
Britons spend average of 24 hours a week online, Ofcom says | Media | The Guardian
Fifth of Britons feel stressed if they can't access internet – Ofcom report
from twitter_favs
18 days ago
Twitter
There’s a PWA by that uses AR to help you find the direction of qiblah (direction of prayer for Muslims)
from twitter_favs
19 days ago
Twitter
🦑 This is so awesome. Spin up a self-hosted e-commerce site. Front end all React, back end mostly handled by Stripe…
from twitter_favs
19 days ago
Twitter
After its completion in 2026, the 21.5km NSC will be SG’s 1st integrated transport corridor with bus lanes and cycl…
from twitter_favs
20 days ago
Twitter
Also catch me presenting on getting started with GraphQL in the cloud using AWS AppSync. =-)
from twitter_favs
20 days ago
Twitter
Optimising my new side-project for 📱 mobile phones:
from twitter_favs
21 days ago
HTML to PDF Conversion with Headless Chrome using Go
HTML to PDF Conversion with Headless Chrome using Go
Thanks to Matthew Molnar for developing this service.
Andrew Zamler-CarhartMay 1
The Compass platform has several products which generate content that we want to export from HTML to PDF. For example, our Marketing Center product lets users design print-ready marketing materials such as flyers, brochures, postcards and notecards. They are designed in HTML / JavaScript / CSS, and need to be converted to a PDF that we can send to our print fulfillment vendor.
Motivation
Previously, we were using a SaaS service called DocRaptor for this purpose. We decided to build our own internal service for several reasons:
Rendering Fidelity: DocRaptor uses the PrinceXML library, which renders HTML somewhat differently than a browser and doesn’t support the latest HTML + CSS standards. This was by far the main reason — we wanted to render the PDF as close as possible to what the user was seeing.
Performance: by keeping requests inside our own cloud infrastructure, we could avoid sending several megabytes of data across the open internet.
Cost: we could avoid paying DocRaptor’s monthly fee.
Security: keeping requests inside our own cloud also has the advantage of avoiding sending our data to third parties, as well as exposing our data in flight.
Headless Chrome Server Architecture
At a high level, we created an Export Service to abstract HTML to PDF conversion. It is written in Go to control Chrome in headless mode. Chrome 59 and above has a new headless mode that lets you run the app programmatically, and has a feature that prints to PDF. It works like Chrome because it is the Chrome app, just without the user interface.
We use a Go library that wraps the Chrome DevTools Protocol for controlling the browser, and a tool called pm2 for making sure that Chrome stays running on the server. Our service has an endpoint that takes the URL of a web page, tells Chrome to navigate to that page, waits for the page to load (this was the hardest part!), prints the page to PDF, and returns the PDF in binary form.
Controlling Headless Chrome using Go
You can launch Headless Chrome from the command line, but how can you communicate with it while it is running? Enter the Chrome DevTools Protocol (CDP). It allows external programs to communicate with a running instance of Chrome to inspect, profile and control it. Headless Chrome supports tabs just like regular Chrome; CDP offers complete control over opening and closing tabs.
Even though the official library for controlling headless chrome is puppeteer, written for node.js, Go is one of our first-class server languages, so we decided to implement the backend service in Go. We needed a library to use CDP. We decided to use the github.com/mafredri/cdp library, which provides a convenient Go wrapper for all CDP functions, such as navigating to a page and printing to PDF.
Our service will communicate with Chrome on the remote debugging port. It’s simply a matter of starting the Chrome executable with the --headless flag, and specifying the remote debugging port with --remote-debugging-port=9222.
Export Service Workflow
Here’s a sequence diagram that shows the complete workflow for creating a PDF, including how the Export service and Chrome interact with other components:
The client makes a request to the frontend that it would like a PDF of the current page.
The frontend makes a request to the Export service.
The Export service tells Chrome to load the page from the frontend and create a PDF. That’s the focus of the rest of this article!
Chrome returns the PDF data to the Export service.
The Export service saves the PDF data to S3, which returns the URL where the file has been saved.
The Export service returns returns the PDF URL to the frontend.
The client downloads the PDF at the given URL from S3.
Code Walkthrough
The process for creating a PDF works like this:
Connect to Chrome
Open a new tab
Connect to the tab
Defer closing the tab
Load the page
Wait for the response to complete
Print the PDF
Here is a simplified version of the function for creating a PDF. For brevity, errors are not handled in these examples (but you should handle them in your code).
Authentication & Working With Multiple Tabs
The export server needs to load a page on behalf of an existing logged in user, a page that normally should not be available to anonymous users. So we needed a way to transfer the current user’s session to the export service. Normally we use a session token stored in the cookies to identify the current user. In this case, we pass the session token as part of the request to the export service, and set the cookie when making a request to the frontend.
cookieArgs := network.NewSetCookieArgs(cookieName, cookieValue).
SetDomain(urlParsed.Host)
_, _ = c.Network.SetCookie(ctx, cookieArgs)
Once we got that working, we ran into another issue when testing the new export service for concurrency. We realized that when we have multiple requests coming to the server on behalf of different users, Chrome was sharing the session information between the tabs. This is the expected behavior for web users, since when opening a new tab on the same website, you want to still be logged in. However, for serving requests on behalf of different users, this was a blocking issue. Rather than opening a tab to the desired page, the solution was to create a new blank tab, then set the cookies, then navigate to the page.
Page Size
Chrome assumes that the page size is 8.5 x 11 inches, and does not automatically pull the page size from the page’s print styles. If our document has a different size, the dimensions need to be passed to the server as part of the API request, and then passed to Chrome.
printToPDFArgs := page.NewPrintToPDFArgs().
SetPaperWidth(width).
SetPaperHeight(height)
Waiting for Page Resources to Finish Loading
One of the biggest challenges we needed to resolve was how to definitively know that a page has been fully loaded. The app is written in Angular 1.5, and we need to account for the Angular loading lifecycle. The browser loads some images, fonts and other resources asynchronously, so the page may not be ready for printing even after Chrome reports that it is done loading the page. For example, we have some Angular components that pull in their own images, and those get loaded by JavaScript after the initial page has been reported as loaded.
We solved this by modifying our web application code. Components that need extra time for loading declare themselves by adding an attribute:
this.$element[0].setAttribute('loadable-component', '');
When a component is done loading, it fires an event:
this.$scope.$emit('LoadableComponentReady');
The page that contains the components finds the loadable components, waits for them to finish loading, and then fires an event when they are all done loading:
When the Go service runs, it injects a small script into the client page which returns a promise that resolves when image loading is complete.
Asynchronous Execution
All requests to our frontend in production are routed through a reverse proxy that has a request timeout of 30 seconds. When processing very large web pages, it may take longer than that to create a PDF, which could lead to request timeout errors.
We have a lightweight digital asset manager that stores URLs of image resources. When it makes a request to the export server to create a PDF, it can run synchronously or asynchronously. Normally, the request hangs until the PDF has been created and saved, and the PDF URL is returned in the response. When background mode is turned on, it creates a resource without a URL and returns immediately with a 201 Created status.
The client can then poll the resource once per second until the URL is available, even if it takes longer than 30 seconds. In this example, the createAsset() function creates the PDF resource, and then the getAsset() function polls for updates until the URL is available. We still set a timeout of 120 seconds to avoid infinite recursion.
Deployment Concerns
Keeping Chrome running
For performance reasons, it’s not practical to start an instance of Chrome every time we want to print a page. For the same reason that you only run one copy of Chrome on your computer, we can keep one copy of Chrome running on the server.
We chose the pm2 process manager to launch Chrome and keep it running in the background. For example, to launch Chrome in headless mode on macOS using pm2, you could run:
pm2 start /Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome \
--headless \
--remote-debugging-port=9222 \
--disable-gpu \
--disable-translate \
--disable-extensions \
--disable-background-networking \
--safebrowsing-disable-auto-update \
--disable-sync \
--disable-default-apps \
--hide-scrollbars
--metrics-recording-only \
--mute-audio \
--no-first-run
The most important argument is of course --headless. You’ll notice that many features like extensions and translation are turned off because we don’t need them. If you’re curious about the other options, here’s an explanation of all of Chrome’s command line switches.
Service Security
In theory, the client could specify the URL of any page on the internet. A hacker could even specify the URL of a local file such as file://etc/passwd and then “print” it to a PDF. Yikes! For security, the service will only load URLs matching a specific pattern.
Handling SSL errors
Our website uses HTTPS for security, but the servers we use for internal testing do not have valid SSL certificates. When loading a page with an invalid certificate, the browser normally displays a warning message that can be bypassed. However, it is not possible to click past this message with Headless Chrome. Instead, we can tell Chrome to ignore these errors.
Local Developer Experience
Running the Server Locally for Development
With DocRaptor, we were sending an HTML string to their service, while Chrome expects to receive the URL of a web page. You can think of this as the … [more]
Sent  from  my  iPhone  from iphone
21 days ago
New OneMap API Docs
OneMap seems to be the "canonical" SG map:
from twitter_favs
21 days ago
Twitter
Version 22 is almost ready! Apologies for the delay, it's certainly long overdue.

I am also planning on posting s…
from twitter_favs
22 days ago
Twitter
“I am not certain about the ranking of SA in as far as crime is concerned... There are countries were children go t…
from twitter_favs
22 days ago
Twitter
. copy support in Firefox on Linux has been broken for the last 8 months (and you know it). Thanks for th…
from twitter_favs
22 days ago
Twitter
And I'm pretty sure everyone in this country agrees on this. The fact that they h…
from twitter_favs
23 days ago
Twitter
Badgen generate badges on the fly, it's stateless! Deploy your own Badgen service to Now Cloud with one single comm…
from twitter_favs
23 days ago
Twitter
We used to configure a separate CDN for our JS bundles.
Ha, ha. Not anymore. Automatic caching worldwide ✅ Zero-con…
from twitter_favs
23 days ago
Twitter
Duct tape, latex gloves, and salted egg fish skins. !
from twitter_favs
24 days ago
Twitter
Be Nice And Write Stable Code:

💬Practical steps toward writing code that remains stable o…
from twitter_favs
24 days ago
Untitled (http://technosophos.com/2018/07/04/be-nice-and-write-stable-code.html)
Be Nice And Write Stable Code:

💬Practical steps toward writing code that remains stable o…
from twitter_favs
24 days ago
« earlier      
access addon ajax analytics android apache apex api apis aplix apple architecture archives archlinux argentina arm audio authentication aws backup bash bbc bios blog bluetooth bookmarks_bar boot broadband browser bts bug bugs build business c camera canvas charts checkout china chrome cognito community conference config configuration contacts cornwall css cycle cycling darwin dash data database debian debug delicious-export design desktop dev development device digital dns doc docker dooh dreamhost dwm dynamodb ecs editing email embedded encoding environment events example extension fcpx ffmpeg filesystem finland firefox flash flickr fonts food from gadget gdata geo gis git github go golang google gps graphics gstreamer gtk hack hardware hash history hosting howto html html5 http i18n identity image immigration input intel internet iphone ipod irc iso8601 japan japanese java javascript jpeg2000 jquery js json keyboard kiosk korea korean language laptop linux live livecd location london macosx maemo mail management map maps memory microsoft mikrotik mobile monitoring mozilla mp3 music my network networking news nodejs nokia npapi odf office offline ogg openid opensource opera oracle osx pdf performance php plugin plugins politics preseed presentation print privacy programming prometheus python qa rails raspberrypi reference rpi2 s3 s60 scripting sdk search security sent shell signage singapore software southafrica spanish ssl standards startup statistics streaming surrey svg sysadmin systemd technology template terminal test testing text thinkpad tips tools toread touch trac travel tutorial twitter ubuntu uk unicode unix unlabeled up update upload usb video vim visualization vmware vuejs w3c web webconverger webdev webkit webpy whatwg widget widgets windows wireless woking wordpress worldcup06 x40 x61 xcode xml xorg

Copy this bookmark:



description:


tags: