Friendly Bit - Web development blog

JustHTML 3.0.0: A new HTML5 parser architecture

2026-06-21T12:00:00+02:00

JustHTML 3.0.0 is out, and the biggest change is not a new API. It's a new parser core.

Up until now, JustHTML looked like most HTML5 parsers. First tokenize the input, then feed those tokens into a tree builder, and only after that apply the default-safe cleanup that makes untrusted HTML usable in applications.

That's the normal structure. The HTML5 spec itself is written that way. The tokenizer is one state machine, the tree builder is another, and the boundary between them is a stream of tokens: start tags, end tags, text, comments, doctypes, parse errors.

html5lib, browser engines, and html5ever all broadly follow that shape, even if the details differ a lot.

What changed in 3.0.0#

JustHTML 3.0.0 collapses that split into one plan-driven parser engine.

So instead of scanning characters into token objects, handing those tokens to a second subsystem, and then applying sanitizer decisions as a later pass, the new engine does that work in one loop.

It still implements the same HTML5 concepts: insertion modes, the open-element stack, active formatting elements, foster parenting, fragment parsing, RAWTEXT/RCDATA handling, foreign content rules, and all the other painful details that make browser parsing browser parsing.

But the control flow is different now. The parser scans the source string directly, decides what the current tag means in context, mutates the DOM immediately, and can apply default-safe policy decisions while it is still in the hot path.

This is a real architecture change, not just another round of optimization.

How it works#

The key idea is the word "plan".

Before parsing starts, JustHTML compiles the requested behavior into an EnginePlan. There are different plans for the common cases:

the default safe path
custom sanitization policies that can be compiled into parser actions
the raw path used by sanitize=False and transform-heavy cases

That plan contains the parser-time decisions that used to be scattered across later steps: tag actions, allowed tags, attribute handling, URL policy hooks, void-element knowledge, formatting-element behavior, and other mode-specific tables.

So the hot path is no longer asking "what should I do with this node later?" It already knows.

In practice the engine now looks more like this:

plan = compile_default_engine_plan(fragment=False)
engine = ParseEngine(html, fragment=False, plan=plan)
root = engine.parse()

Inside parse(), the engine sets up either a document shell or fragment root, then walks the input with a single range parser. On the fast path it uses specialized start-tag and end-tag parsers for compiled-safe mode, so it avoids building generic token objects and skips the tokenizer-to-treebuilder handoff completely.

Attributes are handled differently too. In the old shape, a tokenizer typically parses all attributes into token payloads, and then the tree builder or sanitizer revisits them. In the new JustHTML engine, attribute scanning can be projected directly through the current plan: preserve what is needed, drop what is not, and keep only the state required for correct tree construction.

That last part matters. HTML parsing is not just "keep the allowed attrs". Some information is needed for parser state even if it will never survive serialization.

Why this is faster#

The 3.0.0 changelog reports about a 2x speedup, and the reason is not very mysterious.

Traditional parser structure pays several overhead costs:

token objects have to be allocated
token payloads have to be normalized and handed off
the tree builder has to re-interpret information the tokenizer already discovered
default-safe behavior often becomes a separate tree walk or transform stage

The fused engine removes a lot of that machinery from the common path.

When JustHTML is used in its default mode, the parser can scan characters, recognize a tag, decide whether that tag is allowed, project the interesting attributes, and mutate the DOM immediately. Less indirection, fewer temporary objects, fewer full-tree passes.

This is the kind of optimization that sounds boring until you remember it's happening in Python, where object churn and extra passes cost real time.

The comparison to other parsers#

I still think the standard architecture is the safest place to start.

If you are implementing HTML5 from scratch, tokenizer and tree builder as separate layers is easier to reason about, easier to debug, and closer to the specification. It is also friendlier to test harnesses that want to inspect intermediate token streams.

So I don't think this proves everyone else wrong. html5ever and browser parsers are structured the classic way because that structure maps well to the spec and to large codebases with many contributors.

What JustHTML 3.0.0 changes is the tradeoff. It keeps the browser-style recovery model, but stops treating token emission as a required architectural boundary.

That makes JustHTML a bit unusual among HTML parsers. It is still pure Python, still targets exact HTML5 behavior, and still does safe-by-default parsing for application use. But the parser core is now closer to a fused execution engine than a textbook tokenizer plus tree builder pipeline.

I also think this is a better fit for what JustHTML actually is. Most users are not consuming a token stream. They want a correct DOM tree, and often they want it sanitized. If that is the real product, it makes more sense to optimize around that end-to-end job than around intermediate artifacts.

What did not change#

From the outside, this release is pleasantly boring.

JustHTML(html) still gives you a DOM. Fragment parsing still works. Streaming, source-location tracking, strict mode, and safe-by-default behavior are still there.

The main breaking change is diagnostics. Since the old tokenizer/tree-builder internals are gone, collect_errors=True and strict=True now report a smaller, more intentional built-in error set. If you depended on exact error codes, counts, or ordering, you will need to adjust.

Everyone else should mostly experience 3.0.0 as "the same parser, only faster".

That's exactly what I wanted.

JustHTML is now safe-by-default

2025-12-28T12:00:00+01:00

If you accept HTML from users (comments, profiles, CMS fields), you eventually hit the same problem:

You want to keep some markup.
You really don’t want to ship an XSS.

That’s why JustHTML now includes a built-in, policy-driven HTML sanitizer, and why serialization is safe-by-default.

Safe-by-default serialization#

JustHTML sanitizes when you serialize to HTML or Markdown:

from justhtml import JustHTML

user_html = (
    '<p>Hello <b>world</b> <script>alert(1)</script> '
    '<a href="javascript:alert(1)">bad</a> '
    '<a href="https://example.com/?a=1&b=2">ok</a></p>'
)

doc = JustHTML(user_html, fragment=True)

print(doc.to_html())
print()
print(doc.to_markdown())

This drops <script> and strips dangerous URLs:

<p>Hello <b>world</b>  <a>bad</a> <a href="https://example.com/?a=1&amp;b=2">ok</a></p>

Hello **world** [bad] [ok](https://example.com/?a=1&b=2)

Turning it off (trusted input only)#

If the input is trusted and you want raw output, you can opt out:

print(doc.to_html(safe=False))
print(doc.to_markdown(safe=False))

Custom allowlist policies#

The default policy is intentionally conservative, but you can provide your own SanitizationPolicy. Here’s a small example that only allows p, b, and a[href], and only allows https links:

from justhtml import JustHTML, SanitizationPolicy, UrlRule

policy = SanitizationPolicy(
    allowed_tags=["p", "b", "a"],
    allowed_attributes={"*": [], "a": ["href"]},
    url_rules={
        ("a", "href"): UrlRule(allowed_schemes=["https"]),
    },
)

doc = JustHTML(user_html, fragment=True)
print(doc.to_html(policy=policy))

If you’re sanitizing a full document, safe serialization keeps <html>, <head>, and <body> wrappers. For snippets, pass fragment=True to avoid implicit document wrappers.

There are also a couple of knobs that tend to show up in real systems:

URL proxying (for example, rewriting https://example.com/… to /proxy?url=…)
Optional inline styles, with an allowlist of CSS properties and conservative value checks

Why I built `justhtml-xss-bench`#

If you’ve worked on sanitizers before, you know the hard part isn’t writing a policy — it’s knowing what the browser will actually do with the result.

So I built a tiny benchmark harness: [justhtml-xss-bench](https://github.com/EmilStenstrom/justhtml-xss-bench/).

What it does:

Takes a payload vector and a sanitizer.
Sanitizes the payload.
Embeds the sanitized output into the initial HTML page ("server-side" style).
Loads it in a real Playwright browser engine.
Fails the case if JavaScript executes (including signals like dialogs or attempted external script fetches).

It ships with 7,000+ real-world XSS vectors and can be used to compare JustHTML’s output with other sanitizers.

If you want to explore it locally, the CLI looks like this:

# Run all vector files in ./vectors against the default sanitizer set
xssbench

# Limit to one engine
xssbench --browser chromium

# List available sanitizers
xssbench --list-sanitizers

# Run a subset
xssbench --vectors vectors/bleach.json --sanitizers noop

Threat model (what “safe” means)#

JustHTML’s sanitizer aims to prevent script execution when you sanitize untrusted HTML and embed the result into an HTML document as markup.

It does not make it safe to drop the output into JavaScript string contexts, CSS contexts, URL contexts, or other non-HTML contexts — those need their own escaping/handling.

If you want the details, see the JustHTML sanitization documentation:

https://github.com/EmilStenstrom/justhtml/blob/master/docs/sanitization.md

And the benchmark harness repo:

https://github.com/EmilStenstrom/justhtml-xss-bench

JustHTML: Addressing some questions

2025-12-19T12:00:00+01:00

When Simon Willison wrote about JustHTML [1] [2], suddenly everyone was interested in giving their view. After reading through (what I think is) all of them, I thought I'd address some questions that have arisen.

"This is a copy / this is derived work!"#

It is very unclear if JustHTML is a derived work. About halfway in I did tell the LLM to "port html5ever", but I don't think that's what the LLM did. It started from the code structure of html5ever, but much of the code was trial and error against the html5lib-tests suite. In later versions I asked it to refactor much of the code, so I don't think even the structure is there any more.

I asked an agent to try to find cross references between the two projects that still remain, but all it found were things that were also in the WHATWG HTML5 specification. This doesn't say it's not derivative work, but highlights that it's far from clear.

If you can find cases where the code is very similar (and not specifically required by spec), I would be happy to see it!

"He stripped the authorship / laundered the license!"#

This wording assumes an active attempt to strip it, which is opposite of me adding the html5ever acknowledgement to the JustHTML README. Just stop that nonsense. I have nothing but love for the html5ever developers. To put that to rest, I've decided to 1) add their copyright block to my license anyway and 2) ask them specifically for guidance. I'm looking forward to hearing their view.

For reference, this is the current LICENSE file in JustHTML:

MIT License

Copyright (c) 2025 Emil Stenström
Copyright (c) 2014-2017, The html5ever Project Developers

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
...

"He doesn't understand the code"#

Many commenters were angry that I didn't understand the code. This is such an interesting take, especially in a society where not understanding is seen as weakness. We should be certain! We should know everything! But we don't. We're all fallible and walk around trying to figure things out.

In the specific case of the HTML5 specification, there are very few people—in the world—who understand it. HTML5 is an intricate web of algorithms, that interact in difficult-to-understand ways (see htmlparser.info for a great guided tour). Did you know that the tokenizer and treebuilder affect each other?

Luckily for us, the authors decided to ~~shame~~ help browsers interoperate by publishing the fantastic html5lib-tests suite. It's an incredible feat of engineering, with thousands of integration tests that (almost) completely test the specification.

Here is a representative example of the kind of input those tests test for:

<p>One <b>two <i>three</b> four</i> five
<table><p><tr><td>cell</table>
<svg><foreignObject><p>inside svg</foreignObject></svg>

That single snippet touches multiple "special" parts of the spec: mis-nested formatting elements (the adoption agency algorithm), implied end tags, table insertion mode oddities, and foreign content integration points.

What's fantastic about html5lib-tests is that it gives us a way to look at our code from the outside and see if it works or not, without us having to understand it. If this feels extreme, think of low-level code—assembler—if you will. Do you understand how it flips the transistors in your computer? I don't. And that's fine, because you have other ways to know that your code works. You don't have to go into the details.

LLMs are quickly becoming a new layer on top of the code we write. If we can find a way to prove that it works, we don't need to understand it. That opens up whole new possibilities!

"It's not high quality code, because it won't be maintained"#

I'm very sure that this code is maintainable, because I have been maintaining it for a while already. As I was approaching 100% test coverage, a new HTML5 feature was added to the test suite: <selectedcontent>. This was easily supported with a couple of queries to the LLM.

I am planning to maintain it. The PRs are rolling in, and I have quite a clear image of where I want to take it. I think the API I've put on top of the parser is really attractive, with a very low learning curve. That's worth something, and it's missing from all the other libraries.

The first versions of the library were very hard to maintain, even with LLM help. When I looked under the hood there were messy nested if blocks, that mirrored some of the test data exactly. The LLM was cheating! I have not seen signs of this in the later versions of the code, and especially since LLM models got better.

I'm happy that my little experiment triggered so many discussions. Overall, I think (and you are free to disagree) that having this library is a big net positive for the Python community.

How I wrote JustHTML using coding agents

2025-12-03T12:00:00+01:00

I recently released JustHTML, a python-based HTML5 parser. It passes 100% of the html5lib test suite, has zero dependencies, and includes a CSS selector query API. Writing it taught me a lot about how to work with coding agents effectively.

I thought I knew HTML going into this project, but it turns out I know nothing when it comes to parsing broken HTML5 code. That's the majority of the algorithm.

Henri Sivonen, who implemented the HTML5 parser for Firefox, called the "adoption agency algorithm" (which handles misnested formatting elements) "the most complicated part of the tree builder". It involves a "Noah's Ark" clause (limiting identical elements to 3) and complex stack manipulation that breaks the standard stack model.

I still don't know how to solve those problems. But I still have a parser that solves those problems better than the reference implementation html5lib. Power of AI! :)

Why HTML5?#

When picking a project to build with coding agents, choosing one that already has a lot of tests is a great idea. HTML5 is extremely well-specified, with a long specification and thousands of treebuilder and tokenizer tests available in the html5lib-tests repository.

When using coding agents autonomously, you need a way for them to understand their own progress. A complete test suite is perfect for that. The agent can run the tests, see what failed, and iterate until they pass.

Building the parser (iterations, restarts, and performance work)#

Writing a full HTML5 parser is not a short one-shot problem. I have been working on this project for a couple of months on off-hours.

Tooling: I used plain VS Code with Github Copilot in Agent mode. I enabled automatic approval of all commands, and then added a blacklist of commands that I always wanted to approve manually. I wrote an agent instruction that told it to keep working, and don't stop to ask questions. Worked well!

Here is the process it took to get here:

A one-shot HTML5 parser (as a baseline)#

To begin, I asked the agent to write a super-basic one-shot HTML5 parser. It didn't work very well, but it was a start.

Wiring up `html5lib-tests` (<1% pass rate)#

Next, I wired up the html5lib-tests and saw that we had a <1% pass rate. Yes, those tests are hard. They are the gold standard for HTML5 parsing, containing thousands of edge cases like:

<b><p></b></i>

Iterating to ~30% coverage (refactors and bugfixes)#

After that, we started iterating, slowly climbing to about 30% pass rate. This involved a lot of refactoring and fixing small bugs.

Refactoring into per-tag handlers#

Once I could see the shape of the problem, I decided I liked a handler-based structure, where each tag gets its own handler. Modular structure ftw! I asked the agent to refactor and it did.

class TagHandler:
    """Base class for all tag handlers."""
    def handle_start(self, context, token):
        pass

class UnifiedCommentHandler(TagHandler):
    """Handles comments in all states."""
    def handle_start(self, context, token):
        context.insert_comment(token.data)

Reaching 100% test coverage (with better models)#

From there, we continued iterating to 100% test coverage. This took a long time, and the Claude Sonnet 3.7 release was the reason we got anywhere at all.

Benchmarking and discovering we were 3x slower#

With correctness handled, I set up a benchmark to test how fast my parser was. I saw that I was 3x slower than html5lib, which is already considered slow.

Rewriting the tokenizer in Rust (and barely matching `html5lib`)#

So I tried the obvious next move: I let an agent rewrite the tokenizer in Rust to speed things up (note: I don't know Rust). It worked, and the speed barely passed html5lib. It created a whole rust_tokenizer crate with 690 lines of Rust code in lib.rs that I couldn't read, but it passed the tests.

Discovering `html5ever` (fast, correct, Rust)#

While looking for alternatives, I found html5ever, Servo's parsing engine. It is very correct and written from scratch in Rust to be fast.

Asking: why build this at all?#

At that point I had the uncomfortable thought: why would the world need a slower version of html5ever in partial Python? What is the meaning of it all?! I almost just deleted the whole project.

Pivoting to porting `html5ever` logic to Python#

Instead of quitting, I considered writing a Python interface against html5ever, but decided I didn't like the hassle of a library requiring installing binary files. So I went pure Python again, but with a faster approach: what if I port the html5ever logic to Python? Shouldn't that be faster than the existing Python libraries? I decided to throw all previous work away.

Restarting from scratch (again)#

So I started over from <1% test coverage and iterated with the same set of tests all the way up to 100%. This time I asked it to cross reference the Rust codebase in the beginning. It was tedious work, doing the same thing over again.

Still slower than `html5lib`#

Unfortunately, I ran the benchmark on the new codebase and found that it was still slower than html5lib.

Profiling, real-world benchmarks, and micro-optimizations#

So I switched to performance work: I wrote some new tools for the agents to use, a simple profiler and a scraper that built a dataset of 100k popular webpages for real-world benchmarking. I managed to get the speed down below the target with Python micro-optimizations, but only when using the just-released Gemini 3 Pro (which is incredible) to run the benchmark and profiler iteratively. No other model made any progress on the benchmarks.

def _append_text_chunk(self, chunk, *, ends_with_cr=False):
    if not chunk:
        self.ignore_lf = ends_with_cr
        return
    if self.ignore_lf:
        if chunk[0] == "\n":
            chunk = chunk[1:]
            # ...

Deleting untested code (coverage as a scalpel)#

Later, on a whim I ran coverage on the codebase and found that large parts of the code were "untested". But this was backwards, because I already knew that the tests were covering everything important. So lines with no test coverage could be removed! I told the agent to start removing code to reach 100% test coverage, which was an interesting reversal of roles. These removals actually sped up the code as much as the microoptimizations.

# Before: 786 lines of treebuilder code
# After: 453 lines of treebuilder code
# Result: Faster and cleaner

Fuzzing to find crashes and harden the parser#

After removing code, I got worried that I had removed too much and missed corner cases. So I asked the agent to write a html5 fuzzer that tried really hard to generate HTML that broke the parser.

def generate_fuzzed_html():
    """Generate a complete fuzzed HTML document."""
    parts = []
    if random.random() < 0.5:
        parts.append(fuzz_doctype())
    # Generate random mix of elements
    num_elements = random.randint(1, 20)
    # ...

It did break the parser, and for each breaking case I asked it to fix it, and write a new test for the test suite. Passed 3 million generated webpages without any crashes, and hardened the codebase again.

Comparing against other parsers (how rare 100% is)#

To sanity-check where 100% landed, I ran the html5lib tests against the other parsers. I found that no other parser passes 90% coverage, and that lxml, one of the most popular Python parsers, is at 1%. The reference implementation, html5lib itself, is at 88%. Maybe this is a hard problem after all?

Shipping it as a library (CI, releases, selector API)#

Finally, to make this a good library I asked the agent to set up CI, releases via GitHub, a query API, write READMEs, and so on.

from justhtml import JustHTML, query

doc = JustHTML("<div><p>Hello</p></div>")
elements = query(doc, "div > p")

Decided to rename the library from turbohtml to justhtml, to not fool anyone that it's the fastest library, and instead focus on the feeling of everything just working.

What the agent did vs. what I did#

After writing the parser, I still don't know HTML5 properly. The agent wrote it for me. I guided it when it came to API design and corrected bad decisions at the high level, but it did ALL of the gruntwork and wrote all of the code.

I handled all git commits myself, reviewing code as it went in. I didn't understand all the algorithmic choices, but I understood when it didn't do the right thing.

As models have gotten better, I've seen steady increases in test coverage. Gemini is the smartest model from a one-shot perspective, while Claude Opus is best at iterating its way to a good solution.

Practical tips for working with coding agents#

Start with a clear, measurable goal. "Make the tests pass" is better than "improve the code."
Review the changes. The agent writes a lot of code. Read it. You'll catch issues and learn things.
Push back. If something feels wrong, say so. "I don't like that" is a valid response.
Use version control. If the agent goes in the wrong direction, you can always revert.
Let it fail. Running a command that fails teaches the agent something. Don't try to prevent all errors upfront.

Was it worth it (and what “quickly” meant)?#

Yes. JustHTML is about 3,000 lines of Python with 8,500+ tests passing. I couldn't have written it this quickly without the agent.

But "quickly" doesn't mean "without thinking." I spent a lot of time reviewing code, making design decisions, and steering the agent in the right direction. The agent did the typing; I did the thinking.

That's probably the right division of labor.

3 tips for agile roadmaps

2023-05-18T18:17:00+02:00

"Roadmapping is anti-agile. Instead, we should only have priorities."

This was a response to my product roadmaps article - discuss!

Are roadmaps anti-agile?
— Ant Murphy (@ant_murphy) May 17, 2023

My response: Product Roadmaps can be agile or not, it depends on how you use them.

The idea behind a roadmap is to deliver predictability to the business, but how does that work with also being able to respond to changes?

1. Update it regularly#

Once a year is too slow, and leaves no wiggle room for new insights or priorities throughout the year. But updating it too often makes it hard to use instead. I've seen quarterly updates work well for a 50 people company. Shorter for a smaller or younger company, longer for a bigger or more slow moving one.

2. Schedule broad areas/problems, not features#

Features change more often than problems, so putting problem areas in the roadmap works better for both delivering some predictability and leaving space for learning. The clearer you can be with what effect you are looking for, the more empowered the team can be in solving the problem.

3. Don’t make everything time bound#

If every item in the roadmap is time bound, then updating it regularly will be though. Put a time frame on the nearest items, that you know more about, so the business can plan. But don't worry about the ones longer in the future.

That's it, three quick tips to work with agile roadmaps. Sounds easy? It's not.

How to prioritize a feature list, the hard way

2023-06-08T07:53:00+02:00

What % of the top 10 [features that customers ask for] do you end up implementing typically?
— Jason Cohen (@asmartbear) June 7, 2022

Raw customer feedback is always too much about features, and too little about the problems customers are trying to solve.

So you need to rework the feedback to map to the problems customers have instead, merging and splitting feedback as you go. Hard work. Now you have transformed your list of features to a list of problems.

Can’t you just build things in this list in “most common problems” order? No. Because how often a problem occurs is not the same as how important it is.

Also, the type of user that has that problem is also very important.

In a B2B context, the buyer is much more important than the user. In B2C there are different kinds of users, which have wildly different priority.

And there are likely five other important factors too, all specific to your specific problem area, that we haven't even talked about yet.

Could you build a ranking system weighing these things together, and get a bottom up priority list?

You could, but it wouldn’t be at all as exact as it looks to be. It’s easy to fool yourself. Remember that the raw customer was very biased, is this something to build a solid mathematical model based on? I think not.

A better way is to not build a ranking model, and work from the list of common customer problems.

Instead, look at your company’s business goals, and start mapping customer problems to business goals.

Say some users have problems with the checkout flow. This maps directly to a revenue target.

Someone didn’t like the design of the site. Talking to them, you find that that leads to them not trusting you. Which indirectly leads to lost sales.

So maybe you put up “Increase customer trust” as a node in a tree and continue mapping. As you add more feedback, and more nodes, you get a much better picture of how things fit together.

Looking at the whole tree, you can now prioritize which customer problems to work on, based on business goal priority. This will make a lot more sense.

Because your job is not to build features, or even to solve customer problems. It’s to drive business outcomes.

And yes, you do that by solving customer problems, and building features. But that’s not where you start.

That’s how you prioritize a feature list, the hard way.

The case for national digital currencies, not Bitcoin

2021-06-09T10:06:00+02:00

Bitcoin is now accepted as legal tender in El Salvador. Many believe all countries should do the same, and that the time is right for Bitcoin to take over the world as a global digital currency.

Decentralization is not a good idea for a currency#

The core tenet of most cryptocurrencies is decentralization, the idea that no single entity or individual should have absolute control over it. Instead it's protected by the fact that controlling 51% of the network is hard when lots of diverse entities are involved. The push for decentralization in cryptocurrencies is the root of most of its problems.

For the 51% protection to work you need a large network of independent server operators
To get that you need to give them some incentive to host their servers
It's natural then, to give them some of the new currency in return
But the currency is worth zero when no one is using it...
So everyone who joins early, bets that the value will not be stable, but grow exponentially over time

This is not the recipe for how to construct something that should be stable, and be used as a digital currency.

Instead we should skip the decentralization part, and have a central bank guarantee the value. This makes it stable, and stable is something we definitely need for a currency.

Proof of work makes things worse#

To get any value from a cryptocurrency you need to limit supply somehow, if there was infinite money, it wouldn't be worth anything. For Bitcoin, this is done by requiring each server to randomly try solutions to a tricky algorithmic problem. Solving it gives you coins.

Proof of work for Bitcoin is a colossal waste of electricity. Especially in countries where electricity is generated by non-clean energy sources. But even for renewables, is this really the best way to use our limited energy? Let's charge car batteries, construct fossil free steel, and capture CO₂ from the atmosphere with that electricity instead.

There are other ways to limit supply in other cryptocurrencies, like proof of stake, and I agree that these would be better for a currency. But the easiest solution is to let a central bank guarantee the value, and avoid the need for a "proof of X" system at all.

Ease of use for crime is not a good feature#

There is a reason crime groups love Bitcoin: Transactions are anonymous. All you need is a Bitcoin address to send money to. This means no taxation, as transactions are not tied to any individual. It also means that cyber criminals just need to transfer all proceeds from a hack to their own accounts, no need to hide things further.

When crime groups get the same access to capital as everyone else they can do more damage. I hope you agree that this is something we should try to limit. But this is not something that cryptocurrencies can do, because it requires a centralized authority that can act on suspicion of wrongdoing. We already have that system in place in our governments, it's just that cryptocurrencies are not built with stopping crime in mind. A new digital currency should take crime into consideration.

Of course, accessing your transactions should require reasonable cause as decided by a judge, and be protected by strict privacy laws. Probably better protected than your bank statement is today.

Another look at the El Salvador use of Bitcoin#

When you look closer at how El Salvador plans to use Bitcoin a couple of things stand out:

20% of El Salvador's GDP is people sending money to relatives, and they want to encourage that
All merchants will be mandated to accept Bitcoin as payment
The government will accept the monetary risk of merchants by directly converting Bitcoin to US dollars (their native currency)
There's a trust fund with $150 million to balance the risk
They will continuously sell off Bitcoin to balance the trust fund
There is talk of the state promoting bitcoin mining

This is not something other countries should try to emulate:

Tying a country's finances to a volatile security like Bitcoin is a very high risk move. If the Bitcoin price where to crash, all the Bitcoin the government holds would be worthless. It could also rise, but this is nothing but reckless.
Receiving money from relatives requires their relatives to accept the risk of holding Bitcoin. If you buy Bitcoin one day, it might be worth half what you bought it for the day after. The government will not mitigate that risk.
The tech to accept Bitcoin might not be readily available to everyone, especially a poor country like El Salvador.
As previously stated, betting on Bitcoin is not environmentally sound.

Requirements for an ideal digital currency#

So let's sum up what I've talked about so far, in terms of what I believe are requirements for a digital currency.

It should be stable, so it can be used for day-to-day transactions
Environmental impact of running the system should be low
Law enforcement should be able to access to transaction data when warranted

Let's add some things that Bitcoin is doing well:

It should be available to anyone
There should be a digital wallet that it's easy to send money to
Transaction costs should be low, or even zero

Let's see if we can build a system that would satisfy all six requirements above, fully or partially.

A national digital currency#

Each country should have their own digital currency, backed by that country's central bank. The digital money would be considered the same as paper money, and the central bank would be in control of minting new digital money, just like they are with paper money today.

They don't need a blockchain system for this, as there is trust in this system. So no proof of work, and no server farms. A simple database with a transaction ledger would be fine. Of course it needs to be protected like a bank's IT system would. This makes sure we're operating with low environmental impact.

A new government agency would be created whose job it would be to safeguard the transaction ledger in a privacy sensitive way. The agency should have strict rules for when transaction data can be disclosed to law enforcement and not. I believe separating responsibility like this would be an effective way to make sure there is better balance between the needs of law enforcement and privacy.

Every citizen should get a digital wallet from the government based on their social security number (or equivalent) or registered company ID. This account could be used for government payouts instead of other bank accounts. Ideally there should be a way to easily add accounts for people that live outside the system (paperless, immigrants living in the country, ...), so that these digital accounts really could be used by everyone.

There should be a basic interface supplied by the government for making payments, but also an API so private organizations could build other payment solutions using these digital wallets. Transaction costs should be low or zero when done through the government supplied interface, but third parties could add fees when doing transactions through the API. Allowing fees and an API ensures that there are ways for commercial companies to improve the payment flow in interesting ways.

In summary#

I think a setup like this would be great for simplifying digital commerce in a country. It would expand access to digital services, open plentiful of new use-cases, and decrease the cost of making digital transactions.

Luckily, it seems at least 80 central banks are exploring this use-case. Let's hope we see more action, and less talk.

[The Listserve] How randomness affects your life

2017-12-02T12:04:00+01:00

It's a mistake not to think about randomness, for randomness is what you get if you don't make choices. Do you really want randomness controlling the most important events in your life?

There are two kinds of randomness: The "true" kind, where it's impossible, using the science we know, to know the outcome. Maybe you've heard about the Heisenberg's uncertainty principle. It says that, in some circumstances, if we try to measure, we destroy the thing we wanted to measure. It's not measurable, we don't know, therefor it's random. This is a basic thing in nature, a randomness we can't change no matter how hard we try.

Then there's the more common kind. The kind that SEEMS random to us. If we throw a die, and three comes up, was that randomness? Not really, because if we could throw the die in exactly the same way, the same number would come up. That's not how real randomness works, but it's the kind of random we surround ourself with every day. Let's focus on this kind of every-day randomness.

Is there any way to control randomness? Yes! You can reduce uncertainty. Example: Say you want to meet a partner, something that is seemingly random (we've all hear the stories of how people "randomly" bumped into each other). Is there something you can do to make that more likely (i.e. less random)? Maybe you could hang out in places where a good partner is likely to hang out? Probably not in your livingroom right? At a local bar? At an evening course in pottery? Think about it. This is you consciously reducing the amount of randomness around meeting someone.

So while randomness is something that affects us all, you can affect it by just knowing more, and adapting. Now, knowing this, is there some area of your life that you find extra important? Is there something that you really would miss if it got lost on you? Maybe it's time to reduce the uncertainty and randomness around that thing? Is there something you could do right now?

My point: Don't let randomness control what happens to the most important parts of your life. If you don't do anything, it will.

How to make e-mail encrypted for everyone

2016-10-28T23:26:00+02:00

Update: For a wholesome look at e-mail security this video from the CTO of ProtonMail: https://vimeo.com/216747532

When you send an e-mail today it's sent in plaintext. This means that when you connect to your local coffee shop's WiFi they can intercept all e-mail that is sent through their router. This is probably not the relationship you have with your barista…

E-mail is way more important than this, and should receive the same privacy protection as the web has with HTTPS. I haven't seen any meaningful progress on e-mail encryption in a long time, so this article is a suggestion of how to get things moving.

The base idea is this: We should make e-mail work just like the web in terms on encryption:

Encryption should be added gradually, without breaking backwards compatibility (interoperability)
Intermediate servers shouldn't be able to read the message (privacy)
Users should be able to keep sending and receiving e-mail like they always have (usability)
E-mail clients and e-mail providers should be able to easily implement the scheme (simplicity)

Encrypt as much as possible of the e-mail message#

For web traffic, we've encrypting everything except the domain name of the site we're connecting to, this is how e-mail should work too. Since we need to know who to send the e-mail to, we should encrypt everything except the recipient address.

Some of you might have read about STARTTLS, a way to make SMTP encrypted. Unfortunately STARTTLS has several problems that makes it unfit to solve the privacy problem:

It only encrypts the message between the many servers your e-mail passes through. Your e-mail is decrypted and exists in plaintext on each of the intermediate servers. If any of the intermediate servers is hacked they get full access to all e-mails passing through. You likely don't have that kind of relationship with your ISP network administrator either…
If any of the intermediate servers doesn't support STARTTLS, the connection is downgraded to plaintext instead. One old server and the whole chain breaks.

We should apply encryption on the client when sending, and decrypt at the recipient. No intermediate servers should get access to the message.

Encrypt with public-key cryptography (without the hassle)#

The best way to handle this encryption is with public-key cryptography, where the sender first fetches the public key of the receiver, sends the message encrypted with that key, and then that the receiver decrypts it using their private key.

People are already sending e-mail this way with PGP, so way not just use this everywhere? It has several big usability problems:

You need to do key management yourself. On the web the browser handles all cryptography for you. You type the address in the browser and everything just works. This is not the case with PGP. Look at this guide in the eyes of a normal person: OpenPGP for a complete beginner - it quickly goes into downloading zipfiles, package managers and using the command line to generate keys.
You need to install software on your computer. Most e-mail providers are moving to the web, having to install a browser plugin just to send e-mail is not the experience we should strive for. I understand that this is means your e-mail provider can read your e-mail, so it won't work for everyone. But I think most people will accept that the same way they accept that their bank can read their bank statement. If we can trade this for usability it's still a HUGE step up for privacy.

Here's how this could be done differently:

Encryption:

For each recipient in the To, Cc, and Bcc fields:
1. Do a lookup against their domain to see if it supports encryption. This should be a new DNS record that points to an address that the e-mail client can use to fetch the public key for a specific user of that domain.
2. Fetch the public key for each of the users of the domain
3. If both the domain has encryption support and the public key can be fetched, show a lock symbol beside that e-mail address to show that the message to that recipient will be encrypted. Clicking the lock can show information about the encryption that will be used, just like browsers do today.
When the message is sent, split it into as many parts as the number of recipients and encrypt each part with that recipient's public key. Send it off to each recipient using the normal protocols. Users on domains that don't support encryption will get messages sent in plaintext, just like today. This ensures support can be added gradually.

Note that for the user nothing changes compared to how they send e-mail today, except that the lock shows up. Web users are slowly learning about the importance of the lock symbol when browsing the web, transfering that knowledge to e-mail too will create a push for e-mail server operators to support encryption.

Decryption:

When the encrypted e-mail reaches the intended recipients e-mail server, that server uses the private key for that user to decrypt the message and (securely) deliver it the the users inbox. The user reads their e-mail like they've always have.

Some may frown reading that I suggest leaving the private key in the hands of your e-mail provider. I agree that this means you lose some privacy to your e-mail provider. The upside here is that we gain a lot of usability. An e-mail provider (such as Gmail) could and an e-mail client (such as Thunderbird) could update all their software tomorrow and all e-mail sent from Thunderbird to Gmail would be instantly encrypted. This removes the burden of normal users to learn about cryptography to be safe on the web.

In summary, these are the changes that needs to happen for e-mail move towards being encrypted by default:

Consequences for e-mail clients#

New features needed:
- Fetch a DNS record for each receiving domain they want to send an e-mail to
- Fetch each receiving user's public key in the background
- And encrypt as much as possible of the outgoing e-mail
- Send the e-mail using the normal protocols
Ways to pitch this to users:
- All e-mail sent to e-mail providers that support this scheme will be encrypted by default, with no effort on your part
- You can easily see which addresses get encrypted messages with the lock icon that you know from browsers
- Increased privacy from prying eyes

Consequences for e-mail providers#

New features needed:
- Add a DNS record that points to their public keys
- Serve the public key of a specific user that the e-mail clients ask for
- Decrypt incoming e-mails using the private key of that user
- Deliver the e-mail using the normal protocols
Ways to pitch this to users:
- All e-mails sent from clients that support this scheme will be encrypted by default, with no effort on your part
- Increased privacy from prying eyes, do you really trust your local coffee shop as much as your e-mail provider?
Business reasons:
- Support privacy without letting go of the chance to serve ads based on the content of the users e-mails. This is a very important point for big players like Gmail that serve ads. Users or course still have the option to use another e-mail provider or add PGP if they want.
- Nothing in this scheme blocks smaller players from making their users safe using the same methods as the big players will.

In summary#

I think this is a very straightforward way to make all e-mail encrypted. If anyone has a better suggestion that balances the needs of users, e-mail clients and e-mail providers differently I'm all ears. But let's not let this privacy travesty go on much longer…

Hur säkra är svenska banker? (Swedish)

2015-03-01T19:22:05+01:00

Säkerheten varierar stort bland svenska banker. Bankens storlek hänger inte ihop med hur väl dom hanterar säkerheten. Ingen av de fyra storbankerna kom med på topplistan. Några små banker finns med i toppen, några riktigt stora finns i botten.

För att testa bankernas säkerhet gav jag varje bank två olika betyg:

Kryptering: Krypteras trafiken mellan dig och banken ordentligt?

Utan kryptering kan andra se de uppgifter som du skickar till banken, personnummer, lösenord, och koder. Dessa relativt tekniska tester görs med hjälp av SSL Labs. Betygsskalan kommer direkt från SSL Labs och går från A-F, där A är bäst. Notera att jag testar inloggningssidan, inte startsidan.
Visuellt: Ser allt ok ut i webbläsaren?

Det finns gott om guider som lär ut webbsäkerhet till ovana användare. Här tittar jag på om banksajterna är konfigurerade så att allt ser OK ut för en vanlig användare. Har sajten grönt hänglås? Jag har använt samma betygsskala: A-F, där A är bäst.

Bankerna kommer från Finansinspektionens lista över bankaktiebolag och medlemsbanker. De lokala sparbankerna har inte tagits med, och inte heller de som främst riktar sig till företag. Testet gjordes 2015-02-28. Rådata för testet finns här, inklusive länkarna till SSL Labs där du kan se testresultatet själv.

Uppdaterat:#

Eftersom denna artikel fått väldigt bra spridning har flera av bankerna hört av sig. Ofta med konkreta förbättringar de har genomfört på grund av min artikel. Här hittar du en tidslinje över hur artikeln har ändrats över tid.

2015-03-03: Inga fler uppdateringar görs från detta datum och framåt...
2015-03-02: Danske Bank tillagd. Dom får tyvärr bara ett C.
2015-03-02: Kontrollerade stöd för DNSSEC: Landshypotek, Marginalen, Danske Bank, och Länsförsäkringar är de enda som stödjer det.
2015-03-03: Rättade felaktig klassificering av SBAB. Jag hade testat startsidans certifikat, inte inloggningssidans.
2015-03-04: SBAB Bank har förbättrat sitt certifikat på startsidan.
2015-03-04: Länsförsäkringar har uppdaterat sitt certifik och får nu ett B.
2015-03-05: Ekobanken har uppdaterat sitt certifikat och får nu ett A.
2015-03-05: Forex bank har uppdaterat sitt certifikat och får nu ett A.
2015-03-05: Santander Consumer har släppt en ny internetbank och hamnar nu på B.
2015-03-06: MedMera Bank har uppdaterat sitt certifikat och får nu ett B.
2015-03-07: Carnegie Bank har uppdaterat sitt certifikat och får nu ett A
2015-03-08: SBAB Bank har gjort ytterligare uppdateringar och blir första bank med A/A. Full pott!
2015-03-12: JAK Medlemsbank har uppdaterat sitt certifikat och får nu ett B.
2015-03-12: Länsförsäkringar har uppdaterat sitt certifikat och får nu ett A-.
2015-03-18: Swedbank kör nu HTTPS på sin startsida.
2015-04-11: Resurs Bank har uppdaterat certifikatet på sin startsida, och kräver nu även https där. Inloggningscertifikatet kommer att uppdateras också, men är inte klart ännu.
2015-04-24: Skandiabanken kör nu HTTPS på sin startsida.
2015-10-28: Resurs Bank har flyttat sin inloggningssida till egen domän och certifikat.

Bankerna med bäst säkerhet (A):#

Fantastiska resultat för alla dessa fyra banker. Krypteringen uppfyller best practices och du kan sova gott med vissheten att din bank är bland Sveriges bästa.

	Kryptering	Visuellt
Betyg	Not	Betyg	Not
SBAB	A	1	A
Skandiabanken	A	1	A
Resurs Bank	A	1	A
Landshypotek	A	2	B	4
Carnegie Investment Bank	A		B	4
Ekobanken	A		B	4
Forex Bank	A		B	4
Länsförsäkringar Bank	A-	3	A
Marginalen Bank	A-	3	A
Nordnet	A-	3	A

Noter:

Grattis till SBAB Bank, Skandiabanken och Resurs Bank som får A i båda kategorier.
En extra eloge till Landshypotek som får noll varningar från SSL Labs.
Flera banker får A minus eftersom de saknar stöd för Forward Secrecy. Förenklat innebär det att krypteringsmetoden ändras över tid, så att någon som avlyssnat trafik till en sajt för länge sedan inte kan dekrypteras den om sajten hackas senare. Totalt har endast tre svenska banker har stöd för detta (Landshypotek, Skandiabanken och ICA Banken (se nedan)). Bra jobbat!
Flera bankers startsidor saknar kryptering. Eftersom inloggningssidorna är krypterade är detta inte katastrof, bara olyckligt. Att ha kryptering på hela sin webbplats är en bra idé eftersom kunden vänjer sig vid att se det gröna hänglåset så länge dom har med banken att göra.

Bankerna som nästan kom med på topplistan (B):#

Även här får bankerna bra betyg. Anledningen att dom missar topplistan är att samtliga banker i denna lista har problem med vilka krypteringsalgoritmer/protokoll dom stödjer. Det är enkelt egentligen: Om en krypteringsmetod är bevisat osäker (t.ex. RC4 eller SSL3) ska den plockas bort. Att ha den kvar innebär att kunderna luras att tro att deras uppkoppling mot banken är säker trots att den inte är det.

	Kryptering	Visuellt
Betyg	Not	Betyg	Not
ICA Banken	B	1	A
Avanza	B		A
MedMera Bank	B		A
Santander Consumer	B		A
JAK Medlemsbank	B		A
Swedbank	B		A
SEB	B		B	2
Volvofinans Bank	B		B	2
Ikanobanken	B		B	2
Nordea Bank AB	B		B	2
Handelsbanken	B		B	2

Noter:

ICA Banken är nästan med i topplistan ovan. Dom krypterar sin startsida, dom stödjer Forward Secrecy, och är dessutom den enda bank som stödjer HTTP Strict Transfer Security. Tyvärr används krypteringsalgoritmen RC4, en algoritm som inte är säker alls. Har du en lite äldre webbläsare kan du alltså bli lurad att allt ser bra ut, trots att trafiken mellan dig och banken relativt enkelt kan avlyssnas. ICA Banken borde helt enkelt stänga av stöd för RC4, be sina användare att uppgradera sin webbläsare, och inta toppositionen i denna lista.
Flera bankers startsidor saknar kryptering. Eftersom inloggningssidorna är krypterade är detta inte katastrof, bara olyckligt. Att ha kryptering på hela sin webbplats är en bra idé eftersom kunden vänjer sig vid att se det gröna hänglåset så länge dom har med banken att göra.

Bankerna som du borde vara orolig för (C):#

Någonstans här är det dags att börja bli orolig. Dessa banker har gemensamt att dom inte hanterar sin kryptering på rätt sätt. Extra olyckligt är det ställt för OK-Q8 Bank som använder en färdig produkt från EDB och ändå får sämst betyg av C-bankerna.

	Kryptering	Visuellt
Betyg	Not	Betyg	Not
Danske Bank	B		F	2
Erik Penser	C	3	F	4, 5
OK-Q8 Bank	C	3	F	1, 4, 6

Noter:

OK-Q8 Bank lyckas vända något bra - det gröna hänglåset - till något mindre bra. Ett klick på det gröna hänglåset visar vem som garanterar säkerheten på webbplatsen. Bakom OK-Q8 Bank står EVRY AS, ett norskt bolag, som garant.Hur ska en kund veta att det är banken som står bakom det certifikatet, att dom har kommit till rätt sajt? Underkänt.
Danske Bank har bara har satsat på en grått hänglås. Den gråa hänglåset betyder att uppkopplingen är krypterad, men inte vem som står bakom den (så kallad Extended Validation). För något så känsligt som banktjänster ska ett riktigt EV-certifikat användas.
Flera banker verkar inte ha skyddat sig från ett säkerhetshål som döpts till POODLE. Google publicerade detaljer om hur man använder sig av hacket redan i oktober 2014. Att det fortfarande finns banker som inte skyddat sig är anmärkningsvärt.
Flera bankers startsidor saknar kryptering. Eftersom inloggningssidorna är krypterade är detta inte katastrof, bara olyckligt. Att ha kryptering på hela sin webbplats är en bra idé eftersom kunden vänjer sig vid att se det gröna hänglåset så länge dom har med banken att göra.
Erik Penser gör kapitalfel på sin inloggningssida, den ser inte ut att vara krypterad. Som kund ska man aldrig skriva in känsliga inloggningsuppgifter på en webbsida som saknar hänglås. Erik Penser gör visserligen rätt bakom kulisserna (inloggningssidan ÄR krypterad) men det visas inte upp för kunden alls. Gör om, gör rätt.
OK-Q8 Bank och Danske Bank visar inte något som helst hänglås i webbläsaren. Anledningen är att inloggningssidan hämtar bilder över en helt okrypterad uppkoppling. Det gör att man väldigt lätt kan se att du loggar in på banken. Danske Bank döljer dessutom denna miss genom att visa inloggningssidan i en popupruta. Riktigt illa.

Bankerna du borde undvika (F):#

Dags för bottennoteringarna. Det gemensamma för dessa banker är att dom har allvarliga problem med sin kryptering. Du kan därför inte kan lita på att trafiken till deras sajter är ordentligt krypterad.

Inga banker ligger på den här positionen längre. Bra jobbat av alla fem!

Slutligen#

Det finns väldigt många andra kriterier att sätta betyg på, men jag nöjer mig med detta denna gång. Kanske finns det någon annan som vill fortsätta jämförelsen på sin egen blogg?

Vill du diskutera resultaten vidare finns jag på Twitter: @EmilStenstrom

8 insights from using the Narrative clip

2014-02-22T16:42:03+01:00

For about three weeks, I have been a beta tester and then a regular user of Narrative; a wearable camera that is always on, and that takes two pictures a minute. I thought I'd take some time and share my insights from that experience, and the reason why decided to stop using it.

2014 will be a year of wearable tech and Narrative is clearly a front-runner. It's always on, and nicely captures things around you with iPhone photo quality. At the end of the day you connect it to your computer, which uploads the images to private, encrypted, image storage, that you can access through the Narrative app on your phone. A clever clustering algorithm, based on both GPS data and image similarity, clusters the images to make them easier to browse. It works great, and lives up to the promise of capturing your day as it is.

So, insights, what did I learn from this experiment?

Asking for permission. Narrative takes asking for permission to its limits, since it's always taking pictures. Because of that, I think it follows the same rules as taking a photo with your phone (people tend to think of them as two different things, I don't). Just like you ask permission before taking photos with your phone, you need permission before wearing the Narrative clip on around someone. This was my stance from the very beginning. Not everyone agrees with this, more on that below.

Taking pictures vs. Sharing pictures. Giving someone permission to take pictures, does not mean you automatically get permission to share them. Narrative takes pictures all the time, but they are are not meant to be shared. They are put in the personal diary that is the Narrative app, and are only viewable by one person. Getting permission to wear it while meeting your friends, does not mean you can share the pictures with other people, without getting their permission again. While this might be obvious for the Narrative camera, I think it applies to mobile photos too. "Is it OK if I take a picture of you?" is not the same as "Is it OK if I share a picture of you with my friends?". If you miss this distinction, you will offend people.

Everyday life. If you are an active Facebook or Instagram user and look at your profile, you will see a very "directed" and prettified version of your life. This "fixed version" might match your experience well (doing new things means more vivid memories), but it does NOT match how your life looks when viewed through Narrative's stream of photos. I do programming, and ~6-8 hours of every workday I look at a computer screen. The photos from Narrative makes this painstakingly clear. This is a very different image from the one I get from looking at Facebook or Instagram, where the only picture shared might be the great theatre performance I watched later that night. This is of course obvious when you think about it, but my guess is that most people don't think about it. I definitely did during these weeks. Facebook and Instagram fundamentally change how we remember things, by reminding us of all the cool stuff we do, and forgetting everyday life.

Everyday life of your friends. The difference between your social profiles and your Narrative photo stream of course applies to other people as well. When you look at your friends' social profiles, know that what you see very different from their everyday life. We live in a time where we focus too much of our attention on other peoples experiences.

Asking for permission is uncomfortable. No matter how strict you are in your policy about sharing the pictures you take, there will be a couple of uncomfortable minutes of explaining how the camera works to new people. People assume that the photos are shared automatically. They assume that you just asked for permission on a whim, and could just as easily forgotten about it. This of course goes away with some explaining, but it's a very bad place to be with people you don't know that well. As someone that takes pride in trying to meet new people often, this is a severe blow to the experience using Narrative. A blow hard enough that I decided the camera wasn't for me.

Not everyone will ask for permission. Not everyone using Narrative will ask for permission first. I've met people that decided not to, arguing that if you have showed something to them, it makes sense that you allow them to see it again. I understand this line of reasoning, but I think the problem really is with allowing a digital file is where the problem lies. Snapchat raised the bar on what kind of pictures you could send to your friends, but deleting them after a set amount of seconds. The Narrative images are permanent files. Even though they are not shared, there's the potential of them being shared, and that's bad enough.

So I don't agree with this line of reasoning, but there will be no way of stopping these people from taking pictures of you anyway, and potentially sharing them with their friends. Personally, I think I'm fine with the "taking pictures" part, especially if it's in a public setting: a subway station, a mingle event, or a large party. If it's a private setting with fewer people, I would like to be asked. If I trust the person, I would most often be OK with having pictures of me in a private setting too, but again, I would like to be asked when that person wanted to share the picture with other people. These are my own thoughts, and I don't assume yours to be the same.

My thinking around this means I won't have a problem with other people not asking for permission, since this will happen mostly in public settings. In more private settings I will have to talk to them, and understand their policy around sharing the pictures with friends.

Again: No matter what set of rules you think people should follow when taking pictures of you, your policy when taking pictures yourself should be at least as strict. If you don't expect people to take photos of you using a Narrative camera, don't take photos of them using your mobile camera without asking first. And ask again before sharing.

Different forms of sharing. Sharing does not necessarily mean sharing on Facebook or Instagram. It could just as easily mean showing someone your phone when having the Narrative app open. It could mean having the file on your computer and then flipping past it when looking for other stuff to show your friends. It could mean inadvertent sharing when you get hacked, get your computer or phone held as evidence in a police investigation, or your government spying on you (NSA).

Sharing is tricky, and I've heard people reasoning around it in terms of what group you share to. Some people are OK with government spying since only a few analysts will see their stuff. Other people are OK with you sharing a picture on Facebook if they know you have reasonable privacy settings. Your mileage may vary, but I wouldn't be surprised if the answer to "Can I share this?" will be highly dependent on the group of people that you share to. Now, the people you share to, and the people that are likely to see something you share, are a much smaller group (EdgeRank), and this makes it even more complicated to value how much privacy you are giving away by allowing someone to share something.

Sound and video. Another common question I've gotten was if I'm recording sound and video. The answer is no, and I think this is a good thing. I hadn't thought of this before, but sound is much more intimate than pictures are. Sound is closer to your thoughts, and sharing someone saying something stupid is much more revealing than a picture of them looking stupid. Pictures rarely portray thoughts. Sound raises the bar considerably, and would make all the interaction around the clip even more difficult. I'm not sure people asking for sound and video support to be added to Narrative have considered the social aspects properly.

***

And that's it, my thoughts around the Narrative clip and what I've learnt from it. I hope you have learnt something too. Don't take this as a reason not to try the camera, it's very opinionated, and you might handle the social aspects better than I did.

Feel free to leave a comment if something needs clarification or if I can answer any questions about my experience.

My text editor: Sublime Text

2012-07-07T23:53:03+02:00

What text editor you use when coding is a very serious subject. I'll hardly be able to talk about what I prefer, without some people (other than you, of course ;) taking it as me trying to steal their editor from them. I'm not going to. I'm just going to tell you what I think, and you can just decide of you agree or not. My hope is that it helps you be more certain about your own choice of editor.

What I look for in a text editor:

Support both Mac and PC. I use a Mac for work, and a PC at home, and I want to be able to use the editor skills I've learned at work for my hobby projects, and vice versa.
Less than 3 seconds starting time. Big projects means lots of little files to open. So while programming I will open files and close files all the time. Each delay waiting for the editor to open, or a file to load, means hundreds of seconds over time.
Syntax highlighting for web languages. I work with web stuff, both front end and back end, and I need colors to be able to scan files quickly. I don't really care about the exact ones, light or dark, I just want it to look decent. What "web languages"? HTML, CSS, Javascript, JSON, Python (Django), Ruby (Rails), PHP, SQL.
Handle big files (< 1 Gb). From time to time I work with big files. Can be database dumps or large text files. I'm not expecting < 3 sec to open them, but at least < 15 sec, and then be able to work with the file without the editor lagging.
Some kind of file browsing/project support. There's never just one file and there's needs to be a way to overview all the files in a certain directory structure, and pick from them. I'd like there both to be file searching support and a way to click to a file through the directory structure.
Code auto-completion for web languages. Do I use font-variant, font-style, or font-decoration to set italic? I want my editor to show me a list of options, and let me pick. This is harder to do well for the dynamic server-side languages, but I still want it.
Look good. I don't want something that looks like the terminal, and want something beautiful. For me this means gradients, nice rounded tabs to click on, a nice font, a color scheme that matches.

With that list of priorities in mind, let's look at a couple of popular editors:

Editors that are not available both for Mac and PC: Textmate, Coda, Notepad++, and Visual Studio.
Editors that are slow: XCode, Eclipse, NetBeans, Dreamweaver, and most other editors built with Java.
Editors that don't handle syntax highlighting and code completion: Notepad, Wordpad.
Editors that don't handle big files well: Komodo Edit (what I used til recently).
Editors that look like shit: Emacs, Vim.

And then there's Sublime Text. Which does all of the above, perfectly. You should try it.

[The Listserve] I’m sorry to break this to you

2012-04-19T13:02:33+02:00

I just got the fantastic chance to send an e-mail to ~10.000 people. What would you write if you got that chance? This is what I wrote:

"I'm sorry to break this to you, in an e-mail from a random stranger like this, but it needs to be said: Most of your life won't be fantastic. I'm not joking. The adventures you'll tell your children about will be a minuscule part of it. So if you want to avoid the feeling of utter disappointment as you grow older, you need to accept that fact. Sorry.

This leads us to the insight: You should focus more on the non-fantastic parts. The parts where you eat breakfast, walk to the bus, have a boring day at work, eat your ordinary lunch, shop groceries, and brush your teeth. After all, this is the major part of your life, and neglecting it is a wasted opportunity.

Here's the thing: Most of the boring stuff in your life is so dull, that even the tiniest thing can make it seem fun. The tiniest thing. This means the you could make it better with extremely simple means.

What exactly am I talking about here? Little things. Like these:

Challenges: When brushing your teeth tonight, use your left hand.

Mind games: When you enter work (or school!), imagine the sound "Kabaaaam!" as you enter, as if your presence changed the whole room.

Action: Jump down from the side-walk, instead of just stepping down.

Changes: Buy some fancy tomato sauce tonight, instead of your usual brand.

More action: Count the number of pink things on your way to work, as if your life depended on it.

See? Easy stuff. I really try to live by this "Everyday Action" idea, and I think it works for making the boring parts of life more fun. Because that's the thing: just because the fantastic moments are few, there's no reason to just sit there, waiting for the next big thing to swipe you off your feet. Have some fun meanwhile, it's easy…"

What movies on Piratebay will you like the most?

2012-01-08T22:07:29+01:00

Christmas, and the weeks thereafter, are times for coding. And I've been playing around with piratebay and filmtipset (a Swedish movie recommendation) a little bit. I just pushed it to the filmtipset-piratebay project on GitHub, if you want to take a look.

CSS for screen scraping#

The script is using CSS for screen scraping; something that works extremely well:

from lxml import html
import requests
response = requests.get("http://thepiratebay.org/browse/207/0/7")
document = html.document_fromstring(response.content)
links = document.cssselect(".detLink")
print [link.text_content() for link in links]

Note: You need lxml and requests to run the above example.

Saving the above snippet to a py-file and running it will give you a list of all torrents on the given url. Play around with the CSS selector to get some other data from the page.

Extracting movie titles from torrent names#

It's surprisingly easy to convert torrent names to movie titles. Just follow this simple algorithm:

Split the torrent name into words by treating all non-alphanumeric characters as space.
Loop over the remaining words, and look for a predefined set of "torrent endings".
When you find an ending, cut the name from there
(Optional) Remove the year if there is one at the end of the remaining string
(Optional) Remove all movies which really are bundles of movies, and not single movies. This is easily done by looking for a set of common strongs such as "trilogy" and "series"

Result: "Real.Steel.2011.720p.BluRay.x264-REFiNED" -> "Real Steel"

You can find my movie title finder implementation in parse.py on GitHub.

Cache all HTTP Request#

To both save time, and be nice to the services we're querying, the script caches all HTTP requests for a number of days. I do this by simply saving the returned HTML/JSON to a file, and checking the file system for that file before making a new request. Saving the HTML/JSON, and not the processed result, makes it possible to experiment with the parsing, without having to wait for new requests from the server.

My caching implementation is of course also on GitHub.

***

All and all, this has been a fun little project, and I've learned a lot. But I'm sure we can make this even better. Feel free to send pull requests!

Partial XMLHttpRequest responses?

2012-01-08T15:28:49+01:00

We all know how to make an AJAX request, and fetch some data. But as soon as you need to fetch data incrementally, have the server push data to you, you have to resort to all sorts of complicated stuff. Websockets; with all their different versions and shady support, different kinds of polling, hidden iframes, ActiveX for IE?

The simplest way, that almost workds is partial XMLHttpRequest responses. I first read about them as progressive xmlhttprequests on Kyle Schulz blog, but really think that method should get more recognition.

Note: I've only tested this with Webkit, against Twitter's Streaming API, with a XMLHttpRequest that allows cross-domain requests. I think it works with Firefox too, but it will definitely not work in IE. Sorry.

var xhr = new XMLHttpRequest();
var url = "<streaming-url-on-you-own-domain-or-CORS>";
xhr.open("GET", url, true);
xhr.send();
// Define a method to parse the partial response chunk by chunk
var last_index = 0;
function parse() {
    var curr_index = xhr.responseText.length;
    if (last_index == curr_index) return; // No new data
    var s = xhr.responseText.substring(last_index, curr_index);
    last_index = curr_index;
    console.log(s);
}
// Check for new content every 5 seconds
var interval = setInterval(parse, 5000);
// Abort after 25 seconds
setTimeout(function() {
    clearInterval(interval);
    parse();
    xhr.abort();
}, 25000);

The biggest problem with this method is that the responseText property keeps filling up with data. The longer you receive data, the bigger the data in memory will be. The only way I can see this fixed (today) is to simply kill the connection after a certain amount of data has been received, and open it up again.

I would love to see a better way to do this, from native javascript, without all the numerous hacks that are out there. If you know of a way that fills these requirements, please let me know:

Easy to implement on the client side. Ideally I would like to use XMLHttpRequest, and just get a callback each time the client sends data, with the NEW data specified as a callback parameter.
Easy to implement on the server-side. I can set some headers if you make me, but ideally I would like to use this against existing Streaming APIs (like Twitter's), without adding custom stuff.
As cross-browser, cross-platform as possible.

Is there a way to get this working? It's so annoying to see something that's a curl one-liner, be 100s of lines of code with web technologies…

curl https://stream.twitter.com/1/statuses/filter.json?track=<your-keyword> -u <your-twitter-nick>

Is the web really that far behind?

Suggestions for TweetDeck

2011-07-22T12:35:04+02:00

To be an effective twitter user you really need a permanent window on your desktop, that shows you what's happening right now. I've been using TweetDeck for a long time now, with short detours into Seesmic and various smaller clients. TweetDeck is the best client I've found for my needs, but its interface has lots of little bugs and inconsistencies; small things that together add up to a bad user experience.

I hope someone from TweetDeck reads this, and can start working on these (simple) issues. I really think they would improve the general user experience a lot.

Compose pane should close after a tweet as been sent. I've looked over the shoulder of lots of people using TweetDeck, and the most common pattern is to click the reply button on someone's avatar to write a new message. If writing multiple messages directly after eachother was more common than writing them one by one, then it should stay open, but that's not the case.
Tweets should be as compact height-wise as they can be. All tweets in TweetDeck are displayed as if they where full 140 chars. This means that valuable screen space is lost, something TweetDeck pushes one of its advantages over the official Twitter client or the web. "More Tweets per screen!". What's worse, having tweets aligned horizontally over several columns makes it look like there's some connection between then. Why do they align my third timeline tweet with my third mention? No idea.
All retweets should be shown, not only the last one. Retweets are a very important part of Twitter. By seeing who retweets you, you learn who your active followers are, and maybe even find people that are interesting for you to follow. But TweetDeck does not follow the lead of the official client that shows "Retweeted by [user] and 3 others", instead they show just the last one.
The buttons at the bottom of each column should be hidden. The bottom buttons of each column are for things that people rarely use, but they still are allowed to permanently take of screen estate. Instead they should be hidden behind one button, and put at the top of the window with the column title. This would leave even more space for tweets to be shown.
The "Quick jump to column" buttons at the bottom of the window should be removed. Having clickable icons that set the scroll offset is a very strange interface concept. I understand the need to jump between columns when you have 10-15 columns, but most people don't have that. Letting an interface element that is used seldom by most people take up that much space is just a waste. It users with that many columns is an important part of the target audience, use tabbed workspaces instead. Each workspace could show up as a tab at the top of the window, and each workspace could have its own set of columns. Until that's done, remove the clunky icons at the bottom and free more space for tweets.

Here's a quick sketch of how TweetDeck looks before any changes:

… and here's the same view *after* the five interface improvements have been implemented:

These are changes I really would like to see in TweetDeck, what do you think?

Fixing Microsoft’s bad reputation

2011-06-19T14:07:32+02:00

Microsoft has continuously failed at getting people in the tech crowd to like them. This is a growing problem for them, and something they need to start taking seriously. To understand how to turn this around, let's start a decade ago, with Slashdot.

Slashdot has always been one of the pillars of Microsoft-negative news. They have a whole category on Microsoft (And the others have lots of articles too: "Microsoft" articles on Digg, "Microsoft" articles on Reddit, "Microsoft" articles on Hacker News). With a few exceptions, articles (and comments) are about Microsoft using their monopoly to crush smaller businesses, how their technology is inferior to what the open source world creates, on them creating data lock-in where users are unable to switch away from them, and so on. Slashdot, Digg, Reddit, Hacker News together have millions of daily users, who place as much faith in them as others do in their morning newspaper.

But these sites are only read by disgruntled teens right? No. Hacker news users are ~26, Reddit users are ~24, Digg users are 35-44… When asked: "What computer do you want?" These users answer: "A Mac (or Linux) laptop". When asked: "What software? They answer: "Google Docs (or LibreOffice/OpenOffice)". All those individual choices are starting to pile up, and have far-reaching consequences for Microsoft: Windows and Office licenses are the major part of Microsoft's income. Choosing a Mac or Linux laptop means no Microsoft Windows, choosing Google Docs or LibreOffice/OpenOffice means no Microsoft Office.

Tech users are a major force inside real companies, and not something Microsoft can just ignore. What about beginners and business users? They are both heavily influenced by by tech users. Beginners because they ask others for advise before buying their computers and software. Business users are instead controlled by their IT departments, which in turn try to find the most knowledgeable tech people to work for them. The small hold Microsoft still have on some IT departments is slowly shifted away from them when employees get empowered to choose their own equipment, moving them from business users to beginners in their purchase patterns. The tech crowd has more influence than you'd think.

Question is: where do you start fixing all this distrust? There's only one way: you start talking to tech users on their own terms. Here's how I would do it:

Hire some good of community managers. Their job will not be to market Microsoft, but to be the communication channel that can aggregate community opinions to Microsoft. The more rooted they already are in their respective communities the better; that makes it easier to start get the discussion started at a respectful level. You don't yell "Microsoft sucks!" at someone you respect.
Start by monitoring news about Microsoft and send monthly reports to managers inside Microsoft. Couple each news item with an approximate number of users that read them. This will paint an image of how tech users see the Microsoft brand. Pretty soon they will want to change that image.
Use community managers to ask for opinions on what to do next. Ask for small things that are likely easy to get done, and make sure to manage expectations right away. Windows 8 won't be open sourced :)
Have a small engineering team whose sole purpose is to make the suggestions from the tech crowd happen. They need to be cross-disciplinary, be well connected across the company, and have mandate from high up the organization. Employing this team is a small cost compared to other forms of marketing. Feed back any progress they make back as articles where they fit.
Lastly: Use insights from community managers to create marketing campaigns directly aimed at the tech crowd. A good community manager can have a pretty good guess at what work and what will. Many of the things will not only be marketing, but also require engineering, but there already is a team for that.

Working through that list will make Microsoft slowly earn their trust back from the tech crowd. It won't happen overnight; you don’t reverse 10 years of silence that fast. But I think this is doable, reasonable, and something that really could work.

Would you be willing to be a community manager for Microsoft? What would you suggestion Microsoft did to please the tech crowd? I'd love to hear your opinions.

Geolocation and Google maps

2011-06-18T14:11:11+02:00

Image by heiwa4126 via Flickr

Google Maps has has geolocation support for a long time, but I still find people surprised of how it all works. So here's a short writeup, skip it if you already know all about geolocation.

Lets start at the Google Maps frontpage. Among the zoom controls, above the little old man, there's a button in the form of a small circle. It does not look like much, but what it's doing is incredibly cool. Click it and the map moves to where you are, with an accuracy of ~20 meters. Sci-fi crazy!

How does it work? In the newer browsers (all but IE6, IE7, or IE8) may ask you for your positioning information from the browser. It usually shows up as a bar at the top of the browser. The browser then gathers two specific forms of positioning information from your computer: your IP address and the signal strength of any wireless network near you. That information is then sent, if you approve it, to Google, which returns the coordinates you are at the moment.

Google can do this because they've been driving around with their street view cars all over the world, and measured signal strength of WiFi networks (public and private) at each specific location. Now they can use that information, combined with your local signal strength conditions, to triangulate their way to where you are. And it's both accurate and fast.

If your wireless reciever is turned off, or you're at a stationary computer, all calculations are based on the IP number. These kind of lookups are quite arbitrary and inaccurate, I just get to the nearest big city when trying to use it over a non-wireless line. But mobile connections are slowly taking over landlines, so I guess this problem will solve itself automatically.

The best kind of technology throws you forward in time, and make your realize that we're actually making progress. Geolocation has feel to it.

Looking for the techy explaination of geolocation? Have a look at Robert Nyman's writeup.

Animate from one element to another (jQuery plugin)

2011-05-05T22:52:14+02:00

Image via Wikipedia

Have you even tried clicking an "Add to cart"-button and not understood what happened? I have. An although I understand the idea of adding a product to the cart, and then letting the user continue browsing from where he is, I still get stumped when "nothing happens" when I click the button. So what to do?

Simple: Add a animation from the add button to the cart. That way you communicate what just happened. "The product moved in there, and by clicking the cart you'll find it again. Now continue buying stuff!".

Somebody must have done this before, so I started looking for a jQuery plugin to do this (jQuery was already on the page, so why not?). I found add2cart - A plugin that looks good, but that misses a few features I wanted:

The animation duration is set in seconds, meaning products further down the page will move faster than those further up. I wanted constant speed.
It didn't allow me to customize the look of the animated element.
The code rely on concatenating strings of CSS and generally could use lots of improvement.

So I did what anyone would do: rewrote the code from scratch, and posted it on GitHub.

Get the code: Animate From To 1.0

It's my first jQuery plugin ever, and my first public GitHub project, so let me know if I've done something wrong. Is this something that could be useful in one of your projects?

Will newspapers die?

2011-02-09T00:41:17+01:00

A journalism student in Toronto, Canada, asked me some questions via e-mail about my old article about newspapers and online reading. Instead of just sending an e-mail out in the void I thought I'd persist my answers here:

Do you think it would be best for newspaper publications to get rid of their print and go strictly online only? Why or why not?

I think they should keep their print papers for a while longer. It's a simple question of profit and catering to what users want. If you have thousands of users that want their paper, and it's economically justifiable, just keep it.

The tricky part is determining how much loss in paper subscribers you should tolerate before shutting down the huge printing presses. I have no good answer to that other than to know your numbers. Paper profit needs to cover paper printing and distribution. Journalism costs can be split between channels. Do the math.

Many people believe newspapers are dying and eventually will no longer exist. What are your thoughts on this?

I think they will persist, just not their paper versions. Many of them are looking at dwindling subscription numbers, year after year, and are trying to figure out what to do. How do you move to the digital worlds while keeping quality, relevance, and profit.

The iPad is one way to get people to pay for news again, and some newspapers are seeing success doing that. Over a longer haul just the iPad won't be enough, you need to be able to deliver your news to wherever your customers want them. Generally, this means being able to get your content on all kinds of devices, both mobile and desktop. The only format that can really do that, and still deliver a somewhat consistent design across devices, is HTML and CSS. Building one native app for each platform is just not worth it.

So it's simple really: newspapers that figure out how to transition to digital (while still getting paid enough to afford quality journalism) will persist, all others will (slowly) die. This means an end to both paper-only newspapers, and those that fail to find the right payment model.

HTML5 is not an option

2010-11-03T20:20:42+01:00

HTML5 is HOT! Developers all over the world are adapting their sites, browsers are catching up, and new fallback solutions are released every day. But many developers misunderstand one thing:

You can't choose to use HTML5 or not, your site will be parsed as HTML5 no matter what.

The reason is simple, HTML5 is made to be backwards compatible with the current web, so browsers don't need to keep their current parsers. All of them have soon switched to HTML5 parsers. You want to continue using HTML4? Not possible.

Now. You can choose whether you want to use the new features or not: New semantic tags, Microdata attributes, new form field types, accessibility features, 10-15 new JavaScript API:s (depending on how you count). Lots of new interesting stuff to learn.

So, go read up on HTML5 if you haven't already, but don't think you can keep using HTML4. Your site is being switched over as we speak.

Lazy Loading Asyncronous Javascript

2010-05-08T00:15:06+02:00

Update: This is no longer the best way to load scripts. Use a script tag with async and defer set instead.

Like many of you might know, I'm working on a site called Kundo with a couple of friends. It's kinda like a Swedish version of Getsatisfaction, which means we have a javascript snippet that people add to their site to get feedback functionality. Cut-and-paste instead of writing the code yourself. Simple.

The problem is, how do you load an external javascript with minimal impact on your customer's sites? Here are my requirements:

Small. I don't want a big mess for them to include on their sites. 10-15 lines, tops.
Stand-alone. The environment is unknown, so we can't rely on any external dependencies, like javascript libraries.
Cross-browser. I have no idea what browsers my customer's customers have, so I can't do anything modern or fancy that isn't backwards compatible. I assume at least IE6 and up though.
Asynchronous download. The download of my script should not block the download of any script on their sites.
Lazy Loading. If my site is temporarily slow, I don't want to block the onload event from triggering until after our site responds.
Preserve events. Any events used should not override any events on the customer's site. Minimal impact, like I said.
Don't pollute namespace. Global variables should be avoided, since they could conflict with existing javascript.

Note: I did not make all of this up myself. Lots of people did, I'm just writing it down for you. Thanks: Jonatan, Steven, Peter, and Linus.

Script tag#

<script src="http://yourdomain.com/script.js"></script>

While being the stand-alone, cross-browser, and the shortest piece of code possible; it doesn't download asynchronously and doesn't lazy load. Fail.

Screenshot from Firebug's net console: The script (set to load in 2 seconds) blocks the download of the big image (added after the above script tag, and used throughout this article as a test). Onload event (the red line) triggers after 2.46 seconds.

Async pattern (A script tag written with javascript)#

Steve Souders, the web performance guru, has compiled a decision tree over different ways to achieve non-blocking downloads. Have a look at that graph.

Since we're on a different domain, and only have one script (order doesn't matter), the solution is given: We should create a script tag with inline javascript, and append it to the document. Voila! Non-blocking download!

(function() {
    var s = document.createElement('script');
    s.type = 'text/javascript';
    s.async = true;
    s.src = 'http://yourdomain.com/script.js';
    var x = document.getElementsByTagName('script')[0];
    x.parentNode.insertBefore(s, x);
})();

Note: async is a HTML5 attribute, doing exactly what we're trying to simulate with our hack, so it's added for good measure. Also, wrapping the code in an anonymous function prevents any variables to leak out to the rest of the document.

This is a pattern that is getting more and more popular nowadays, especially since Google Analytics uses it. But there's an important distinction here: The above snipped blocks onload from triggering until the referenced script is fully loaded. Fail.

Update 2010-09-01: Steve Souders adds that the above is only true for Firefox, Chrome, and Safari, but not IE and Opera. So for a IE-only site, this might be the best method.

Screenshot from Firebug's net console: The script (set to load in 2 seconds) downloads in parallell with the big image. Onload event (the red line) triggers after 2.02 seconds.

Lazy load pattern (Async pattern triggered onload)#

So, how to you make sure you don't block onload? Well, you wrap your code inside a function that's called on load. When the onload event triggers, you know you haven't blocked it.

window.onload = function() {
    var s = document.createElement('script');
    s.type = 'text/javascript';
    s.async = true;
    s.src = 'http://yourdomain.com/script.js';
    var x = document.getElementsByTagName('script')[0];
    x.parentNode.insertBefore(s, x);
}

This works, but it overrides the onload event of the site that uses the script. This could be OK in some cases, where you have control over the site referencing the script, but I need to cater for that. Fail.

Unobtrusive lazy load pattern#

The logical solution to the above problem is to use an incarnation of addEvent. addEvent is simply a common name for an cross browser way to take the current function tied to onload, add it to a queue, add your function to the queue, and tie the queue to the onload event. So which version of addEvent should we use?

There's been competitions for writing a short and compact version of addEvent, and the winner of that competition was John Resig, with this little beauty:

function addEvent(obj, type, fn)  {
  if (obj.attachEvent) {
    obj['e'+type+fn] = fn;
    obj[type+fn] = function(){obj['e'+type+fn](window.event);}
    obj.attachEvent('on'+type, obj[type+fn]);
  } else
    obj.addEventListener(type, fn, false);
}

Note: This is unsafe code, since it relies on serializing a function to a string, something that Opera mobile browsers have disabled.

Thing is, we don't need all that generic event stuff, we're only dealing with onload here. So if we first replace the type attribute with hardcoded 'load', replace obj with 'window', and remove the fix for making 'this' work in IE, we've got four lines of code left. Let's combine this with the above lazy load pattern:

(function() {
    function async_load(){
        var s = document.createElement('script');
        s.type = 'text/javascript';
        s.async = true;
        s.src = 'http://yourdomain.com/script.js';
        var x = document.getElementsByTagName('script')[0];
        x.parentNode.insertBefore(s, x);
    }
    if (window.attachEvent)
        window.attachEvent('onload', async_load);
    else
        window.addEventListener('load', async_load, false);
})();

This is exactly what we're looking for here. Finally!

Screenshot from Firebug's net console: The script (set to load in 2 seconds) downloads after the onload event has triggered. Onload event (the red line) triggers after 0.41 seconds.

And that wraps up this article: Different tactics works for different scenarios, and only understanding them all makes you capable of picking the right one for your problem. As always, I'm waiting for your feedback in the comments. Thanks for reading!

My iPad – a short review

2010-04-25T20:33:32+02:00

About two weeks ago, I got my hands on an iPad. For those of you who have been living under a rock for the last months, and iPad is something that looks like a big iPhone, but behaves much like a small laptop.

Since people who just spent over $500 for a toy, are very subjective in their judging (buying something bad would make you look stupid!), you have to take this review for what it is: Me trying to justify buying an expensive toy. To help my mind balance things a bit, I'm going to talk about two good things, and two bad things about the iPad.

Good: Looks, UI and surfing the web#

[](/files/post-media/ipad_and_banana.jpg)

An iPad, here shown next to a old banana for comparison. Click for bigger image.

The area where the iPad really shines is when it comes to the look and feel. It's simply gorgeous, feels good in your hand, and makes all your friends instantly drool (like you expect them to). The screen is as sharp (or sharper) than a computer monitor, and its brightness could light up a dark room.

Working with it is just as pleasant. You click, swipe, pinch and type, like you've done nothing nothing else in your life. As someone who haven't used an iPhone before, I'm impressed. Those that have, say that the iPad is even snappier than the iPhone. Some application are of course better fit for a touch interface than others. Google Maps is one of them, when a large display combined with panning and zooming makes the old arrow and zoom buttons on the traditional Google Maps interface feel like stone-age.

They keyboard surprised me by being even better than I expected. In landscape mode it's almost as large as a laptop keyboard, and in some cases even faster than a traditional keyboard since you don't have to push, just touch the buttons. The biggest keyboard problem though is of course that you usually only have one hand available. When you have two hands, it's great.

Another area where the iPad really works is as a window to the web. Safari comes pre-installed (Microsoft monopoly anyone?) but works like a charm. Pages load quickly, and swiping to scroll and pinch to zoom works just like it should.

Web standards are well supported, like you expect from Safari. But some new problems emerge when looking at HTML5 demos though:

The Safari inside the iPad isn't as performant as a desktop browser. So there's lots of lagging when trying to rotate things in advanced ways.
Even though Apple has decided to ban Flash and pick a fight with Adobe, many new apps are built to take advantage of the mouse moving around. On the iPad, there's no mouse pointer, so you have to simulate one by touching the screen and moving the finger around.
Keyboard navigation is strange, because most of the time, you don't have a keyboard. The keyboard only shows up only when you focus a text field, at other times, you're lost.

These things sound like annoyances, and they are, but they are also quite minor. For all the surfing you usually do, the iPad works just like it should.

Bad: A horrible Developer Platform#

As you've probably heard during the last few weeks, Apple have no clue whatsoever about how to cultivate a vibrant developer community. Let's look at the facts:

To distribute your app you need to join the iPhone Developer Program, that costs you $99-$299 per year.
To download the SDK you need to fill out a huge multi-step form (where almost all fields are required), and where you're asked to give away lots of personal information.
When you "publish" your app it has to go through a manual approval process, which could take months if you are unlucky.
You need to approve to a ridiculous legal statement preventing you from using some programming languages, and making calls to some APIs.

To quote:

"3.3.1 — Applications may only use Documented APIs in the manner prescribed by Apple and must not use or call any private APIs. Applications must be originally written in Objective-C, C, C++, or JavaScript as executed by the iPhone OS WebKit engine, and only code written in C, C++, and Objective-C may compile and directly link against the Documented APIs (e.g., Applications that link to Documented APIs through an intermediary translation or compatibility layer or tool are prohibited)."

Needless to say, I'm never going to be develop anything for the Apple Platform. Apple and its developers is much like one big company (with strict rules). The only difference is that instead of a salary the developers get to pay to work for them. Good for Apple, horrible for developers.

Good: Games that use the touch interface#

There are lots of good games that have started to take advantage of the big touch surface to make fun interaction styles possible.

Tap Tap Radiation builds on you tapping the screen in pace with the music playing. The place to hit move around, just like the markers that indicate where to tap.
Asphalt 5 is a beautiful racing game where you use the iPad as a steering wheel and tilt it back and fourth to steel. Since the accelerometer is so sensitive it works remarkably well.
N.O.V.A. - Near Orbit Vanguard Alliance is a first person shooter that uses gestures to control who you shoot and how. Advanced gestures makes it possible to throw grenades that fly around corners, and targeting multiple enemies at once.

Simply put, there are lots of fun games for the iPad, and the reason is experimentation with the touch interface. Combine this with several days of battery power, and you have many hours of fun ahead of you.

Bad: Downloading apps through iTunes#

[](/files/post-media/ipad_closeup.jpg)

A closer look at the iPad, still with the banana present. Click for bigger image.

Sadly, the only way to get apps legally into the iPad is through iTunes. Since my iPad was bought in the US, the App Store icon leads to a "Your request could not be completed" with no further info. So it seems that the App Store is only launched in the US. Silly me, that thought the web was an international thing.

Anyway, it is possible to get apps into the iPad without getting a fake american App Store account. You just use iTunes to download the apps to a desktop computer, and then sync your app to the iPad. It's easy.

Well, strike that, using the iTunes store is not easy. Not being a previous iPhone using I had thought that downloading apps from iTunes was like shopping on Amazon. IT'S NOT EVEN CLOSE. This is how Apple built iTunes:

Take one fully functional e-commerce web site.
Make it slow so that people like me that surf on a 100 Mbit/s fibre connection have to wait several seconds after each click.
Make sure you can only access the store through a special downloadable app, that requires you sign several legal agreements, and leave away personal information, before you even bought anything.
Remove the possibility to sort through listings, remove any information what the list is currently sorted by, and hide options to only show apps for the device they know you just connected (the iPad).
Tada! You've built iTunes!

Needless to say, each visit to the iTunes App Store is a bad experience, and no something you want to do often.

Summary#

Situations where I see myself using my iPad:

In bed, watching public television through Svtplay. They have a great app, that is mainly adapted for iPhone, but scales up when used on a bigger screen.
Casually surfing the web from the couch. This includes checking my e-mail (which is IMAP over SSL, which works fine), checking my feeds through Google Reader, checking Facebook through the iPhone app (which can be pixel-doubled to fit the screen nicely), and twitter through twitterific.
Surfing all kinds of maps, and wikipedia in some social context. If I ever get into an argument about where a country is exactly, I'm pull up my iPad instantly and check it.

And that concludes this iPad review.

[](/files/post-media/ipad_off_and_banana.jpg)

iPad rests here, next to banana.

As usual I'm looking forward to your thoughts in the comments. Thanks for reading!

Rendering a web page – step by step

2010-01-11T01:04:33+01:00

Have you ever thought about what happens when you surf the web? It's not as simple as it seems:

You type an URL into address bar in your preferred browser.
The browser parses the URL to find the protocol, host, port, and path.
It forms a HTTP request (that was most likely the protocol)
To reach the host, it first needs to translate the human readable host into an IP number, and it does this by doing a DNS lookup on the host
Then a socket needs to be opened from the user's computer to that IP number, on the port specified (most often port 80)
When a connection is open, the HTTP request is sent to the host
The host forwards the request to the server software (most often Apache) configured to listen on the specified port
The server inspects the request (most often only the path), and launches the server plugin needed to handle the request (corresponding to the server language you use, PHP, Java, .NET, Python?)
The plugin gets access to the full request, and starts to prepare a HTTP response.
To construct the response a database is (most likely) accessed. A database search is made, based on parameters in the path (or data) of the request
Data from the database, together with other information the plugin decides to add, is combined into a long string of text (probably HTML).
The plugin combines that data with some meta data (in the form of HTTP headers), and sends the HTTP response back to the browser.
The browser receives the response, and parses the HTML (which with 95% probability is broken) in the response
A DOM tree is built out of the broken HTML
New requests are made to the server for each new resource that is found in the HTML source (typically images, style sheets, and JavaScript files). Go back to step 3 and repeat for each resource.
Stylesheets are parsed, and the rendering information in each gets attached to the matching node in the DOM tree
Javascript is parsed and executed, and DOM nodes are moved and style information is updated accordingly
The browser renders the page on the screen according to the DOM tree and the style information for each node
You see the page on the screen
You get annoyed the whole process was too slow.

I, too, get annoyed when the above steps take longer than one tenth of a second. But now at least I have some documentation to look at, while waiting the remaining fractions of a second before the page renders.

Spoiled we are, all of us.

(Feel free to add more steps, through the comments…)

Position: fixed CSS templates

2009-12-31T15:14:26+01:00

In 2006 I wrote an article about simulating Frames and Iframes and from time to time, I get questions of how to make modifications to the templates presented. But one big thing has changed since 2006: Perfect support for IE6 is no longer mandatory.

So yesterday, when Brandon Cobb of Super Fighter Team asked about a design with a fixed header, fixed left column, and scrolling right column, I thought I'd renew my take on simulating frames with CSS (The original article is still good for background information, so I still recommend reading it).

Position: fixed CSS templates#

The idea is this: Instead of specifying what parts of the page should scroll, we can specify what parts should stay fixed when scrolling. This makes it easier to deal with several fixed parts of the same layout, but also allows us do to things frames cannot do.

So what's the trick? Well, position: fixed does exactly what we want. It works just like position: absolute, but it stays still when the page is scrolled. So it's really just a matter of making sure things don't overlap.

Try resizing the page, and see how the scrolling works. The templates have been tested in: Fx 3.5, IE8, IE7, IE6 (see note below), Opera 10, Safari 4; all on Windows. Let me know if you can test it on more browsers, or find bugs.

You're of course free to use this templates as you see fit, with or without a link back, commercially or not. Consider it public domain.

Fixes for IE6#

As I've said, these designs don't act the same in IE6. Instead of some parts being fixed, the whole page scrolls in IE6. The good thing about this is that IE6 users won't see that your site is "broken", they will just get another design. And as the number of IE6 users goes towards zero, your design will be more and more consistent :). The fallback is very simple, IE6 gets position: absolute instead of fixed, using the !important hack (you should probably use conditional comments instead).

Hope these templates are useful for some of you!

Google support gone wrong

2009-10-24T15:00:28+02:00

Google really produces great software. I use many of them: Web Search, Picasa, Reader, Feedburner, Analytics, Images, Groups, Docs, Translate, Code, Chrome, Maps, Video, Blog Search, Youtube, AJAX API, Webmaster Central, and Site Search.

Just to name a few :)

Problem is, with many of the above, if something breaks you're out of luck. Because it's god damn impossible to get a hold of someone that you can talk to. Do you reach anyone at Google with your e-mails? Does anyone from Google read your forum posts? Highly unlikely.

Google is a great example of support gone wrong. I think the explanation is easy: Few of the programmers I know want to deal with support. They want to deal with coding! And since Google is a company of programmers, it doesn't want to do support.

Let me tell two stories:

Feedburner#

Feedburner once was a great company, with a simple but thought out service. They gave you subscriber statistics of your RSS feed. You just redirected your feed to them, and made all your subscribers sign up to their generated feed, and they did all the tricky calculations. Simple, efficient.

As a company Feedburner understood the value of being personal. Messages throughout the site stated "My feeds like a little namespace to call their own", "Sometimes your feed just wants to look good. Spruce it up in the following ways", "Oh dear, what kind of trouble?". Nice personal touch :) Feedburner developers where easy to get in touch with, an e-mail and you had a friendly, knowledgeable person to talk to.

Then Google came along and bought the whole thing. Everything was rewritten to Google's platform, domains where switched, and chaos ensued. Of course, during that time, Feedburner's previous support people totally vanished, and everyone was directed to the Feedburner Status Blog, when you could confirm they where not working on your problem. Just great.

For three weeks now, my subscriber stats have jumped from 800 to 3000, back an forth, several times per week. Nice, isn't it?

Feedburner is apparently broken, and the simple service of delivering feed statistics doesn't work.

So I post to their Google Group, the place where you should ask questions about the service. Three months later, several other users have reported having the same problem, but no reply from a Feedburner developer.

Now I'm thinking of switching. Anyone have a good alternative ready?

Google Wave#

Services that start at Google are even less likely, compared to acquired services like Feedburner, to have proper support. It's just not a priority, and there's far too many interesting programming challenges to deal with users.

So when Wave gives me an old picasa name as my username ([company name].|place of one of our conferences]@googlewave.com), I know there's very little chance that I'll get help from anywhere. The official channel to ask for help is their support forum, but it doesn't seem that they reply to issues there. My message there from two weeks ago stands on it's own, with no replies.

It's the proposed revolution that will replace e-mail, and I can't use it :(

Bad trend#

This trend of Google not offering even basic support from dedicated people is a unfortunate development, that should be taken seriously by executives at Google, and dealt with at a very high level.

Launching new web services is easy, improving a services based on feedback is much harder. Google has not yet managed to crack that nut.

Google Chrome as an Internet Explorer plugin

2009-09-23T16:15:49+02:00

I just encountered a quite mindboggling piece of news on the Google Wave blog. It seems they have decided not to support Internet Explorer at all. Not IE6, IE7 or IE8. Surprised? I sure was. What's even better is that to still give these users the opportunity to use Google Wave, they've built a plugin for Internet Explorer, that loads the Chrome browser with Google Wave inside it.

What? Load a browser as a plugin inside another one? I guess this is a real war after all? Who will build the first browser to load Firefox into Chrome? Internet Explorer inside Firefox?

Why? Well, now they don't have to care about Internet Explorer support at all. Felt that heavy weight lift from your shoulders? Suddenly large parts of HTML5, CSS3, and modern javascript, is available to them, with no backporting and performance tuning needed. Must be fabulous.

Now. Combine that with the possibility for the developer to choose when to trigger the different browsers. We're up for a whole new ride. You're invited.

SpriteMe – Combine images and get fewer HTTP requests

2009-09-19T17:09:02+02:00

Those of you that care about website performance have probably read about combining images, something that's called "CSS sprites". The idea is that by using the same (somewhat larger) image several times, you get fewer HTTP requests to your site, and therefore speed it up. Problem is, most of your images are CSS background images, that are positioned using clever background-positions and repeats.

Now, this makes to tricky to combine images. Something that repeats horizontally can't be combined with something that repeats vertically (unless there's transparency involved), and wide images can't be combined with narrow ones. So combining is usually tedious, manual work, both to combine the images and then calculate the new background-positions required.

Lucky for us then, that there is a tool called SpriteMe, available as a bookmarklet (a bookmark containing javascript), and as an excellent online demo that walks you through how it works. SpriteMe does all the hard work for you, you just have to copy and paste both the combined images, and the new background rules, to your site.

Decreasing the number of HTTP requests have never been simpler. Well done Steve Souders!

Voddler – movie streaming for the masses?

2009-08-31T00:12:46+02:00

I've recently managed to get a hold of a beta invite to Voddler, and thought I should tell you a little about my experience of it. But first, big thanks to Gunnar Bark (Tweets in Swedish, follow him!) who tipped me off about a Voddler invitation competition. It was run by Christian Rudolf at mjukvara.se (In Swedish, but definitely something to add to your feeds) who had 15 invites to give away.

Anyway.

Two years ago, I got to test Spotify, a music streaming that will soon launch in the US and is heavily anticipated. It completely changed how I listened to music, I deleted all my mp3:s and is now streaming all my music. Voddler is a similar streaming service, but for movies.

So, what do I think of it?

First of all, Voddler is in a very early phase of testing. It's what I would call a "real beta", where the client crashes frequently, subtitles sometimes lag behind, the interface is tricky to use, and several features just don't work.

Thing is, I'm quite certain Voddler will revolutionize how I watch movies, anyway. Because at the core of Voddler is HD-quality streaming, that just works. Believe me, I normally don't watch that many movies, but have seen four, ****in the last 48 hours. They are just a couple of clicks away, and start in 10 seconds (during which you'll watch a commercial). Here's some screenshots.

Now. I'm saying that the client is a beta, and that there are serious bugs left, what kind of bugs do I mean?

About half of the menu options don't work. "Tv series" lead nowhere, I can't browse movies by actor, director, studio, and several genres are empty. In my opinion, they would gain a lot by just removing those options until they lead somewhere! This is a no-brainer.
The client freezes and crashes regularly, and I have never been able to watch two movies in a row without restarting in between. This of course requires squashing bugs that are left in the code, and improve the testing procedure, so that such bugs never reach production code. Using mozilla-style reporting that allow people to send in crash reports is probably a good idea at this stage. The code is based on XBMC, and people say stuff like "how could they add that many bugs to a product that good?!". Voddler, don't make it that easy to dismiss your hard work!
Sometimes the subtitles skip some lines. I watched Antikiller, a russian action movie, five times important lines were shown only half a second on the screen, making me miss important pieces of the story. I'm not sure if this had something to do with the subtitles for that exact movie, or lag because of Voddler, but I know that it was a crappy first impression of the client. It just has to be fixed.
The support forum is so slow it's unusable. My guess is that it's on the same network as the movie streaming service, and therefore goes down together with that one. Those two just have to be independent, as you need the forum most when the other is down.
The UI doesn't work usability-wise. First of all, it's built for television sets, which means that only the keyboard works (arrow keys, space and esc), no mouse support. Several of the menues cycle (when you're on the end the first item appears again), so you'll easily loose track of where you are. There are small errors all over the place, things that makes finding and starting movies difficult.

So, in summary, the core of Voddler is good, but the interface needs quite a lot of work before it can be released to the public. No matter what, I'll keep following the development of this product!

In general, it seems that when you Americans stop us from using your servies (ie. Hulu), we Swedes build our own services. First Spotify, now Voddler.

Techniques to use when IE6 dies

2009-08-26T23:54:51+02:00

Everyone except Microsoft themselves are talking about the death of IE6. I've tried motivating people to drop support, arguing that you at least can show IE6 users a message. Many have replied with "but our IT department doesn't let us…", and I can say nothing more than that the IT department is filled with humans. Which means they are lazy, and upgrade when people whine enough about it. It's a shame it has to be that way, that we have to punish people for their IT departments, but that's how it has to be.

As web developers, we should be the ones complaining the loudest. We have so much to win on IE6:s death it's silly. Just look at the below list of things IE6 can't do, but IE7 can (the next worst browser):

Native PNG alpha transparency without silly hacks.
Several selectors: Child selector (">"), Adjacent sibling selector ("+"), Attribute selectors ([attr=value]), and General sibling selector ("~", CSS3)
Min-height, Max-height, Min-width, Max-width
Multiple class/id selectors in the same ruleset (ie. .class1.class2 { … })
:hover on all elements, not just links
position: fixed
Native XMLHTTP (Without ActiveX)
International Domain Names (IDN), the ability to use UTF-8 chars in domains
Page zoom that zooms the whole page (don't worry about zoom)

This is a huge step forward for us web developers. Huge! Bigger then when CSS3 comes out, because we won't be able to use that one for years. Someone is pushing the front of what's possible, I'm pushing for the front of what's usable.

What can you do to help me get everything in the above list? And, did I forget something?

Sources:

Friendly Bit - Web development blog

JustHTML 3.0.0: A new HTML5 parser architecture

What changed in 3.0.0#

How it works#

Why this is faster#

The comparison to other parsers#

What did not change#

JustHTML is now safe-by-default

Safe-by-default serialization#

Turning it off (trusted input only)#

Custom allowlist policies#

Why I built justhtml-xss-bench#

Threat model (what “safe” means)#

JustHTML: Addressing some questions

"This is a copy / this is derived work!"#

"He stripped the authorship / laundered the license!"#

"He doesn't understand the code"#

"It's not high quality code, because it won't be maintained"#

How I wrote JustHTML using coding agents

Why HTML5?#

Building the parser (iterations, restarts, and performance work)#

A one-shot HTML5 parser (as a baseline)#

Wiring up html5lib-tests (<1% pass rate)#

Iterating to ~30% coverage (refactors and bugfixes)#

Refactoring into per-tag handlers#

Reaching 100% test coverage (with better models)#

Benchmarking and discovering we were 3x slower#

Rewriting the tokenizer in Rust (and barely matching html5lib)#

Discovering html5ever (fast, correct, Rust)#

Asking: why build this at all?#

Pivoting to porting html5ever logic to Python#

Restarting from scratch (again)#

Still slower than html5lib#

Profiling, real-world benchmarks, and micro-optimizations#

Deleting untested code (coverage as a scalpel)#

Fuzzing to find crashes and harden the parser#

Comparing against other parsers (how rare 100% is)#

Shipping it as a library (CI, releases, selector API)#

What the agent did vs. what I did#

Practical tips for working with coding agents#

Was it worth it (and what “quickly” meant)?#

3 tips for agile roadmaps

1. Update it regularly#

2. Schedule broad areas/problems, not features#

3. Don’t make everything time bound#

How to prioritize a feature list, the hard way

The case for national digital currencies, not Bitcoin

Decentralization is not a good idea for a currency#

Proof of work makes things worse#

Ease of use for crime is not a good feature#

Another look at the El Salvador use of Bitcoin#

Requirements for an ideal digital currency#

A national digital currency#

In summary#

[The Listserve] How randomness affects your life

How to make e-mail encrypted for everyone

Encrypt as much as possible of the e-mail message#

Encrypt with public-key cryptography (without the hassle)#

Consequences for e-mail clients#

Consequences for e-mail providers#

In summary#

Hur säkra är svenska banker? (Swedish)

Uppdaterat:#

Bankerna med bäst säkerhet (A):#

Bankerna som nästan kom med på topplistan (B):#

Bankerna som du borde vara orolig för (C):#

Bankerna du borde undvika (F):#

Slutligen#

8 insights from using the Narrative clip

My text editor: Sublime Text

[The Listserve] I’m sorry to break this to you

What movies on Piratebay will you like the most?

CSS for screen scraping#

Extracting movie titles from torrent names#

Cache all HTTP Request#

Partial XMLHttpRequest responses?

Suggestions for TweetDeck

Fixing Microsoft’s bad reputation

Geolocation and Google maps

Animate from one element to another (jQuery plugin)

Why I built `justhtml-xss-bench`#

Wiring up `html5lib-tests` (<1% pass rate)#

Rewriting the tokenizer in Rust (and barely matching `html5lib`)#

Discovering `html5ever` (fast, correct, Rust)#

Pivoting to porting `html5ever` logic to Python#

Still slower than `html5lib`#