Wanted: universal element separator symbol; successful applicant must fix CSV too

Mathematical coordinate notation is a mess, but it could be be fixed now that Unicode is close enough to universal. While we’re at it, we can fix the CSV data format too.

The root of the problem is the comma, that unassuming but highly useful perky little squiggle of a punctuation mark, which unfortunately has been overloaded with too many globally conflicting meanings over the centuries. Were the comma only used in text, we could eliminate one of the most persistent headaches in software internationalization.

How do you write a list of values that represent coordinates? The traditional way is to put parentheses around the list and separate each value by a comma. For example, here’s a 3D coordinate:

(4, 3, 8)

But what if the values are decimal numbers, like this?

(12.7, 33.912, 81.07)

Here we encounter the major issue with the comma: in many countries it’s used as the decimal separator. These include not just Estonia and Insignificantistan, but major countries like Germany, France and Brazil. In handwriting, you might get away with using commas for both decimals and coordinate separators, but on a computer that kind of symbolic overlap won’t fly… So the usual recommendation is to use the semicolon:

(12,7; 33,912; 81,07)

This starts to look quite unpleasant, with all those visually heavy punctuation marks surrounding the actual data. Worse than that, the inconsistency of these notational systems makes it difficult to write software that works with coordinates and other numeric element lists represented as plain text.

If the coordinates could be thought of as a matrix row, you could write them within square brackets:

[12.7 33.912 81.07]

The problem with this notation is that coordinates can just as well be matrix columns rather than rows, so the matrix row representation carries a meaning that goes beyond a plain list of elements and is thus unwanted. Let’s forget about it.

If we were to use a unique separator rather than a comma or a space, then we could do away with the surrounding brackets completely. The presence of the unique “list element separator symbol” would be enough to indicate that this is a list.

Let’s try that idea with the backtick, which is one of the more rarely used ASCII characters:

12.7`33.912`81.07

This could work! But the backtick has problems of its own. It’s probably too similar to a number of other symbols like the apostrophe and the prime symbol, which is heavily used in math. The backtick is also already heavily overloaded in various programming languages, so it would have to be encoded in some representations and contexts. This is not good because we’d ideally like to establish an element separator symbol that could be truly universal.

There’s nothing left in 7-bit ASCII that hasn’t been overloaded a thousand times in computing history, so we’ll have dig into Unicode to find such a symbol.

One candidate I like is the “dot-above”, U+02D9, which looks like this: ˙

It’s a non-combining version of the “combining dot above” character used for writing diacritic letters such as .

When writing coordinates the dot-above has a visual association with the regular point as decimal separator, which I find quite pleasing:

12.7˙33.912˙81.07

But the downsides of the dot-above are many. It’s probably too small to be practical: in typical programming fonts, it looks too much like a space. It also might be confused with the interpunct a.k.a. middle dot, which is commonly used as a multiplication sign (often in the same countries that use a comma as decimal separator).

If we expand the dot-above into a vertical line, we get a non-combining macron, like this: ¯

The macron is wide enough to work as a programming symbol. Visually it’s a counterpart to the widely used _ underscore character. It’s not easily confounded with any other mathematical symbols:

12.7¯33.912¯81.07

As an additional bonus, an HTML entity shortcut exists for this character, it’s ¯. For people who produce HTML by hand, that’s definitely nicer to write than the raw Unicode encoding ¯

Fixing CSV

The macron is a suitable candidate for the universal element separator. In addition to coordinate lists, we can also apply it to fix the CSV format… Which, it must be said, is not so much an actual data format as an ancient bugbear that hides endless hideously incompatible plain-text encodings of all sorts of data. (CSV is like Oogie Boogie from Tim Burton’s Nightmare Before Christmas: a colony of bugs wrapped in a potato sack, pretending to be a single monster.)

The name “CSV” was originally short for “comma-separated values”, although it was later generalized to “character-separated values”. This is indicative of the utter lack of consistency surrounding this format — people can’t even agree about its name.

CSV is important because so many systems accept and produce it. In particular, it’s one of the most common interchange formats for spreadsheet data. Spreadsheets work mostly with numbers, and thus the unfortunate global overloading of the comma becomes eminently painful in any kind of international setting.

The German language edition of Excel will by default produce CSV files that use commas as decimal separators and semicolons as value separators. An American Excel user will receive the file, take a look at it, and complain loudly that the crazy German has produced a file of “comma-separated values” where the commas separate anything but… In a world of growing international tensions, this is no way to promote global peace and harmony.

By using the macron as a separator for CSV data, we can turn CSV into an actual file format — one with a simple and commonly agreed definition and syntax, just like JSON. Wouldn’t that be amazing?

Such an MSV (macron-separated values) file format would fix major issues with CSV by standardizing the following:

  • Unicode support. By using a Unicode character as the value separator, we ensure that non-Unicode-aware systems can’t mess with MSV.
  • Line breaks. The ASCII newline mess of LF vs. CR+LF characters is never going to die, unfortunately. There is a CSV semi-standard that specifies CRLF, so it probably would make sense for MSV to adopt the same.
  • Supported value types. Unicode strings and IEEE floating point numbers are a given. Anything else can be application-specific and treated as a string by other parsers.
  • Varying-length lines. Sure, why not? It should be allowed for the number of values on a line to change from line to line. This makes it easy to concatenate MSV files from different sources.

Besides, “macron” is just a cool word. It’s fun to pronounce and has a vague technological aura, so it can make you look smart in the right company. That reminds me, I must rush off to reserve the domain “macron.io” before the inevitable gold rush of new macron-related startups begins…

Posted in Names, Programming | Leave a comment

Passion is exploitable

In The Moderately Enthusiastic Programmer, Avdi Grimm is worried about a growing trend amongst software companies looking to hire programmers. He writes:

“But when I look at job adverts demanding ‘passion’ I get a little involuntary shiver. I remember needing a job. These are the kinds of job ads I’d be looking at, if I needed one again. And that makes me feel threatened, because they are looking for something I’m not sure I can give them.”

Such job ads have a fundamental similarity to Nigerian scam letters — you know the kind: “25 MILLION US dollars await in deposit to be transferred to your overseas account”… Those letters are well known for their clumsy and overwrought language, but it’s no accident. The language acts as a filter: if you’re rational enough to consider the form of the letter while reading it, you’re not the person that the scammer wants to attract anyway. They want you to develop an emotional attachment that they can manipulate, so that you’ll eventually send all your money to Nigeria. The job ads pointed out by Avdi Grimm are seeking a similarly irrational motivation on part of the employee.

Passion is an emotionally driven mental state that overrides reason. When you’re passionate, you’re not being rational. This is illustrated in expressions like “passionately in love” and “in the heat of passion”. Passion is the catalyst that sees people abandon their families, gamble away everything, commit manslaughter.

When an employer writes:

“We seek passionate software developers”

They really mean:

“We seek someone who cares so much about programming in itself that he/she will make irrational decisions to our advantage”.

In other words, the employer wants someone exploitable. Working for someone is always a trade. Because of the power dynamic, involving an emotional component like “passion” will make it unbalanced. The company gives you the means to express your burning desire for creating software, and in return you will work “passionately” — that is, disregarding reason.

Video game studios have a long history of this kind of exploitation. It’s an industry that attracts young people who love the products since early childhood and have long dreamed of making games themselves. The initial enthusiasm sees them happily doing overtime, and then comes “crunch time”: a long period of overtime and unreasonable demands which will slowly turn excitement into grinding anxiety… But the worker is emotionally committed to the project and won’t just walk away. (Electronic Arts famously paid $15 million in 2006 to settle a lawsuit which was sparked by an anonymous blogpost written under the pen name “EA widow”.)

This problem spans a wide range of creative or highly competitive professions. It’s not a question of salary: a musical production will pay relatively little, a marketing company will pay more, and a well-funded startup may reward its stars with excellent perks, yet any of these can be a workplace built on exploiting passion. Telltale symptoms include unpaid interns, chains of short contracts, and early employees without meaningful equity (in the case of startups).

This is a fairly new twist on employee exploitation. The 19th century industrial worker was not passionate about making steel or textiles. The employer exploited her basic need to survive, and this could be effectively combated by legislation and unionization.

Under Stalinism, those who claimed to fight against exploitation became the exploiters themselves. Their ideologues did in fact invent a prototypal form of the passionate industrial worker, The Hero of Socialist Labor… But this was always more of a fantasy than a real-world occurrence.

Yet governments have a history of being much more adept than companies at using people’s passions against themselves. If you’re going to ask a million young men to risk death in war for an ill-defined goal, it greatly helps if you can incite their patriotism into a genuine passion. World War I was the outcome of infectious passion inflamed by miscalculating reason.

Today such lofty, larger-than-myself passions are out of fashion. The postmodern worker is expected to be a creative individualist shaping out her own niche in the world. This is why companies rather than governments have become the harvesters of passion.

In the process, passion has become docile and domesticated. In 1914, the Kaiser and the French Republic wanted a violent manifestation of your passion: you were expected to kill someone, anyone, again and again, as long as they were on the other side. In 2014, the Bay startup and the Omaha marketing agency just want you to sit at a desk for 12 hours a day and produce stuff onto a computer screen. It’s passion stripped of its fundamental emotional basis.

The solution is to bring back the inherent unpredictability of passionate acts. If the company expects you to be passionate, do it consistently. Swear at the stupid client. Fall in love, drink and roam the town all night, show up to work at 1 pm. Quit your job to work on your photo-sharing startup. Start horseback riding and BASE jumping.

Individually you’ll pay a price. But when you’re unemployed, alcoholic, lying alone in a hospital bed with two broken legs, you’ll find comfort in the certain knowledge that you did it for the greater good. As soon as everyone does what you did, it will become the equivalent of a trade union for passionate people — employers will know what to expect when they put that fateful word “passion” in a job ad.

Posted in Names, Programming, Society | Leave a comment

Retreat, rethink, regroup, relaunch

(or: The year when my app was #2 in the Productivity category on Mac App Store but I didn’t know what to do)

I’m not American so I’m a bit fuzzy on the exact details of Thanksgiving protocol, but the basic idea seems good enough that it could be a worthwhile cultural export (like what has happened with Halloween). As the year draws to a close, it’s too easy to let one’s mind linger on all the things that didn’t really work out. Why not take a day to sit back, forget regret and think of all the things that are worth thanking for?

I ought to give thanks for all that has happened this year, even if’s not been all roses. My last update on this blog was in January… As the lack of updates suggests, I’ve been going through some turmoil. The things I was doing in early 2013 simply were not taking off. I spent a lot of time on relaunching PixelConduit, improving Radi and building up a multimedia publishing service called See.io which – so I hoped – would provide an avenue for monetizing Radi.

These efforts didn’t reach many people, and soon I reached the end of my runway as an entrepreneur. After being on my own since 2005, that was a bitter pill to swallow. Luckily it’s not difficult to find employment as a programmer, so at least my family didn’t have to suffer for my financial troubles.
Continue reading

Posted in Business stuff, Programming | Leave a comment

Conduit reaches version 3.0 and is now free

Here’s something that’s a big change for my business. Conduit is a realtime video app and graphics platform on which I’ve been working for a long time… It’s now been updated to version 3.0, but more importantly, the software is now available as a free download!

PixelConduit is the new name for Conduit Live. It’s the same powerful realtime video app you know, but now it’s free to use. There are no limitations to available resolutions, file formats or effect complexity.

Screenshot of PixelConduit

The PixelConduit app

This is not a gimped trial version, but the full unlimited software. The same applies to the Conduit plugins for Final Cut Pro, After Effects and other apps. You can download the plugins for free, use them anywhere you like and give the software to anyone – no more licenses to worry about.

Of course I’m still in business and hence would like to make money somewhere in this equation… The thing is, I haven’t really been in the traditional app business for a while now anyway. Most of my work recently has been involved with building custom solutions on top of the Conduit app and its underlying framework, now called Conduit Effect System. It makes a lot of sense for me to “liberate” the core software and concentrate on building added value on top of that.

Some of these custom developments are now available as an add-on pack called PixelConduit Complete. It includes Stage Tools, a powerful toolset for live shows and video control; Stereo 3D Tools, a set of highly useful nodes for working with stereoscopic 3D footage; and more! If you’re a pro user of Conduit, check out what PixelConduit Complete has to offer.

I’ve also consolidated all the Conduit documentation into a 145-page book, the PixelConduit User’s Guide. It’s available as a full PDF or individual chapters. Hopefully this will make it easier to learn Conduit and find your way in the app.

Thanks to all of you who have expressed your support and interest in Conduit! I hope you like where it’s going with this new approach. Naturally I’d love to hear your thoughts and comments on this.

(You can find the list of changes in PixelConduit 3.0 on the Lacquer blog.)

Posted in Business stuff, Mac-related | Leave a comment

The Union of Councils

After some technical posts here’s finally one that is back on my blog’s stated topic, ‘naming things’.

When I was a kid, we lived practically next door to some nice people. They were called the Council Folks. These people, it was said, considered all the people of the Earth their friends. They were hospitable to strangers and had an interest in all walks of life. When the Council Folks set their mind on something they would eventually excel, whether in agriculture, chess or music. The Council Folks were at home even in space where they kept a permanent presence aboard an orbital station named Peace.

Not a lot of people ever believed the official lithurgy, I suppose. When I was 11 the Union of Councils collapsed and the ‘Council Folks’ reverted to being just Russians, which is what most everyone in Finland had been calling them anyway.

Continue reading

Posted in Names, Society, Stories | Leave a comment

Win-win with Cocotron and Xcode 4.4 — code for Mac, build for Windows (Part 2)

At long last, this is part 2 of my Cocotron tutorial. Sorry to have kept you waiting for so long!

While I was dragging my feet with this tutorial, several milestones were reached in Cocoa-land. Xcode was upgraded to 4.4 and the OS itself was upgraded to 10.8. The instructions herein have been tested to work with these latest versions of the Apple development ecosystem. Hopefully you’ll find that to be the case as well – if not, please post in the comments and let’s try to work it out. (Personally I reached a milestone as well: I’m a father nowadays. We had a healthy and lively baby girl… Who sleeps so well that I’ve been able to find the time for some Cocoa hacking again.)

In Part 1 of this tutorial, we looked into what Cocotron is made of and how it’s installed into Xcode. We managed to build the basic Cocoa frameworks, Foundation and AppKit, for Windows, but didn’t yet create a Windows app that would actually use those libraries.

That will be the focus of this Part 2. We’ll start with a barebones Cocoa app created from the default Xcode template and tweak the magic settings in order to build a Windows app from the same code.

Before getting into that, let’s dwell a while longer in the Cocotron framework projects. AppKit’s configuration options provide us the opportunity to practice something that will eventually become useful in almost any larger project: how to link with existing Windows libraries from third parties.

Continue reading

Posted in Mac-related, Programming | 11 Comments

Crunching megapixels in WebGL

I recently finished Radi 0.9. It has a major new feature (well, from my point of view at least!) which allows the user to design realtime image filters and publish them to the web. The big idea is that nothing gets pre-rendered: instead, the filter is always rendered in the browser, which allows the web page to modify and change the filters on the fly. (I’ve made some elementary demos, you can see them here.) This feature is implemented using WebGL, a 3D extension of the HTML5 Canvas interface.

This is the first time I’ve used this fairly new browser API, which could be reasonably characterized as “bleeding-edge”… Although WebGL is supported in Chrome, Firefox and Opera, even in these browsers it may be disabled on systems whose graphics hardware has not been tested and qualified. WebGL is also available in Safari, but it’s off by default and needs to be manually enabled through the special ‘Develop’ menu. To further complicate things, WebGL is a very large API that needs to interact directly with both browser internals as well as graphics drivers, which makes it even more of a moving target for development. Given all this, I’m somewhat amazed that WebGL works as well as it does already!

In this post, I’d like to share a few things I learned about WebGL. If you’re not a JavaScript developer, you won’t find most of this post very interesting. However the first sections may be worth your time even if you’re not a developer but just interested in using Radi’s WebGL filters, because they explain how Radi’s WebGL filters work and also some of their limitations.

Continue reading

Posted in Programming, Web | 1 Comment

Radi 0.8.1 and EPS import musings

I finally managed to complete another update to Radi, my HTML5 content creation app. This new version 0.8.1 is all about fixing bugs and lots of workflow annoyances. Even though the version number increase is tiny, I feel that the improvements are substantial especially if you’re doing vector graphics animation. You can read more about it on the Radi news page, or go straight to the v0.8.1 release notes. (Radi remains a free download, of course.)

I’ve also started writing a real guide book for Radi. The title is Learn HTML5 Multimedia With Radi, and it’s still a work in progress. Only the first chapter is mostly done. I’m hoping to get it finished within the next months.

Well, I claimed that 0.8.1 doesn’t add new features, but actually I did sneak in one new feature that’s fairly major, at least in terms of programming complexity. You can now import vector graphics in the EPS format. This feature is “experimental” — that is to say, not working very well much of the time… Therefore it’s a bit buried away. (You can find the use instructions in the release notes.)

EPS (Encapsulated PostScript) is a complex format that’s designed more for printers rather than transferring graphics between applications. But EPS has been around for so long and is so widely supported by all the important vector graphics apps, so I felt that it’s the only practical choice if I have to pick one vector format right now… There are plenty of EPS graphic files around for things like company logos and icons, and now you can bring them into Radi as real vector shapes that remain scalable and editable.

Simple graphics like logos and outline drawings are mostly what the importer is suitable for. Complex fill styles, clipping masks and texts won’t be imported at all. (To import text, you should vectorize it first… But it’s generally better to recreate the text within Radi if possible.)

By the way, if you’re more interested in the second part of my Cocotron tutorial than Radi, fret not, it’s coming this month. (Edit in June: sorry, I was overly optimistic — the rest of the tutorial is still unfinished… Apologies for keeping you waiting.)

Posted in Animation, Mac-related | Leave a comment

Win-win with Cocotron and Xcode 4.3 — code for Mac, build for Windows (Part 1)

This is the first part of a tutorial in which I’d like to introduce you to one of my favorite open-source projects, The Cocotron (imagine the ominous silence of a Soviet nuclear reactor as the backdrop for pronouncing this name).

Cocotron lets you take a Cocoa-based Mac app and port it to Windows without ever leaving Xcode — just add a Windows target to your project and you’re set. Sounds too good to be true? Well, as usual, some limitations apply. Cocoa is a large and rapidly evolving framework, and it would be very difficult for a loose team of open-source volunteers to achieve 100% compatibility with Apple’s latest and greatest. But Cocotron can take you most of the way, and if you have the possibility to plan ahead for compatibility, the odds of being able to make use of Cocotron are further increased.

The Cocotron is not just a vapor framework outlining some kind of theoretical feasibility: it’s been publicly available since late 2006, and ports accomplished using Cocotron are out in the wild. Personally I have used it deploy Mac-developed custom apps on Windows, including a fairly complex GUI subtitling tool used by the Finnish Broadcasting Company for a popular TV game show. (Based on activity in the Cocotron mailing list, there is also a company with a well-known consumer-oriented Mac product who are nearing completion of a Windows port using Cocotron.)

Something that appeals to me in Cocotron is its strict, almost austere “no-frills” policy. Christopher Lloyd, the project’s initiator, runs a tight ship. Unlike some other cross-platform toolkits (* cough Gtk+ and Qt *), the framework is lean: Cocotron’s Windows DLLs for Foundation and AppKit only take about 6-7 MB. And it’s self-contained – there are no other dependencies by default. As fits this culture of independence, Cocotron is MIT-licensed, so you can basically do anything you please with the code as long as credit is given.

True cross-compiling on the Mac is a core part of Cocotron’s promise. As such, its fortunes are closely tied to Apple’s Xcode development environment which has undergone a lot of changes over the past five years. Luckily Xcode has consistently remained open enough to allow custom cross-compilers (perhaps largely thanks to the advent of iOS and Cocoa Touch on ARM CPUs, which has made cross-compiling an essential part of Xcode).

Xcode 4 was the biggest change in the product’s history. I had been long dreading making the leap to Xcode 4 for two reasons: the new UI would force me to re-learn many workflows, and I was worried that Cocotron might not work properly anymore. Now it has been a week since I finally made the change, and I can report that I’m pretty happy about it. That means you could be happy too with Cocotron and Xcode 4.3! This tutorial will tell you how.
Continue reading

Posted in Mac-related, Programming | 19 Comments

Good life at Happy Meal Central

It’s been about a decade since the last time I had a meal at McDonald’s. I heard that they’re trying to change their image, and indeed it turns out that they’re nothing as stuck to their singular success recipe as I had assumed:

Vegetarian McDonald's meal in Helsinki

That’s a vegetarian McFeast with rye bread; fresh apple slices instead of fries; and orange juice instead of a soda.

These options don’t cost extra over the standard fare. This was actually a very decent meal for the price, so I don’t think McDonald’s will have to wait another ten years until I bring them another 6 euros.

This begs the question — is it like this everywhere? Or does Finland, by some strange twist of fate, have the best McDonald’s in the world?

Posted in Uncategorized | 1 Comment