Highway 61 revisited

As I sit here with a Cesária Évora CD on in the house, I have an update to the car AV system issue, wherein it couldn’t stop playing Bob Dylan. That is, I found out why it’s playing a disproportionate amount of Dylan.

I noticed, as it played more songs, that it was not just playing a lot of Bob Dylan and the Beatles, but that it wasn’t playing anything beyond C in the alphabet. I have the files on the microSD organized in folders (directories) based on the artists and albums, so at the root level there’s a Bob Dylan folder, and that has sub-folders called Blonde On Blonde, Blood On the Tracks, Desire, and so on. In those folders are the MP3 files for the songs. I used the touch-screen interface to look in the folder of the current song, then went up to the artist level, and then to the root. I scrolled the list of artists, which should have gone from 10,000 Maniacs to Youssou N’Dour. But the list stopped somewhere near the end of the Cs.

Ha! There appears to be a limit to the number of directories. And with only around 1000 songs active, instead of the 4000 on the chip, the chances of Dylan had been multiplied by 4 for each play. No wonder I was getting so much! OK, I can work around that limit.

I took the chip into the house, put it in my computer, and wrote a script to pull all the files out to the root level, so there are no directories/folders. /Bob Dylan/Desire/03 Mozambique.mp3 became /Bob Dylan-Desire-03 Mozambique.mp3, and now I have 3984 files in the root directory, and no folders. Pop the chip back into the car system, and try it out.

Great! There’s a D... now an L... a G. Much better!

But it didn’t take too long to notice that it never played anything beyond L. I went to the list and scrolled again (and was happy that one can scroll backward, and it wraps around).

This time, it was easy to tell exactly: the files in each directory are numbered sequentially by the system, and with everything in the root directory I could see what the actual limit is: 2500 files, exactly. That’s horrid!

2500 files might be a reasonable limit when microSD chips only went up to 2 GB. But that was a while ago, and it’s perfectly easy to have 8000 files or more now, and higher-capacity chips are coming out all the time. It’s absolutely ridiculous to build in limits like this, considering how the technology is moving forward. Any reasonable file system has tossed such limits away long ago.

I’m trying to delete 1500 files from the microSD card, but it’s tough: the music on my computer is already selected from my far more extensive CD collection, and represents my favourites. How do I pick 1500 favourites to eliminate? The first 500 went gradually, but it wasn’t too awful. The second 500 were a real challenge. I’m still working on the last 500, and it’s very tough!

I’ll be writing to Pioneer, to express my displeasure and to see if there’s anything that can be done. And I guess I’ll go back to streaming the music from my BlackBerry, which still has all the songs, and for which there’s no such limitation.

---

Update, 4 p.m.: Pioneer's customer service gave me a prompt response, which confirms what I saw:

The maximum number of files on USB or SD that this unit will support is 2500. Currently there are no plans to change that specification. Your feedback is appreciated and will be passed along to product planning.

Hands-free is not enough

The other day, we heard that the U.S. NTSB would be proposing a nationwide ban on mobile phone use, and people were speculating that it’d increase sales and use of hands-free calling. I thought that would be odd, since a number of studies have made it clear that it’s mostly talking on your mobile phone that’s dangerous, whether it’s hands-free or not. There’s cognitive interference when you talk to someone who isn’t in the car with you, and having the device be hands-free only helps with the mechanical aspects, not with the cognitive ones, and those appear to be more important from a safety point of view.

And, as I expected, the proposed ban includes hands-free devices. Quoting from their news release of yesterday:

No call, no text, no update behind the wheel: NTSB calls for nationwide ban on PEDs while driving

December 13, 2011

Following today’s Board meeting on the 2010 multi-vehicle highway accident in Gray Summit, Missouri, the National Transportation Safety Board (NTSB) called for the first-ever nationwide ban on driver use of personal electronic devices (PEDs) while operating a motor vehicle.

The safety recommendation specifically calls for the 50 states and the District of Columbia to ban the nonemergency use of portable electronic devices (other than those designed to support the driving task) for all drivers. The safety recommendation also urges use of the NHTSA model of high-visibility enforcement to support these bans and implementation of targeted communication campaigns to inform motorists of the new law and heightened enforcement.

According to NHTSA, more than 3,000 people lost their lives last year in distraction-related accidents, said Chairman Deborah A.P. Hersman. It is time for all of us to stand up for safety by turning off electronic devices when driving.

No call, no text, no update, is worth a human life.

Note nonemergency use and other than those designed to support the driving task. There is no exception for hands-free devices in their recommendation.

The NTSB has no standing to force this; in the United States, the states make their own traffic rules. But Congress can back the recommendation with funding incentives, as they did with the now-defunct 55 MPH speed limit, and as they have done for laws requiring seat-belt use.

Car navigation and audio system

I have a new car (as of October), a Subaru Legacy. The new car has a Pioneer AVIC-X930BT navigation/audio system (I mentioned its anti-theft mechanism last week). After years of using maps and printed directions, or relying on my BlackBerry for the GPS task — it’s effective, but small and hard to use while one is driving — it’s good to have the nav system, with a nice, large touch-screen (the next model up also does voice-command activation).

I’m mostly happy with it, and only find two annoying quirks in the navigation system. The more irritating of the two is that there’s a safety disclaimer that I have to touch OK to dismiss every time I start up the system:

Caution!

This Navi product is intended solely as a driving aid. Review instruction manual and select route before driving. Navi is not a substitute for your attentiveness, judgement, and care while driving or moving your vehicle. Always observe safe driving rules and driving laws, and follow road signs even if they contradict Navi’s instructions. By pressing OK key, you accept the license agreement in the instruction manual.

I get the issue here: there are a good many documented cases of people driving onto railroad tracks, going the wrong way on one-way roads, and other such because they blindly and stupidly followed (what they thought were) their GPS systems’ instructions. But, really, I ought to be able to accept that safety and license-agreement message once, and be done with it. Or if they must remind me periodically, how about once a month? Even weekly would be better than having to deal with it every time I start the car.

The other annoyance is that the positioning system doesn’t seem to understand reverse gear. When I pull up my driveway and into my garage, the system doesn’t know where I am with respect to the roads. When I start up again, back out, and head up the road, it thinks I’m on the next block, and remains confused about that for a few minutes, while it acquires the GPS satellites and sorts out its actual location. That’s mostly comical, because I don’t need the GPS location to be accurate when I’m near home. Still, it’s rather goofy.

The most interesting thing about the system is that it replaces the audio system in the car, and includes AM and FM radio, CD and DVD player (including video), bluetooth audio streaming from your smartphone, playing your iPod, playing Pandora or Aha from the Inernet via an iPhone app, and playing music or video files from a USB device or microSD card. It makes for quite the music system.

I had been streaming music from my BlackBerry, but there’s not really a need — microSD cards are very cheap these days. I got a new card and copied all my music onto it. Thousands of music files live with the car. Very nice.

But there’s a problem, caused by a combination of an odd software choice in the Pioneer system and what happens to the microSD card on my MacBook. When I first plugged the microSD card into the audio system and turned on random play, it played the first song, gave me a popup message saying that unplayable files would be skipped, picked a random next song, and turned the random-play feature off.

Unplayable files?

I investigated. There turn out to be three things causing this, all related to hidden files (files whose names begin with ., which are hidden by the Unix file system that’s used on the Mac):

1. iTunes, which manages my music library on the Mac (and whence I copied the files), keeps a hidden file associated with each song, to keep track of metadata. When I copied the music directories, I copied all those as well.
2. Spotlight, a Mac feature that helps you search for things, creates a hidden directory structure called .Spotlight-V100 when it indexes the drive. This happens just because you plugged the microSD card into the Mac.
3. The operating system and the MacOS Finder create various hidden files and directories, both in the root of the drive and in its subdirectories: .Trashes, .fseventsd, and .DS_Store (that last exists in every subdirectory that’s been touched by the Finder).

I configured Spotlight not to index the microSD drive (which you can only conveniently do after it’s already done it), and then wrote a shell script to delete every file and directory whose name begins with . (and one had best be very careful about writing and running such a script). Every time I plug it into my Mac, I have to run the script on it just before I eject it when I’m done.

Now everything works great. The audio system no longer complains about unplayable files, and the random-play feature doesn’t get turned off. I find that truly an odd programming choice: not only to display the message (which is odd enough), but then to stop random play. But it’s also bad that MacOS treats removable media that way... it should assume that removable media formatted with FAT(32) will be used on non-Mac platforms, and not pollute it with Mac-specific stuff.

Anti-theft?

The navigation system in my car has an anti-theft feature that’s interesting, in that it relies entirely on a sort of herd immunity. The system is installed in the car’s dashboard, so it’s somewhat involved to pull it out. Easy for a pro, to be sure, but I mean that it’s not like one of those that sits on top, and one can just grab it and run.

When it’s first powered on after installation, the owner has the option of setting a password. If a password is set and the unit is ever disconnected from the battery, as it would be if it were stolen (or, of course, when the car battery is replaced, or when servicing the car requires disconnecting the battery), the password has to be entered in order for the device to be used again. The only way to recover from a forgotten password is to have the manufacturer reset the system — and they will, one presumes, take some measures to ensure that you hadn’t simply boosted it.

The interesting thing about this mechanism is that there’s no way for a thief to know whether or not a password is set. This anti-theft feature does nothing to actually prevent theft, but only to prevent the use of the system after it’s stolen. That’s only a deterrent if the thief knows two things: that this model has this feature and that almost all owners set a password (so that the likelihood of stealing a usable unit is too low to be worth the trouble).

Setting a password does absolutely nothing for your own device’s security — once it’s stolen, no thief will come put it back when he finds that he can’t use it nor sell it. Rather, we all depend on the widespread knowledge, at least among thieves, that everyone sets one. If I opt out, I’m covered by the rest of you. But if too many people opt out, then no one’s unit is safe.

And there is a big down side to setting a password: when your battery’s disconnected for service, if you’ve forgotten the password (which you only used once, maybe several years ago) your nav system becomes a brick.

Perhaps all in-dash navigation systems use this mechanism, and thieves are well aware of that (and new thieves soon will be). I wonder, though, how many owners choose not to set a password.

Degrees of separation

New Scientist tells us about Facebook’s analysis of the friend relationships in their social network. Only four degrees of separation, says Facebook, goes the New Scientist headline. Here’s their summary:

A few months ago, we reported that a Yahoo team planned to test the six degrees of separation theory on Facebook. Now, Facebook’s own data team has beat them to the punch, proving that most Facebook users are only separated by four degrees.

Facebook researchers pored through the records of all 721 million active users, who collectively have designated 69 billion "friendships" among them. The number of friends differs widely. Some users have designated only a single friend, probably the person who persuaded them to join Facebook. Others have accumulated thousands. The median is about 100.

To test the six degrees theory, the Facebook researchers systematically tested how many friend connections they needed to link any two users. Globally, they found a sharp peak at five hops, meaning that most pairs of Facebook users could be connected through four intermediate people also on Facebook (92 per cent). Paths were even shorter within a single country, typically involving only three other people, even in large countries such as the US.

The world, they conclude, just became a little smaller.

Well, maybe. There are a lot of things at play here, and it’s not simple. It is interesting, and it’s worth continuing to play with the data, but it’s not simple.

They’re studying a specific collection of people, who are already connected in a particular way: they use Facebook. That gives us a situation where part of the conclusion is built right into the study. To use the Kevin Bacon comparison, if we just look at movie actors, we’ll find closer connections to Mr Bacon than in the world at large. Perhaps within the community of movie actors, everyone’s within, say, four degrees of separation from Kevin Bacon. I don’t know any people in the movie industry directly, but I know people who do, so there’s two additional degrees to get to me. We can’t look at a particular community of people and generalize it to those outside that community.

There’s also a different model of friends on Facebook, compared with how acquaintance works in the real world. For some people, they’re similar, of course, but many Facebook users have lots of friends whom they don’t actually know. Sometimes they know them through Facebook or other online systems, and sometimes they don’t know them at all. Promiscuous friending might or might not be a bad thing, depending upon what one wants to use one’s Facebook identity for, but it skews studies like this, in any case.

People would play with similar things in the real-life six degrees game. Reading a book by my favourite author doesn’t count, but if I passed him on the street in New York City, does that qualify? What about if we went into the same building? If he held the door for me? If I went to his book signing, and he shook my hand and signed my copy of his book? Facebook puts a big e-wrinkle on that discussion.

But then, too, it’s clear that with blogs and tweets and social networking, we have changed the way we interconnect and interact, and we have changed how we look at being acquainted with people. I know people from the comments in these pages, and from my reading and commenting on other blogs. Yes, I definitely know them, and some to the point where I call them friends in the older, pre-social-network sense. But some I’ve never met face to face, nor talked with by voice.

So, yes, the world probably is a little smaller than it used to be. It didn’t just get that way suddenly, of course; it’s been moving in that direction for a while. Everything from telephones and airplanes to computers and the Internet have been taking us there.

Inventing the Internet

I’m in Québec City this week for the IETF meeting. A group of us were having dinner last evening, and at the end of the meal, as we were paying, the waitress asked us what we were all in town for. We told her were were at a meeting to work on standards for how things talk to each other on the Internet.

So she tells us about a crazy lady who comes in the restaurant every afternoon. The lady claims to have invented a bunch of things, and one thing she says is that she invented the Internet. After someone makes the required Al Gore joke, I say, well, to tell you the truth, no one at this table qualifies but we do actually have some people in our group who actually did invent the Internet. She says It’s one person who did it?, and we say no, maybe eight or ten or so... and at least four of them really are here this week.

Number 2 and trying harder?

Yahoo news notes that LinkedIn is now the number 2 social network, behind, of course, Facebook. Brent Hailpern has an amusing way of pointing out what that means, really:

In related news, Beta is now the No. 2 video tape format after VHS.

Indeed, at what point is number 2 so far behind that it simply doesn’t matter?

In this case, LinkedIn isn’t really even relevant: its focus is entirely different from Facebook’s, and one wouldn’t really say that they compete with each other. This is really saying that MySpace has fallen so far back that it’s even gone below LinkedIn.

But another point is that the newcomer, Google+, is way down there at number 4 or lower. It’s in beta, of course, but, well, that’s just Google, where pretty much everything is in perpetual beta. But Google+ is aiming to be a Facebook competitor. Is there any hope? Should they bother? Shouldn’t they put their resources where they might do more good? Won’t Google+ just go the way of Google Wave?

It certainly happens that something new comes from way, way back there and pushes its way to the front. That can sometimes be due to the prominence of the company backing it, as happened when Microsoft Internet Explorer took over the world, to the dismay of Netscape (Who?). Google certainly has a prominent, powerful position, but it seems unlikely that that alone would bump Facebook out of the number 1 spot, or even seriously threaten to.

The other way for a newbie to move up is by providing important improvements over what’s already out there. Facebook’s recent partnership with Skype gives it immunity from Google Voice, but Google is marketing Google+ as having better privacy than Facebook — and, &deity knows, the latter has had a great deal of bad press for its handling of privacy issues and controls.

So, is Google+ a better social-networking choice from a privacy standpoint? We have one datapoint so far, and it doesn’t look good: the folks at F-Secure, a Finnish anti-malware company, note that as part of the Google+ rollout, Google will be deleting all private profiles, thus requiring you to make your profile public if you want to keep it. What’s more, they’ve done a lot of the same things that Facebook has done, quietly making new things public and/or enabled by default, so you really have to keep on top of things to be sure you avoid information leaks.

That doesn’t sound like an improvement to me.

Netflix abuses its customers

In December, I complained about Netflix streaming: not enough of what I want to watch is available for streaming. But some is, and getting one DVD at a time in addition to the streaming makes up for the lack, at least somewhat. In the end, then, we decided to keep the $10 Netflix subscription.

But Netflix has just announced that it’s increasing the cost of that plan by 60%. That’s a lot!

They’re actually doing it by separating the streaming and DVD plans, and charging $8 for each. Here’s what they say about it in their email message:

We are separating unlimited DVDs by mail and unlimited streaming into two separate plans to better reflect the costs of each. Now our members have a choice: a streaming only plan, a DVD only plan, or both.

Your current $9.99 a month membership for unlimited streaming and unlimited DVDs will be split into 2 distinct plans:

Plan 1: Unlimited Streaming (no DVDs) for $7.99 a month

Plan 2: Unlimited DVDs, 1 out at-a-time (no streaming) for $7.99 a month

Your price for getting both of these plans will be $15.98 a month ($7.99 + $7.99). You don’t need to do anything to continue your memberships for both unlimited streaming and unlimited DVDs.

These prices will start for charges on or after September 1, 2011.

The good part, I suppose, is that people who do only want one of the services can save $2 a month (let’s skip the penny here or there). But the assholes nice people at Netflix are doing a massive 60% rate hike for those who want the same package they’ve been using.

And one thing that’s particularly irritating about this is that if Cablevision, Time Warner Cable, Comcast, or Verizon wanted a 60% rise in rates, they’d have to get permission for it from regulatory agencies, and they wouldn’t be allowed to dump it all on us at once. Netflix has no such restriction, and can do what it wants... it’s up to us to say No! by not buying their service.

And so I’m really undecided about what to do. On the one hand, I’ve gotten used to the streaming, despite its limitations, and it’s nice to have stuff available and to watch things on the laptop when I’m travelling (in the U.S.). It’s tempting to just drop the DVD service and continue with $8/month for the streaming.

On the other hand, I very much want to give Netflix a clear message that they can go fuck themselves, and hope they lose 80% of their customers and go out of business.

Misconceptions about DKIM

I chair the DKIM working group in the IETF. The working group is finishing up its work, about ready to publish an update to the DKIM protocol, which moves DomainKeys Identified Mail up the standards track to Draft Standard.

DKIM is a protocol that uses digital signatures to attach a confirmed domain name to an email message (see part 7, in particular). DKIM started from a simple place, with a simple problem statement and a simple goal:

• Email messages have many addresses associated with them, but none are authenticated, so none can be relied on.
• Bad actors — spammers and phishers — take advantage of that to pretend they are sending mail from a place (a domain name) the recipient might trust, in an attempt to fool the recipient.
• If we can provide an authenticated domain name, something that’s confirmed and that a sender can’t fake, then that information can be used as part of the delivery system, as part of deciding how to handle incoming mail.

It’s important to note that mail signed with DKIM isn’t necessarily good mail, nor even mail from a good place. All we know is that mail signed with DKIM was digitally signed by a specified domain. We can then use other information we have about that domain as part of the decision to deliver the message to the user’s inbox, to put it in junk mail, to subject it to further analysis or to skip that analysis, and so on.

Domain example.com signed this message, is just one of many pieces of information that might help decide what to do.

But some people — even some who have worked on the development of the DKIM protocol — miss the point, and put DKIM in a higher position than it should be. Or, perhaps more accurately, they give it a different place in the email delivery system than it should have.

Consider this severely flawed blog post from Trend Micro, a computer security company that should know better, but doesn’t:

In a recently concluded discussion by the [DKIM Working Group], some of those involved have decided to disregard phishing-related threats common in today’s effective social engineering attacks. Rather than validating DKIM’s input and not relying upon specialized handling of DKIM results, some members deemed it a protocol layer violation to examine elements that may result in highly deceptive messages when accepted on the basis of DKIM signatures.

The blog post describes an attack that takes a legitimately signed message, alters it in a way that does not invalidate the DKIM signature (taking advantage of some intentional flexibility in DKIM), and re-sends the message as spam or phishing. The attacker can add a second from address, and appear to the user to be from a trusted domain, though the DKIM signature is not.

The attack sounds bad, but it really isn’t, and the Trend Micro blog’s conclusion that failure to absolutely block this makes DKIM an EVIL protocol (their words) is not just overstated, but laughable and ridiculous. It completely undermines Trend Micro’s credibility.

Here’s why the attack is overstated:

1. It relies on the sender’s ability to get a DKIM signature on a phishing message, and assumes the message will be treated as credible by the delivery system.
2. It ignores the facts that delivery systems use other factors in deciding how to handle incoming messages and that they will downgrade the reputation score of a domain that’s seen to sign these sorts of things.
3. It ignores the fact that high-value domains, with strong reputations, will not allow the attackers to use them for signing.
4. The attack creates a message with two from lines, and such messages are not valid. It ignores the fact that delivery systems will take that into account as they score the message and make their decisions.

Apart from that, the blog insists that the right way to handle this attack would be to have DKIM go far beyond what it’s designed to do. Rather than just attaching a confirmed domain name to the message, DKIM would, Trend Micro says, now have to check the validity of messages during signature validation. Yes, that is a layer violation. Validity checking is an important part of the analysis of incoming email, but it is a separate function that’s not a part of DKIM. All messages, whether DKIM is in use or not, should be checked for being well-formed, and deviations from correct form should increase the spam score of a message. That has nothing to do with DKIM.

In fact, the updated DKIM specification does address this attack, and suggests things that delivery systems might do in light of it. But however good that advice might be, it’s not mandated by the DKIM protocol, because it belongs in a separate part of the analysis of the message.

---

Others have also posted rebuttals of the Trend Micro blog post. You can find one here, at CircleID, and look in the comments there for pointers to others.

Ephemeral clouds

I’ve talked about cloud computing a number of times in these  pages. It’s a model of networking that in some ways brings us back to the monolithic data center, but in other ways makes that data center distributed, rather than central. A data cloud, an application cloud, a services cloud. An everything cloud, and, indeed, when one reads about cloud computing one sees a load of [X]aaS acronyms, the aaS part meaning as a service: Software as a Service (SaaS), Infrastructure as a Service (IaaS), Platform as a Service (PaaS), and so on.

I use email in the cloud. I keep my blog in the cloud. I post photos in the cloud. I have my own hosted domain, and I could have my email there, my blog, there, my photos there... but who would maintain the software? I could pay my hosting service extra for that, perhaps, but, well, the cloud works for me.

It works for many small to medium businesses, as well. Companies pay for cloud-based services, and, in return, the services promise things. There are service-level agreements, just as we’ve always had, and companies that use cloud-based services get reliability and availability guarantees, security guarantees, redundancy, backups in the cloud, and so on. Their data is out there, and their data is protected.

But what happens when they want to move? Suppose there’s a better deal from another cloud service. Suppose I, as a user, want to move my photos from Flickr to Picasa, or from one of those to a new service. Suppose a company has 2.5 terabytes of stuff out there, in a complex file-system-like hierarchy, all backed up and encrypted and safe and secure... and they want to move it to another provider.

In the worst case, suppose they have to, because their current service provider is going out of business.

Recently, Google Video announced that they would take their content down, after having shut the uploads down (in favour of YouTube) some time ago. This week, Friendster announced that they would revamp their service, removing most of their data in the process.

Of course, you understand that when I say their data, here, I really mean your data, yes? Because those Google Video things were uploaded by their users, and the Friendster stuff is... well, here’s what they say:

An e-mail sent Tuesday to registered users told them to expect a new and improved Friendster site in the coming weeks. It also warned them that their existing account profile, photos, messages, blog posts and more will be deleted on May 31. A basic profile and friends list will be preserved for each user.

Now, that sort of thing can happen: when you rely on a company for services, the company might, at some point, go away, terminate the service, or whatnot. But what’s the backup plan? Where’s the migration path? In short...

...how do you save your data?

Friendster has, it seems, provided a exporter app that will let people grab their stuff before it goes away. Google Video did no such thing, and there’s a crowd-sourced effort to save the content. But in the general case, this is an issue: if your provider goes away — or becomes abusive or hostile — how easy will it be for you to get hold of what you have stored there, and to move it somewhere else?

Be sure you consider that when you make your plans.

[Just for completeness: I have copies on my own local disks of everything I’ve put online... including archives of the content of these pages. If things should go away, it might be a nuisance, but I’ll have no data loss.]

Reasonable network management

Back in December, the U.S. Federal Communications Commission released a Report and Order specifying new rules related to network neutrality. The rules have since been challenged in court in separate suits by Verizon and Metro PCS. They’re also under attack by the House of Representatives, though whatever they do is unlikely to pass the Senate and the president.

The Report and Order is quite long and involved, a typical federal document that runs to 194 pages (here’s a PDF of it, in case you’d like to read the whole thing). On page 135 there begins a statement by FCC Chairman Genachowski, which contains, on page 137, five points, key principles, as Mr Genachowski says, that lead to key rules designed to preserve Internet freedom and openness. That’s sort of an executive summary of the document.

I’ll note principles four and five here:

Fourth, the rules recognize that broadband providers need meaningful flexibility to manage their networks to deal with congestion, security, and other issues. And we also recognize the importance and value of business-model experimentation, such as tiered pricing. These are practical necessities, and will help promote investment in, and expansion of, high-speed broadband networks. So, for example, the order rules make clear that broadband providers can engage in reasonable network management.

Fifth, the principle of Internet openness applies to mobile broadband. There is one Internet, and it must remain an open platform, however consumers and innovators access it. And so today we are adopting, for the first time, broadly applicable rules requiring transparency for mobile broadband providers, and prohibiting them from blocking websites or blocking certain competitive applications.

In apparent response to those points, and taking transparency seriously, Verizon Wireless has recently updated their Customer Agreement (Terms and Conditions). If you scroll down to the bottom of that document, you’ll find a section called Additional Disclosures, the first paragraph of which says this:

We are implementing optimization and transcoding technologies in our network to transmit data files in a more efficient manner to allow available network capacity to benefit the greatest number of users. These techniques include caching less data, using less capacity, and sizing the video more appropriately for the device. The optimization process is agnostic to the content itself and to the website that provides it. While we invest much effort to avoid changing text, image, and video files in the compression process and while any change to the file is likely to be indiscernible, the optimization process may minimally impact the appearance of the file as displayed on your device. For a further, more detailed explanation of these techniques, please visit www.verizonwireless.com/vzwoptimization

That URL at the end lacks the http at the beginning and has not been made into a clickable link, but if you copy/paste it into your browser’s address bar, you’ll be redirected to a long page called Explanation of Optimization Deployment, full of technical details. It’s perhaps the most detailed and technical disclosure I’ve seen presented to consumers, full of terms such as Internet latency, quantization, codecs, caching, transcoding, and buffer tuning.

I have to say that the policy looks reasonable. They say that they apply their optimization (not really the right term, here, but that’s the marketing spin) to all content, including Verizon Wireless branded content. They compress images and transcode video to reach a compromise between fidelity to the original content and what’s likely to be useful on a mobile device, conserving transmission resources by doing it. But it also benefits the consumer by way of reduced data charges. They also, basically, stream the content (buffer tuning), so if you stop a video in the middle you don’t have to transmit (nor pay for the transmission of) the unwatched portion.

The only disadvantage of any of this as I see it is that there’s no way to turn it off. If you notice degradation of your video content and want to watch the original — and are willing to pay for extra data transmission that entails — you can’t.

As a first step, this looks good: it’s a reasonable policy that preserves the essence of neutrality and fits the reasonable network management model. Of course, Verizon Wireless may just be testing the water, introducing changes a little at a time, with the most benign changes first. We’ll have to see.

URL shorteners

If you’re a twit a Twitter user, you’ve likely used one or another of the URL shorteners out there. Even if you’re not, you may have run across a shortened URL. The first one I encountered, several years ago, was tinyurl.com, but there plenty of them, including bit.ly, tr.im, qoiob.com, tinyarrow.ws, tweak, and many others.

The way they work is that you go to one of them and enter a URL — say, the URL for this page you’re reading:

http://staringatemptypages.blogspot.com/2011/03/url_shorteners.html

...you click a button and get back a short link, such as this one:

http://bit.ly/eqHg3S

...that will get users to the same page. The shortened link redirects to the target page, and won’t take up too many characters in a Twitter or SMS message. It also may hide the ugliness of some horrendously long URL generated by, say, Lotus Domino.

On the other hand, it will also hide the URL that it points to. When you look at the bit.ly link above, you have no idea where it will take you. Maybe it’ll be to one of these august pages, maybe it will be to a New York Times article, maybe to a YouTube video, and maybe to a page of pornography. Click on a shortened URL at your own peril.

In addition, any URL you post, long or short, might eventually disappear (or, perhaps worse, point to content that differs from what you’d meant to link to), but if you post a load of shortened URLs to your blog or Twitter stream and then the service you used goes out of business, all your links will break at once. That didn’t used to happen, but can now. And because some of them use country-code top-level domains (.ly, .im, .tk, and .ws, for example), the services may be subject to disruption for other reasons — one imagines that the Isle of Man and Western Samoa might be stable enough, but if you’ve been watching the news lately you might be less sanguine about Lybia.

The more popular URL shorteners can also collect a lot of information about people’s usage patterns, using cookies to separate the clicks from distinct users. If they can get you to sign up and log in, they can also connect your clicks to your identity. There are definite privacy concerns with all this. URL shorteners run by bad actors can include mechanisms for infecting computers with worms and viruses before they send you on to the target site.

Of course, any URL can hide a redirect, and any URL can hide a redirect to a page you’d rather not visit. It’s just that URL shorteners are designed to hide redirects, and there are no lists of best practices for these services, along with lists of reputable shorteners that follow the best practices.

What would best practices for URL shortening services look like? Some suggestions, from others as well as from me:

• Publish a usage policy that includes privacy disclosures and descriptions, parameters, and limitations for other items such as the ones below.
• Provide an open interface to allow browsers to retrieve the target URLs without having to visit them. This allows browsers to display the actual target URL on mouse-over or with a mouse click. Of course, shortening-service providers might not want you to be able to snag the URL without clicking, because they may be getting business from the referrals. Services such as Facebook, while not shorteners, front-end the links posted on their sites for this reason. So we have a conflict between the interests of the users and the interests of the services.
• Filter the URLs you redirect to, refusing to redirect to known illegal or abusive sites. Provide intermediate warning pages when the content is likely to be offensive, but not at the level of blocking.
• Provide a working mechanism for people to report abusive targets, and respond to the reports quickly.
• Don’t allow the target URL to be changed after the short link is created.
• Related to the previous item, develop some mechanism to address target-page content changes. This one is trickier, because ads and other incidental content might change, while the intended content remains the same. It’s not immediately clear what to do, or whether there’s a good answer to this one.

Meanwhile, I never use URL shorteners to create links, and I try to avoid visiting links that are hidden behind them. I like to know where I’m clicking to.

---

Update, 11 March, this just in from BoingBoing:

Dear readers! URL shorteners’ popularity with spammers means we’ve blocked some of the big ones (at least temporarily) to cut down on the spammation. Sorry for the inconvenience! While we plan a long-term fix, just use normal URLs. You are welcome to use anchor tags in BB comments, too.

Interactive Voice Response (IVR)

I’ve been meaning to change my credit-card PIN (not PIN number, please; PIN already includes the word number) for a while now. I don’t need it to be reset... I know the current one, and I just want to change it. For whatever reason, one can’t do that from the web site, but only by calling in. Having just returned last night from meetings in Orlando, where I’ve been all week, I decided to call in.

When I connect, I first get a cheerful voice telling me the great news (their phrase, not mine): they have changed their system, and now I can speak things like my account number, my selections, and such, rather than just entering them from the number pad. In other words, they have a new IVR system.

Joy.

It suggests that I might press 2 para español (I don’t), and then asks for my account number. I choose to enter it the old-fashioned way, and I follow with my zip code when the prompt requests it. It correctly identifies my account and spends a minute or two reciting every detail about my account that it can think of, whether I want to hear it or not: my account balance, my remaining available credit, the portion of my remaining available credit I can use for cash advances, the amount and date of my last payment (along with thanks for sending it in), the minimum payment currently due and the due date. I wait all this out.

It then tells me what to say if I want to hear all that again (&deity, no!), suggests two other things I might say, and gives me a fourth choice, I want to do something else.

I want to do something else, I say.

Briefly tell me what you would like to do, it says. For example, you could say, ‘I want to change my PIN number.’ Yes, it says PIN number; waddyagonnado? But it’s funny that the very thing I want to do is the example it gives. I say, in the nice, clear voice I speak in, I want to change my PIN number, including the word number, just as prompted.

I’m sorry; I didn’t quite understand you. Not quite, you see. Almost, perhaps, but not quite. It asks me to try again, to just say a few words. I guess some folks bloviate, become logorrhetic, or otherwise confuse the electrons.

I want to change my PIN number.

No joy.

It fails on the third try, as well, and then sends me to a human, who, as they’re trained to do, apologizes for the trouble I’m having, and tells me that he can transfer my call to the PIN-changing system. Great! So he does. I wait a moment...

...and find that I’m back to the beginning of the whole process, from the Spanish prompt to the account-number prompt to the zip-code prompt, and I listen again to the account status message. It’s so nice that my minimum payment is only $24, though, of course, I would never pay off my balance at that rate. Nevermind. I again tell it that I want to do something else, I again tell it that I want to change my PIN, and it again fails to understand me thrice.

I get a second human.

I moan to this second human that the IVR system isn’t understanding me, and he offers to stay on the line with me while I try it again. This way, he can hear what’s going wrong and direct it to the right place anyway. Great!

I/we go back through the whole thing again... Spanish, account, zip, status info, do something else, change PIN, change PIN, change PIN. See?, I say, while the IVR system says it will connect me with a human operator. But my friend isn’t there after all, and in a moment a third person responds and, as the others, sympathizes with me for the trouble I’m having. She tells me that they are having problems with their system, implying that they know about it but are inflicting it on everyone anyway.

She offers me an easy solution: I can use the option numbers instead of the speech recognition. Of course, now that the speech reco is in there, they don’t list the numbers any more, but she tells me what they are: press 4, then 2. Great! She sends me back into the abyss.

Spanish, account number, zip code... but now, as it starts to read my account status to me I barge in with an aggressive 4 on my number pad, and it stops in its tracks and asks me to say what I want to do, again suggesting that I might say, I want to change my PIN number. Instead, though, now savvier, I press 2.

I’m sorry; I didn’t quite understand you. It didn’t understand the number on the pad either. I press 2 again and get the same second oops message, and a third try brings the promise of a human. It’s possible that it had understood me all along, but the PIN-setting system is what’s really broken. Human number four comes on the line, and, yes, I did get four different people.

I tell this one what happened, and he says that the only way to change my PIN is to go through the system that way — it’s so sensitive that they don’t want human operators to know the customer’s PIN (I suppose that makes sense). He tries to get me to do it again, but, feeling like a mouse in a maze or, perhaps, a Candid Camera victim, I decline, say that he should please report that the system is horridly broken, and I’ll try calling in another time in hope that it will have been fixed. He tries not to let me go, but I say, No, thanks very much for the help. Bye, and I hang up.

I should have stayed in Orlando.

Watson’s third day

I hadn’t planned to make three posts, one per day, about Watson on Jeopardy!, but there ya go. The third day — the second game of the two-game tournament — was perhaps even more interesting than the first two.

Watson seemed to have a lot more trouble with the questions this time, sometimes making runs of correct answers, but at other times having confidence levels well below the buzz-in threshold. Also, at many of those times its first answer was not the correct one, and sometimes its second and even its third were not either. Some of the problems seemed to be in the categories, but some just seemed to deal with particular clues, regardless of category.

Watson also did not have domination of the buzzer this time, even when it had enough confidence to buzz in. I don’t know whether they changed anything — I suspect not, since they didn’t say so. It’s likely that Mr Jennings and Mr Rutter simply were more practiced at anticipating and timing their button-presses by then (remember that the three days’ worth of shows were all recorded at the same time, a month ago).

Those factors combined to make Watson not the run-away winner going into the Final Jeopardy! round that it was in the first game. In yesterday's final round (category: 19th-century novelists), all three contestants (and your reporter, at home) came up with the right answer, and Watson pulled far ahead with an aggressive bet that Mr Rutter didn’t have the funds to match. Mr Jennings, meanwhile, chose to be conservative: assuming he would lose to Watson (the first game’s results made that certain), he made his bet of only $1000 to ensure that he would come in second even if he got the answer wrong.

The result, then, was Watson winning the two-game match handily, and earning $1 million for two charities. Other charities will get half of Mr Jennings’s and Mr Rutter’s winnings (whether that’s before or after taxes, I don’t know; I also don’t know whether taxes will reduce Watson’s million-dollar contribution).

One other thing: in a New Scientist article yesterday, talking about the second day and the first Final Jeopardy! round, Jim Giles makes a sloppy mistake (but see update below):

Watson’s one notable error came right at the end, when it was asked to name the city that features two airports with names relating to World War II. Jennings and Rutter bet almost all their money on Chicago, which was the correct answer. Watson went for Toronto.

Even so, the error showed another side to Watson’s intelligence: knowing that it was unsure about the answer, the machine wagered less than $1000 on its answer.

Of course, Watson’s wager had nothing to do with how sure it was about the answer: it had to place the bet before the clue was revealed. Its wager had something to do with the category, but likely was far more heavily controlled by its analysis of the game position and winning strategy. In determining its bets, it runs through all the bets it and its opponents might make, and decides on a value that optimizes its own position. And its strategy in the second game was different from that in the first

---

Update: The New Scientist article was updated shortly after it was published. It now says this:

Even so, the error did not hurt Watson too much. Knowing that it was far ahead of Jennings and Rutter, the machine wagered less than $1000 on its answer.

Watson’s second day

Commenting on yesterday’s entry, The Ridger notes this:

I find looking at the second-choice answers quite fascinating. "Porcupine" for what stiffens a hedgehog’s bristles, for instance. There is no way that would be a human’s second choice (after keratin). Watson is clearly getting to the answers by a different route than we do.

That’s one way to look at it, and clearly it’s true that Watson goes about determining answers very differently from the way humans do — Watson can’t reason, and it’s all about very sophisticated statistical associations.

Consider that both humans (in addition to this one, at home) got the Final Jeopardy question with no problem, in seconds... but Watson had no idea (and, unfortunately, we didn’t get to see the top-three analysis that we saw in the first two rounds). My guess is that the question (the answer) was worded in a manner that made it very difficult for the computer to pick out the important bits. It also didn’t understand the category, choosing Toronto in the category U.S. Cities, which I find odd (that doesn’t seem a hard category for Watson to suss).

But another way to look at it is that a human wouldn’t have any second choice for some of these questions, but Watson always does (as well as a third), by definition (well, or by programming). In the case of the hedgehog question that The Ridger mentions, keratin had 99% confidence, porcupine had 36%, and fur had 8%. To call fur a real third choice is kind of silly, as it was so distant that it only showed up because something had to be third.

But even the second choice was well below the buzz-in threshold. That it was as high as it was, at 36% confidence, does, indeed, show Watson’s different thought process — there’s a high correlation between hedgehog and porcupine, along with the other words in the clue. Nevertheless, Watson’s analysis correctly pushed that well down in the answer bin as it pulled out the correct answer at nearly 100% confidence.

In fact, I think most adult humans do run the word porcupine through their heads in the process of solving this one. It’s just that they rule it out so quickly that it doesn’t even register as a possibility. That sort of reasoning is beyond what Watson can do. In that sense it’s behaving like a child, who might just leave porcupine as a candidate answer, lacking the knowledge and experience to toss it.

No one will be mistaking a computer for a human any time soon, though Watson probably is the closest we’ve come to something that could pass the Turing test. However good it can do at Jeopardy! — and from the perspective of points, it’s doing fabulously (and note how skilled it was at pulling all three Daily Doubles) — it would quickly fall on its avatar-face if we actually tried to converse with it.

Watson’s first day

Interesting.

Watson did very well on its first day. In order to have time to explain things and introduce the concept of Watson, they set it up so that only two games are played over the three days. The first day was for the first round, and the second day (this evening) will have Double Jeopardy and Final Jeopardy.

It wasn’t surprising that there were a few glitches, where Watson didn’t fully get the question — for instance, answering leg, rather than missing a leg, in describing the anatomical oddity of an Olympic winner. And, as we knew might happen, Watson repeated an incorrect answer from Ken Jennings, because the computer has no way to know what the other contestants have said.

What I found interesting, though, is that Watson does have a very strong advantage with the buzzer. Despite the attempts to smooth that out by setting up a mechanical system whereby Watson sends a signal to cause a button to be physically pushed, and despite whatever the humans can do through anticipation, it’s clear that people just can’t match the computer’s reactions. Almost every time Watson was highly confident of its answer — a green bar (see below) — it won the buzz. Surely, on things like the names of people in Beatles songs, Mr Jennings and Mr Rutter were as confident of the answer as Watson was, and had the answers ready well before Alex finished reading. Yet Watson won the buzz on every one of those.

It was fun to have a little of Watson’s thought process shown: at the bottom of the screen, we saw Watson’s top three answer possibilities, along with its confidence for each, shown as a percentage bar that was coloured red, yellow, or green, depending upon the percentage. That was interesting whether or not Watson chose to buzz in. On a Harry Potter question for which the answer was the villain, Voldemort, Watson’s first answer was Harry Potter — it didn’t understand that the question was looking for the bad guy, even though the whole category related to bad guys. But its confidence in the answer was low (red, and well below the buzz threshold), it didn’t buzz in, and Mr Rutter gave the correct answer (which had been Watson’s second choice).

Of course, they didn’t use any audio or video clues, according to the agreement — Watson can neither hear nor see — but they didn’t seem to pull any punches on the categories or types of questions. It feels like a normal Jeopardy! game.

Oh, and by the way: the TiVo has it marked as copy-protected, so I can’t put it on a DVD. Damn. I don’t know whether regular Jeopardy! games are that way or not; I’ve never recorded one before.

Government oversight of the Internet

Now that the protests in Egypt have led to a change in leadership — an outcome that seemed inevitable for a while, though now-former-President Mubarak denied that it would happen — I want to go back and look at a key event during the last few weeks, when the Egyptian government disconnected the country from the Internet

It appears that removing an entire country from the internet is surprisingly easy, by making changes in a system known as the border gateway protocol (BGP). This system is used by ISPs and other organisations to connect to each others’ networks, so the Egyptian government just had to order ISPs to alter the BGP routing tables to make external connections impossible.

Looking at BGP data we can confirm that according to our analysis 88 per cent of the ‘Egyptian internet’ has fallen off the internet, reports Andree Tonk of BGPmon, a site dedicated to monitoring changes in the BGP. A recent report for the OECD cited the BGP as a weak point in online infrastructure that needs to be secured — a prediction that seems to have now come true.

As the report makes clear, it’s not technically difficult, at least not for a relatively small country with a relatively centralized connection to the Internet. And we see countries such as China and Iran using similar techniques to do more selective blocking (the latter has, I understand, responded to the events in Tunisia and Egypt by joining the former in blocking access to blog sites such as this one). The issue isn’t technical, but one of policy: is the government allowed to cut off the Internet?

Of course, with countries where the government makes its own authority, the answer is always Yes. But what about in the U.S., where the government was limited, at least through the end of the 20th century, to abiding by its constitution, legislation, and a judicial system?

For one answer to that question, we can look to Senator Joe Lieberman of Connecticut, who, along with Senators Susan Collins (Maine) and Tom Carper (Delaware), introduced legislation to enhance the security and resiliency of the cyber and communications infrastructure of the United States.

The Protecting Cyberspace as a National Asset Act of 2010, S.3480 (here’s a PDF of the latest version as of this writing) was introduced last June and was entirely replaced by Senator Lieberman in December (you have to go to the bottom of page 197 of the PDF to see the new version). The December version was reported to the Senate from the Committee on Homeland Security and Governmental Affairs, which Mr Lieberman chairs (and on which his cosponsors sit). It’s now on the Senate’s legislative calendar. (The corresponding House bill is H.R.5548.)

The bill, if it should become law, would create a new operational entity within [the Department of Homeland Security]: the National Center for Cybersecurity and Communications (NCCC).

The NCCC would be led by a Senate-confirmed Director, who would regularly advise the President regarding the exercise of authorities relating to the security of federal networks. The NCCC would include the United States Computer Emergency Response Team (US-CERT), and it would lead federal operational efforts to protect public and private sector networks. The NCCC would detect, prevent, analyze, and warn of cyber threats to these networks.

The bill creates, in addition to the NCCC, quite a number of offices, councils, task forces, and programs, some of which make sense and some of which probably don’t. It creates the Office of Cyberspace Policy, whose Director is appointed by and reports to the President. It creates the Federal Information Security Taskforce, comprising executives and representatives from more than a dozen government agencies. And so on.

The entire bill is quite extensive, running well over 200 pages. And what’s frightening about it is that it puts the U.S. government right in the middle of the operation and management of the Internet within the United States and its territories — and keep in mind how central U.S. operations and U.S.-based services are to the Internet as a whole. It’s difficult to understand the effect that all this new administration will have on the operation of the Internet within the U.S., and the effect that it could have if it’s mismanaged, if it tries to respond to perceived threats, if it’s affected by right-wing zealots or other dubious elements that inhabit the U.S. political community.

I have read the bill’s summary, along with parts of the bill itself, but haven’t had time to read the whole bill yet. It’s not clear how bad it could be, nor, indeed, whether it will be bad at all... but I’m very skeptical of the result of putting such a large set of deep layers of U.S. government bureaucracy in the middle of the operation and management of the Internet. And I’m deeply worried about giving authority to make operational decisions to people who have insufficient technical knowledge to understand the ramifications of those decisions, who may have political or ideological motivations that do not coincide with what’s best for the Internet, and who can implement their decisions without the checks-and-balances oversight that protects us in other parts of our lives.

I have lots more reading to do.

Jeopardy! tomorrow

Monday through Wednesday are the days when the Jeopardy! games will air that pit IBM Research’s Watson computer against former champions Ken Jennings and Brad Rutter.

My TiVo is set to record them, and it’s also recorded last week’s NOVA program, Smartest Machine on Earth (which you can watch on the PBS site). I’m eager to see how the games, recorded last month, came out.

---

Update, 15 Feb, answer to Nathaniel’s question in the comments: Ken Jennings says this, on his blog:

On Twitter, Watson (okay, his human handlers) have said that video will be posted on Watson’s website on Thursday, for those unable to watch one or more of the games live. You know: non-Americans, the gainfully employed, the Tivo-less, those with significant others expecting a romantic night out tonight instead of a quiz show, etc.

And visions of greengage plums dance in my head

A week ago, New Scientist told us about some new research technology by Toshiba, a system that recognizes fruits and veg at the self-checkout station:

Its system, developed by Susumu Kubota and his team at Toshiba’s research centre in Kawasaki, Japan, uses a webcam, image recognition and machine-learning software to identify loose goods, such as fruit. The company claims the system can tell apart products that look virtually identical, by picking up slight differences in colour and shape, or even faint markings on the surface.

When shoppers want to buy, say, apples at existing self-service checkouts they must choose the right product from a long list of pictures on a screen. Toshiba’s technology, part of which was presented last year at the 11th European Conference on Computer Vision in Chersonissos, Greece, compares the image captured by the webcam against a database of images and detailed information on the item’s appearance. The software uses an algorithm to produce a list of pictures of similar items, with its choice for the closest match at the top. If this choice is the correct one, the checkout user presses a button to confirm the purchase.

The system isn’t quite ready yet, and Toshiba hopes to commercialise the system within three years. They note, Similar ideas designed to identify products without barcodes have never made it to market in the past.

Indeed. Let’s go back to this item from 2003, where USA Today talks about some IBM research, including a system called Veggie Vision:

Researchers at IBM recently assembled several of the high-tech machines for a demonstration at their Industry Solutions Lab in Hawthorne. Among them were the smart shopping cart, a computerized produce scale called Veggie Vision, and a fascinating projection tentatively dubbed the Everything Display.

[...]

There doesn’t seem to be any controversy about Veggie Vision, a scale for fruits and vegetables that is hooked up to a digital camera and a library of hundreds of pictures of produce. When a shopper puts tomatoes on the scale, the machine evaluates their color, texture and shape to determine what they are, then weighs and prices the purchase.

Not only can it tell an apple from a tomato, but unlike some checkout clerks, it can tell a McIntosh apple from a Red Delicious.

Sound familiar? It did to me, because I knew some of the people who worked on Veggie Vision, colleagues at IBM’s T.J. Watson Research Center. And, while the USA Today article is from 2003, the conference papers about Veggie Vision, as well as the patents covering the technology, are from 1996 and 1997 (see this page for the IBM Research description, and links to the papers and the patents). It’s all there, complete with reading through the bag and machine learning.

I remember being impressed with the system (and the cool name), back when my colleagues were working on it and demonstrating it within the research lab. We had a good one, thought I, and according to the IBM Research web page, The system is now ready for prime time, and its developers have signed field test agreements with two scanner manufacturers and one company that makes self-checkout systems.

So, what happened? Why isn’t the IBM system out there at all the self-checkout stations? Why is Toshiba making the science-and-technology news for re-inventing what IBM had ready for market ten years ago? I’d hate to see Toshiba get the credit for what my IBM colleagues did so much earlier.

I have no information about that, alas... only the vague frustration that I often found, where good research projects would never seem to go where we thought they should, after they left the lab.

The thing that hath been, it is that which shall be;
and that which is done is that which shall be done:
and there is no new thing under the sun.

— Ecclesiastes, chapter 1, verse 9 (King James Version)

The Internet is falling!

The big Internet tech news this week is that the last block of Internet addresses, for the version of the Internet Protocol (IP) that we mostly use (IPv4), has been allocated. Or, as the headlines are saying, we have now run out of Internet addresses. Of course, it’s filled the tech media, as above, but it’s shown up in the mainstream press as well; here it is from the New York Times, and from The Guardian.

What does it really mean, that we’ve run out of IPv4 addresses?

Well, for one thing, it doesn’t mean that we’ve run out of IPv4 addresses. The Times gets it better than the other articles, in its headline:

The Last Block of IPv4 Addresses Allocated

The last address has not been assigned, not by a long shot. IPv4 addresses are allocated to organizations in large blocks — sometimes blocks of 60,000 or so, sometimes blocks of more than 16 million. Those organizations then assign addresses within those blocks, sometimes individually and sometimes in sub-blocks. What has just happened is that the last large block of addresses has been allocated. There are still many, many IPv4 addresses available for assignment, within many of the blocks that have been allocated.

For example, IBM has a 16-million-plus block of addresses comprising all addresses that start with 9 (that is, every address of the form 9.x.x.x; they also have some of the 129.x.x.x range). Those 9.x.x.x addresses are assigned within the company’s network. Not all of them are assigned, of course; there aren’t more than 16 million devices within the company.

Similarly, Internet service providers, such as Comcast and Verizon, have large blocks of their own, some for use within the company, and some to provide to their customers.

Many companies have blocks that are much larger than they need, far more than they could ever imagine using for their normal networks. Those blocks were allocated to them in earlier times, before the worldwide web and the explosion of Internet usage, when we never thought it would matter. Or they were assigned later, when we assumed that IPv6, with many orders of magnitude more addresses, would be well deployed by now. (I’ll note that it would be very difficult, even though large portions of the allocated blocks remain unused, to reclaim the unused bits and to reallocate them.)

Let’s not be Chicken Little, here; the sky is not falling, an the Internet is not imminently doomed. Indeed, the Internet will mostly run fine, as it is, for many years yet. We’ll all be able to read our email, buy from Amazon and eBay, use Facebook, and see YouTube videos.

Eventually, we’ll be crowded out by expanding Internet use, though we have techniques to keep that at bay for a long time. What will be blocked by this are — and this should be a familiar refrain to readers here — new applications, new uses of the Internet. To move into the future, beyond email and eBay, Facebook and YouTube, we need to move to IPv6.

We have enough IPv4 addresses for now, and for a while, to accommodate putting every computer on the Internet, as long as we’re thinking of computer as we have been: desktops and laptops. Maybe iPads, too. But now add Kindles and other eBook readers. Add smart-phones. Consider that every mobile phone is a smart-phone. Do we have enough v4 addresses for all of that?

Now move into the Internet of things: add every car, because our cars need to be online. Add every television (they’ll stream video directly), every stereo receiver (streaming music, radio stations, and other audio from the Internet), every portable music player from boom box to iPod Nano. Are we getting there? Include appliances: alarm clocks, refrigerators, coffee makers. Include home- and building-automation targets: thermostats, light switches, and so on. Put in sensor networks, traffic-control and monitoring systems....

Well, given all that, we ran out of v4 addresses long ago. It’s not really the v4 address-space depletion that should be driving the move to IPv6, but the need for more address space for future applications. If you don’t think that sort of thing is important, consider this news item about electric-grid problems stemming from the recent ice storm in Texas:

FORT WORTH, Texas — A high power demand in the wake of a massive ice storm caused rolling outages for more than eight hours Wednesday across most of Texas, resulting in signal-less intersections, coffee houses with no morning java and some people stuck in elevators.

The temporary outages started about 5:30 a.m. and ended in the afternoon, but there is a strong possibility that they will be required again this evening or tomorrow, depending on how quickly the disabled generation units can be returned to service, the chief operator of Texas’ power grid said in a release.

Consider the potential consequences of intersections without traffic signals and people stuck in elevators. We’d like to shut the power down in an area selectively, killing most of it but leaving the elevators running (at least until they open on the next floor), leaving a trickle of emergency lighting, leaving the the traffic lights running. We can do that, if everything’s addressable, and the power control system is set up to allow distribution with sufficient granularity.

But if it takes a Chicken Little scare — The Internet is falling! The Internet is falling! — to get IPv6 out there, well, here it comes.