Zigurd Mednieks's Blog, page 2

December 29, 2014

Telirati Analysis #16 Practical project management


Software projects and traditional project management tools have always been a dangerous combination. But, you can understand the attraction: Gantt and CPM chart editors can be a delightfully visual way to plan a project. Even a novice gets a thrill of insight and the feeling of control.

Project management tools were originally made for creating big, expensive physical objects, like bridges, subway lines, and skyscrapers. These projects merited hiring professional project managers, and the nature of the projects left little room for ambiguity: Either 27 floors of steel are up, or not. Either the interior framing up to the 10th floor is ready for the electricians and plumbers, or not. In this world every measurement of task completion is represented by a physical object that can be directly inspected. Almost all the tasks in these physical-world projects have been done thousands, even millions of times before. Anomalies are relatively easy to spot, and few such anomalies are dangerous to an entire project.

The mismeasure of softwareContrast this with software development: If you use the same tools as people who build skyscrapers, you are locked-in to a largely "waterfall" model of development. That's out of fashion for good reason. The plan is rigid and subject to misreported completion. Projects die with 80% of every task complete but with few of them actually finished, done, put to bed. Many of these pathologies can be traced to two differences in software creation:

Software creation is all about doing new things. unlike buildings or roads, once you have built one kind of software, you can make a billion copies at negligible cost. So all new software is really new. Each significant task has never been done before and need never be repeated. If bridges were made of tens of thousands, or tens of millions of unique, hand-crafted parts, with the quality of each part highly dependent on the skill of the maker, you would find traditional project management tools completely inadequate to the task.

It is easy to mismeasure software task completion. Module tests tell you only so much. You can be convinced a task is complete and it is only half done.

Then combine the rigidity of traditional project management methods with misuse by inexperienced but enthusiastic software team managers and you have a truly deadly mix: "All these tasks can run in parallel, right?" "Resource leveling? What's that?"

Project management made for softwareSoftware creation has bred it's own style, not to say dogma, of project management: Agile. As with other organized religions it is a canonicalization of a collection of possibly useful parables combined with a reframing of existing myths and some novel vocabulary for the priesthood to sling. Although the True Believers will tell you it's not a fault of Agile but rather the application, Agile has become notorious for allowing projects to run, incrementally, out of control, while providing the illusion of tight control.

We won't rehash all of it here, but suffice it to say that Agile has earned a backlash. Pathologies typical of Agile projects may well reflect problems in the organizations using Agile rather than Agile itself, but that's hardly an excuse, any more than the pathologies of traditional project management are an excuse for its unsuitability to software projects.

That said, Agile has key benefits:
Accessibility and inclusivity in planningAgility: adapting a plan to changed requirements is easyChange tracking, accountabilityDone is done, and what's not done is tracked
How can we forge a practical, undogmatic tool set for software project management that works in the reality of how software is created? And, how can you, as someone not steeped in traditional project management and in Agile, feel confident enough to commit the heresy of a mixed approach?

For one thing, you can benefit from our experience. We use available tools in a practical combination, with some key goals in mind:
Determine if a project is realistic, and go back to the basics of the idea if it isn'tKeep tactical implementation decisions flexibleRetain data for measuring plan-to-actual performance and other accountability metricsDon't bog a project down in replanning, but track changesThe most important overall goal is to retain the control over total project scope and cost that an up-front analysis provides while reaping as many benefits of Agile methods and tools as possible.
Toward a practical hybrid approach to project managementThese are some of the ways we work toward those goals:
Use a traditional resource-loaded CPM/Gantt chart project scheduling software package to find places where the project is resource-bound.Use resource leveling, either performing this aspect of planning manually, by using resource loading reports to identify over-committed resources, or by using automatic resource leveling if you have traditional project management software that has this capability.Find milestones that can be used to funnel multiple task completions to a choke point that must be completed before subsequent tasks are started, and get a best-estimate for total project completion. This is for planning only, not tracking.Defining a minimum viable product is very useful for planning. Take your task list and draw a horizontal line. Some tasks must be part of a minimum viable product (MVP), and some are not. Those go below the line. Make sure your project can reach a complete minimum viable product as quickly as possible. When composing task "stories" do not allow those stories to stray outside the MVP definition, unless your project has ample slack to accommodate the extra work. You won't know that until late in the project's timeline.Use a "swim lane" style task manager to run tactical resource allocation and keep decisions about resources fluid enough to not be bound to a rigid waterfall process. JIRA, in particular, has more than enough parameters and options to be configured for a hybrid project management approach.Use your up-front project analysis as a way of preventing task-by-task mission creep. If a scrum adds up to a lot more hours than your up-front analysis predicted, review each task and watch that tasks do not stray beyond the functional requirements, and if they do, capture the intent by comparing stories to the preliminary task descriptions.Keep a version-controlled spreadsheet (Google Sheets is convenient for this purpose) that tracks added and deleted tasks and their resource and time estimates.Use milestones as gates that must be crossed before earlier parts of the project can be confirmed as completed. If the milestone is not complete, the project is in a state of day-for-day slippage.When creating scrum boards refer to the CPM chart and avoid spanning major milestones with sprints.Defer resource allocation to each sprint planning session to avoid locking the project into fragile up-front predictions about who should implement what.This keeps a project nimble, not to say "agile," it provides adequate discipline before you find yourself at the end of the project schedule and are surprised it didn't all come together in the last week, and it gives me, as a consulting participant in projects, the documentation needed to take to a client when change requests add up and a re-estimation is needed, and it does so in a responsive framework that avoids re-bidding the whole project.
 •  0 comments  •  flag
Share on Twitter
Published on December 29, 2014 08:26

November 18, 2014

Telirati Analysis #15 Write Less Android Software


You have a substantial budget. You have an army, perhaps a foreign mercenary army, of developers. You've got experience delivering big Web projects this way. But your Android project might as well be Afghanistan. Unexpected limitations, difficult bugs, poor performance, and bloat plague you for weeks and months.

Android devices have become big, but Android is still for small, limited devicesSome Android devices have become capacious and powerful. The processor benchmarks for flagship handsets rival processors commonly found in PCs. The RAM capacity in most phones is still less than half what you get in a typical PC, but that's still quite a lot.

You have cross-trained some of your best Java experts on Android. They know how to engineer big Java projects. But you find unexpected problems. Why don't the same engineering approaches that work for your Web projects work for Android?

The reason is that Android is, still, a clever little OS for clever little devices. Android may run on flagship phones, but it will also, still, run on lo-spec devices with remarkably good performance. It was designed for devices with modest resources. More specifically, and although the minimum specs for Android have gone up-market since then, it was originally designed to enable multiple Java runtime instances run on the same kind of hardware Blackberry was running on 10 years ago. Many of those design elements still permeate Android's architecture and color the engineering approach needed for Android.

Many VM instances in a small systemAndroid enables a multi-process, multiple runtime-instance userland and middleware layer to run on top of a Linux kernel. If you have ever started multiple Java VM instances on  a PC and seen the resulting performance hit, this sounds like the opposite of a small Java OS for small systems, but it turns out to be the central paradox of Android. Android enables these characteristics by pre-loading classes in a "zygote" process, efficiently sharing memory with copy-on-write, limiting per-process heap size, dividing applications into small components with potentially short lives, and requiring all components to handle life-cycle events. There are cases where whole processes are "reaped" to free and compact memory. Even though Android's runtime environment implements Java language semantics, it is very unlike the environment your Web server code runs in.

Android is ill-suited to run the product of large software projects. Big libraries. Phalanxes of coders measuring their productivity in klocs. Layers of code. Abstractions. Utilities to help keep junior coders out of trouble. All these activities produce bloat, encourage monolithic applications, and hide problems. Especially so on Android.

Bigger heaps are not the answerAndroid has an aggressive memory recovery strategy built deep inside the Android architecture. Is Android's architecture obsolete in an age of multi-gigabyte, multi-gigahertz flagship phones? The answer is No for two at least two reasons:
Android has a very wide range of hardware targets. For every flagship phone and tablet, dozens of lo-end phones will be sold. Apps will go into embedded devices in cars, home appliances, industrial computing devices, and non-phone consumer electronics devices.Java style garbage collection works only within one process. Android's memory recovery strategy implies that the larger the maximum heap size, relative to total RAM, the more often processes get all their components destroyed (and subsequently reconstituted in a new process) in order to recover memory system-wide. There is a sweet spot for maximum heap size and it's a relatively small fraction of total RAM.Moreover, Android apps have to work across the whole range of heap sizes in use for the Android devices targeted. Limiting your app to high-end devices just to accommodate your engineering approach isn't a solution.

Were Android's architects just messing with you?Multiple (dozens!) of processes, each an instance of a Java runtime, each with a limited heap size... is this an environment created specifically to be mind-bending to the typical server-Java engineer? You may very well think so. However, where Android imposes constraints in some dimensions, it offers opportunities, and novel features in other ways: Android has an unusually rich architecture for inter-process communication and remote procedure calls.

Write less code, in smaller modules, and embrace the Android archiectureThe way to avoid unpleasant discoveries, such as Android's per-APK method limit, heap size limits, etc. divide your software into a suite of separate but communicating modules, and divide those modules among multiple APK files. Moreover this is an opportunity to structure your projects along these lines: Smaller projects, smaller teams, and abstractions that follow the contours of the Android system. Avoid libraries that are redundant functionality provided by the Android OS. If the choice is portability-with-bloat vs more porting effort, the porting effort will be the cheapest investment you make to fight bloat.

If you have data that multiple modules might need, use ContentProvider components to share it. If you are rendering large images, say for an e-reader, do that in a process and heap dedicated to that part of your application. Break down your monoliths, and don't try to subvert Android's limitations.

Facebook's approach to unbundling functionality makes Facebook's products more visible on mobile devices. Keeping your apps simple, relatively small, and having multiple cooperating apps isn't just smart engineering, it's better for the user and better for you commercially.


 •  0 comments  •  flag
Share on Twitter
Published on November 18, 2014 14:10

July 10, 2014

Telirati Analysis #14 Google's Social Menagerie and its Android and Web Habitats




N.b.: The tables in this article are available as a single table here

Welcome to the menagerieWho knew how social Google really is? Google has at least nine properties that can be considered "social." Back in our analysis #11 we took a quick inventory and found that social characteristics permeate Google's collection of tools and applications, and that Google+ missed some key opportunities in high-value areas.

Here we expand our collection of specimens and apply taxonomy to Google's ecology of things with social characteristics and how well adapted they are to Web and Android habitats. We divide the analysis into two parts: content characteristics and social features.

Content characteristicsIn addition to search, email, and office productivity, Google runs at least nine "applications" that deal in user-provided content. We can't call them "Web sites" since most of them are presented through both Android applications and a Web user interface. But the extent and quality of this presentation is unequal and uneven. The content varies by media type and long form/short form characteristics. The intended persistence of the content also varies, though persistence often really means "ease of discovery" which can diminish quickly if a chronological update stream is the principal means of discovery.

Content CharacteristicsPresentationContentPersistenceGoogle+Web-first, Android app has a subset of functionalityShort-ish form, longer than twitter, shorter than a blog postShort persistence - blink and it's goneOrkutWeb-first, has a really bad Android appShort to medium formShort persistence - blink and it's goneYouTubeWeb-first, Android app has a subset of functionalityVideoMedium to longPhotosNear-parity between Web and AndroidImages, storiesLongDriveWeb-first but Android at near-parityLong formLongBloggerWeb-first, Android app has a poor subset of functionalityMedium to long-form, essay-like articlesLongCodeWeb-onlyCodeLongSitesWeb-only (duh!)Multi-page, long persistenceLongGroupsWeb-onlyShort to medium formMedium to long (pinned)
Social FeaturesYou can think about managing social applications as an exercise in applying common tools along common axes of characteristics and functionality. Does an application lack something basic like an update stream? Even code-oriented applications like JIRA and Github have update streams. If Google Code deserves to survive, it will have to catch up to competitors by implementing those social features.
Across all properties with user generated content and social characteristics, Google can gain an advantage over competitors by applying common social tools, such as an update stream, threaded comments, and user management features with a more-uniform level of sophistication. This means breaking up monolithic implementations and applying the modules to applications. An obvious example would be to make Blogger the long-form presentation variant for content that's too long and too persistent for Google+. But this principle can be applied to applications as varied as "social coding" and Web site hosting.
Social FeaturesUpdate streamDiscussion and commentsUser and group managementGoogle+Big, fast update stream, manageable with circlesReplies, but not branching threadsFlexible circle management integrated with Google address bookOrkutFast update streamLinear comments Friend listYouTubeWeak Linear comments, now integrated with Google+ commentingCan follow users/channels; Users frquently maintained separate identities for YouTubePhotosFull (too-tight) integration with PlusPlus commentsPlus circlesDriveNoneCollaborative editing, comments embedded in documents, document sharing in hangoutsDocument sharingBloggerWeak. Can "follow" blogs, but it's a different "follow" than in Google+Weak, but optionally integrated with Google+NoneCodeNoneIssue tracking and discussionEverything publicSitesNoneWiki-like comments and attachments are optionalSites can be private to a groupGroupsChronological view of posts, post-by-emailRich threaded branching discussionsGroup management not integrated with other Google properties
DiagnosisEach of Google's properties needs to be adjusted in different ways to fit into a grid of social characteristics and tools that extract maximum utility for those characteristics. Google is well-placed to do this, and a rational, consistent approach is what Google often, but not always, converges on in the end.
NeedsGoogle+Disaggregation into separate photo and social update appsOrkutWill be EOL'ed in SeptemberYouTubeAutomatic integration of videos into the Plus update stream, consolidation of follower features with PlusPhotosNeeds a little more separation from Plus, a separate update stream of shared imagesDriveDocument updates should be posted to an update stream visible to BloggerIntegration of long form content discovery with Plus, perhaps in the form of a seprate content streamCodeNeeds to be EOLed but for the huge potential to show what "social coding" could be like in the googleverse.SitesNeeds a feature upgrade to make it competitive with first-tier Web site building tools like SquarespaceGroupsAdding "pinning" to Plus could integrate all of Groups functionality into Plus
TreatmentGoogle needs to apply disaggregation, combination, and cross-domain implementation of a common set of social functions. Photos and YouTube are media-specific views into a common social update stream, but with variants in presentation that account for discover-ability requirements. There are also opportunities to combine the way some data is stored. Groups, for example, may boil down to a variation of how Google+ content is acquired and presented.
It wouldn't be worth doing without an  upside. The measure of the upside is to be found in the fact that many domains that seem unrelated can all be enhanced using social media tools. A deeper analysis of this form should be able to tell Google's product managers what to keep and what's in a hopeless competitive situation, and, if something is worth keeping, what are the resource requirements to make it first-rate.





 •  0 comments  •  flag
Share on Twitter
Published on July 10, 2014 04:53

June 23, 2014

Telirati Analysis #13 Missed It By Thaaaat Much: Why Chrome OS Athena isn't Chrome OS for Tablets

It's hard to drag a legacy UI into a touch worldAs Microsoft well knows, it's hard to drag a legacy UI system onto touch devices successfully. Microsoft tried numerous times, notably in the past for the UMPC format and Windows mobile, and more recently in Windows 8.1 to evolve Windows into a touch OS. or to bifurcate it and leave the legacy UI behind. Neither approach worked.
Touch and the browserMicrosoft has, however, dragged OEMs into building touch laptops. Google acknowledged this trend by creating the Chomebook Pixel with a touchscreen. Google felt they needed to "own" the issue of touch and Chrome in case touchscreen PCs were successful. Chrome OS and the Chrome browser, as well as Google's Web apps would have to adapt to a world with lots of touch systems running Windows. Chromebook OEMs are following Google with their own touchscreen products in laptop form factors. But that never happened and touch in Chrome OS has become a stagnant and underdeveloped capability.
Dubious ergonomicsIt's an open question if laptops are any good for touch. The screen is an awkward distance from the user. The laptop hinge can't resist the pressure of touch near the top of the screen. You can't pick up a touch laptop and wield it like a clipboard. Software aside, there are good reasons for the idea of a touch laptop to fail.
An accidental featureWith a track record of failure and obvious ergonomic issues, touch in PCs is a kind of accidental feature. It propagates among products via weak product management that's fearful of competitors' bad decisions. Nobody got fired for aping a larger competitor. So touchscreen laptops, and even large touchscreens in desktop all-in-one PCs ripple through the industry like the echo of a Rick-roll.
Athena is comingNow that Windows 8.1 has landed with a thud, and OEMs are kvetching about being misled by predictions about the success of touch, there are some signs that Google has decided to take another shot at touch in Chrome OS. Or, perhaps, the Chrome OS developers are just tidying up loose ends in the Chrome OS touch interface, of which there are a considerable number at the time of this writing, based on a search of open issues mentioning "touch."
As of this writing, Athena is the next version of Chrome OS. Chrome OS is developed in the open, with the issue tracker for the project open for anyone, friend or foe, to examine. This enables speculation by pundits on topics like touch. But it also enables a deeper examination of the Athena feature set.
More than touchIn addition to touch, Athena is a further evolution of Chrome OS's UI stack and desktop. Under every "Web OS" is a limited window system that enables menus and title-bars and an inevitably ever-growing set of UI widgets. Athena ads support for a card-like UI in the widget set that, in turn, supports added UI features in Chrome OS.

Indeed Web UI, in general, needs to be evolved toward app-orientation at least as much as touch interactions need evolving. One reason mobile apps have taken off and outrun Web apps on mobile devices is that native UI is not just more touch friendly, having been designed to be so, it is generally more amenable to direct manipulation and other GUI convention and design idioms.

Chrome will continue to adhere to browser standards, but, within those standards, will adapt itself to making it possible for apps to better-integrate individual app experience with Web browser UI.

Evolution, not revolutionAs much as improvements in touch and general UI interaction are needed for Web apps to catch up to native mobile apps, those improvements will be the result of evolutionary change. It is unlikely, for example, that Athena will result in Chrome OS tablets. more likely is it that Athena will make Chrome OS better suited to touch laptop form factors, which are themselves an evolutionary step on their way to some other destination.

The lesson for developers in this is that Web and native platforms are of roughly equal importance, though hardly any developers act as if they truly believe that. Web interfaces continue to outrun mobile interfaces for features, because that's what backend designers design for, and mobile apps outpace Web apps for direct manipulation, fine-grained control, and concise, consistent interaction.

Neither Web nor native are going to dominate mobile applications within the next 5 years, never mind consolidating a new paradigm for laptop and desktop UI. Implement for both native and Web UI as if they are your primary platform.
 •  0 comments  •  flag
Share on Twitter
Published on June 23, 2014 05:13

June 20, 2014

Telirati Analysis #5 An Android-based Asha?

One of the little-known projects inside Nokia before all Linux-based OSs were eradicated by Stephen Elop was called Meltemi, a lightweight Linux-based OS for Asha phones. The project made sense: As we have seen from declining sales numbers for Asha, a Series 40-based Asha could only be a stopgap. But, for Elop, any alternative to Windows endangered his strategy, so Meltemi had to die.

With Elop gone and Microsoft not yet having found the org chart, Nokia's R&D elves have come up with what is reported by Ars Technica to be an Android-based Asha OS.

Technology-wise this makes a great deal of sense: After a few generations of adding features, scaling up to tablets, and adding visual "buttery smoothness," Android 4.4 trimmed some fat and shipped a preview of a more-efficient VM.

So if you can't beat a lighter, faster Android, you might as well join the technology platform, as Amazon did long ago. Nokia is well-placed to deliver an Asha-ized AOSP, with their own maps, media ecosystem,  app store, and other platform elements.

This also makes sense based on what is known about the Asha team's leadership and previous explorations. With some cross-fertilization from Jolla, Nokia had been looking at Android compatibility for Asha for a while now, and it looks like they decided to flip that over and build Asha, and Nokia ecosystem compatibility, for Android. That follows logically from the fact that creating Android compatibility on the S40 OS would have been unfeasible.

What can we predict will happen? Like Danger and Kin, this product is likely to be doomed within Microsoft. It will arrive on Microsoft's doorstep bundled in with a grab-bag of dying products Microsoft acquired along with the rest of Nokia's handset business. It will be a management mess to clean up, with Ballmer leaving the buyer's remorse for the next guy. Hey new guy, know what a Series 30 is? You own it!

And what don't you own? You don't own Nokia's location services, maps, media distribution business, and other elements of a mobile ecosystem, and, except for  a 10 year brand license for low-end phones Microsoft would probably rather not have to deal with, you don't have the Nokia brand, either. In other words, Nokia unloaded their legacy products, completely irrelevant to anything at Microsoft, and including the whole manufacturing infrastructure for them, on Microsoft, as Ballmer had already announced his departure and Elop was engineering a fall upward and back into Microsoft.

This project is another indicator that once Nokia is no longer contractually kept out of the handset business, it could readily rebuild a first-rate handset product line backed by a formidable mobile ecosystem. With Jolla operating outside of these agreements, but within an existing investor relationship with Nokia, there is also plenty of scope to fudge the milestone dates on when Nokia can do what. Don't be surprised if the next place you see this idea pop up is at Jolla.

 •  0 comments  •  flag
Share on Twitter
Published on June 20, 2014 21:57

Telirati Analysis #7 2014 predictions

30+ megapixels under $500: As low-end cameras get kicked out of the market by phones, all the effort goes into cameras that can't be put into phones. This is already happening and as this trend matures, price/value competition will become fierce and specs now found on expensive luxury/prestige cameras will end up in products with affordable prices. By 2015, this will include specs found only on insanely expensive studio equipment.

Consumer pricing (albeit at the high end) for 4k TVs and 4k video cameras by the end of 2014: Content publishers will scramble to keep up. Or, to put it another way, there won't be anything to watch. Some will say that makes 4k a failure. It won't be because passive media availability isn't the deciding factor.

Cinema, photography, and related media will start pulling away, qualitatively, from the analog past. Except for laser enlargers and large format color prints on photographic paper, analog and chemical imaging has lost the last bastions of quality. As digital quality continues to improve for subsequent product generations, creators of media product will see new forms of expression, as yet unexplored.
Wearables do not find a mass market product format: But the quantified self moves forward as an identifiable mass market app for wearables. Apple makes something, maybe a watch, that will be an object of desire, but hard to justify based on actual benefits. Apple will find that digital "jewelry" doesn't hold the same kind of appeal as adornments that don't become obsolete in a few years. If you own a fine watch, consider how old it is.
Android in cars, cameras, and perhaps other products creates new domains in software development. Google will take advantage, but could miss opportunities by not developing APIs for new "verticals" for Android.
Android becomes vastly bigger and more important in software development than nearly anyone expects. The realization that Android will be as big and long-lasting in dominance as Windows begins to dawn on many more people.
Sales lost by US tech firms due to lack of trust exceeds worst-case estimates: Some tech firms become, essentially, government contractors. Some dither and lose even more. Some new entrants offer privacy-protecting products, which will remain an embryonic, but highly visible business. What to do about the knowledge of pervasive surveillance and "Big Brother Inside" remains muddled at the end of 2014 as lost sales mount. Most tech companies will hope the problem goes away. It won't. By the end of the year, a non-US Internet service company will offer  verifiable secure communications and storage.
Sensor-oriented Bluetooth devices will flourish and at least one that doesn't exist today will be innovative and become very popular.

The Internet of Things will see slow adoption due to privacy concerns.
The oil/gas boom in the US will paper-over many problems, as long as it lasts. Everything from avoiding a hard landing in winding down QE, to the Democrats' being blamed for the economy, to avoiding dealing with the many ill effects from the cost of the wars will be made much easier by the US oil boom. Things will be just enough better to not learn anything from the wars and the 2008 crash. 
Social liberalism in the US will march on: Gay marriage, legal cannabis, Obamacare and other progressive issues will all be more popular at the end of 2014. Hard-line social conservatism will harden and become more culturally isolated. The US will be less of one nation and one culture. A kulturkampf looms, but Republicans will try to steer clear and win based on a tepid economy.
Crypto-currency meets sovereign governments: A US Native American government is considering experimenting with crypto-currency. It's possible that a nation with a currency in crisis will take it to the next level and adopt crypto-currency as an economic confidence-building measure, or will strongly consider it. It will be compared, eventually favorably, to "dollarization" or other measures that had been taken in such situations.
Transitioning from oil, gas, and coal remains a daunting task, and we will barely move the percentages in 2014. Progress in inherently safe uranium/plutonium reactors and throium reactors will be disappointingly slow. High-tech renewables are growing fast, but from a much lower base than nuclear. The most visible positive sign in the US might be the closure of a handful of dirty old coal plants. Don't mistake renewables' share of electricity generation for fast progress. Transportation is a big energy cost in households and industry.
 •  0 comments  •  flag
Share on Twitter
Published on June 20, 2014 16:08

Telirati Analysis #10 To Change the Terms of the Privacy Debate Protect All Bits

The trust problemUS technology and Internet services companies have a deep trust problem. They are accused of collaborating with the NSA and, on top of collaboration, being exploited by the NSA. The NSA, in turn, is seen as operating without boundaries, turning America and much of the world into a glass-walled panopticon, devoid of privacy and confidentiality.
This loss of trust has already cost tens of billions, and will cost tens to hundreds of billions more in lost sales outside the US and the "Five Eyes" nations most closely collaborating in NSA surveillance. Any nation that aspires to have practical sovereignty, competitive industry, and independent decision-making finds they cannot trust US technology and services.
Solving the trust problem is one of the most valuable goals in the technology and Internet services industries, and it has proved to be a sticky problem. The key may be to change the terms of the discussion.
Describing the threatThe NSA has taken most of the headlines, but it isn't the only threat to privacy. Without understanding the whole threat, some people may conclude that they trust the US government and/or the NSA, and have "nothing to hide."

This approach ignores the non-US state actor threat and the criminal threat to data and communications security. In corrupt places, the criminal and state actor threat are combined, and there is nobody to trust. Where laws mostly work, they offer only variable protection, and none offer absolute protection against the state, and no laws restrain foreign threats.
The bottom line is that you can't rely on a service to protect you, and you can't rely on laws to protect you. You have to protect yourself.
The role of technology and service providersThe key to regaining trust is to enable individuals to protect themselves. The role of technology and service providers in this is to support individuals' ability to protect themselves. Trust can't be regained directly. It must be earned back by providing tools for privacy.
Tools for individualsTo earn back trust, technology and service providers have to enable "end to end" security that is fully controlled by individuals and enterprises. Some say this would be hard to use, but services like Skype provided a high level of security while growing on the basis of the best ease of use in their product category.

There really are no excuses for not enabling individuals to have simple access to privacy and security, and the ability to deliver high security and ease of use have only improved since Skype was introduced. For example, a "web of trust" removes the need to trust an authority to anchor a chain of trust in the identity of the person you are communicating with, and the validity of their public key.

Social networks provide a means to distribute public keys. Ephemeral keys and "perfect forward secrecy" (PFS) remove the need for individuals to manage keys for real-time communication.
A solution for individual privacy and security must include these elements:Open source client software for communication and data storageThe use of ephemeral keys and perfect forward secrecy in real-time communicationA web-of-trust for public keys and a convenient key signing and key exchange mechanismThe public has to be confident their software does not include back doors. It has to employ simple-to-use technologies where possible, and it has to make the more-complex aspects of security as simple and powerful as possible. These goals are within reach of all major Internet services and technology providers. By reaching these goals, technology and service providers will earn users' trust.

The effect of end-to-end encryption and related technologies that remove the need to trust the operators of networks and the equipment makers who built it is to reduce the value of mass surveillance. By encrypting all personal communications and personal data, for everyone, all the time, the cost of extracting that information by more powerful tools becomes impractical to apply at a mass scale.
What is a sufficient solution?Protecting against a sophisticated state actor threat is a daunting task. The NSA actively subverts security technologies. The public can't verify proprietary security technologies. Security agencies worldwide stockpile vulnerabilities and buy them from hackers across the black-to-gray spectrum.
But protecting privacy isn't an impossible task. The ability of state actor and criminal hackers to take advantage of vulnerabilities is limited by independent discovery of the bugs enabling those exploits.  The lifespan of most vulnerabilities is in the range of a few months to two or three years. Many vulnerabilities are only suitable for targeted attacks and cannot be scaled-up for for mass surveillance without being quickly detected and fixed.
Defense against vulnerabilities must be defense in depth. Vulnerabilities will never all be fixed. Other tools, like intrusion detection and postmortem analysis tools need to be developed in the open so that they can be trusted to work against all classes of threats. Enterprises that make use of open source software should form cooperative organizations to test and audit that software and fix vulnerabilities.
A sufficient solution consists of:Finding vulnerabilities and reducing the number of vulnerabilitiesDetecting threats and intrusionsEnd-to-end encryption of all data and communication
The most valuable secret of surveillance is that it mostly depends on data being weakly defended and available in cleartext. If all data is encrypted end-to-end and never available in cleartext except at the intended recipient's system, and all systems are secured to a high standard, we can have privacy, confidentiality, and security in communications and data storage.
America's blind spotAmericans, and even American corporate leaders with plenty of international exposure now have, and are likely to continue to have a blind spot regarding the severity of the trust problem they face.
Snowden's files, and subsequent developments such as the allegations that NSA knew of and exploited the Heartbleed bug, have put the US government and the US-based technology industry in disrepute worldwide. 
You might think a problem that large would have set off alarms. But the response of US equipment and services companies has been timid: Some have issued some indignant press releases. Some have participated in proposing reforms that have so far failed to fill even a teaspoon full of the credibility hole. Some have touted wider use of SSL, while retaining access to your data in cleartext. So far, the only major Internet service to have even floated a trial balloon by means of a trade press rumor is Google, who are said to be considering implementing end-to-end encryption for GMail.
Many Americans, even those with exposure to and experience in international markets live in an "America Bubble." This bubble is made of kind assumptions about the American government: The NSA and FBI protect us. They catch Bad Guys. Some of what they do is a direct service to American businesses: catching credit card frauds, for example.

The fact is that spy agencies and law enforcement have numerous tools other than mass surveillance. Among Snowden's revelations one finds that the US government has extraordinarily subtle listening devices and transmitters available for high-value cases. Ending mass surveillance won't take away from these high-value tools.

The only way to win is to not play the gameIt has been a year since the Snowden revelations, and US technology companies have not taken the required steps to regain trust. At both the national and industry level, the only way to regain trust is to not play the conventional game of laws and treaties and weakly protective technologies. By securing users' data against all threats, the terms of the negotiation are changed and the current deadlock can be broken. State security apparatuses will only re-think mass surveillance in an environment where mass surveillance is less valuable.
While many nations have surveillance operations in their state security mechanisms, some nations apply vastly more resources to these operations than others. The US, for example, spends more on its military than the rest of the world combined. Spending on the NSA and other signals intelligence is likely to be proportionate to military spending overall.

If some nations come to realize they can't compete with the NSA, they will then conclude they must change the ground on which the game is played, both to secure their sovereignty, and to secure the competitiveness and trust in their technology industries.
It is an open question when the US technology industry will take affirmative and effective steps to regain user trust, or whether the US will end up importing that approach from outside after a painful lesson in lost business. The cost to US industry is high and mounting. Likewise the frustration with the US among its allies, not to mention non-aligned nations is also mounting. Some nation's political and business leaders will say "Enough!" and decide that the best way forward is to provide people with the means to have privacy and confidentiality.

Technology alone cannot give us a system of laws, treaties, and security mechanisms that respects privacy, but, by making it harder and less valuable to violate privacy on a mass scale, technology can change the terms of the political debate, and steer it toward a better outcome. Not just for Americans, but for everyone living with a too-intrusive government.
 •  0 comments  •  flag
Share on Twitter
Published on June 20, 2014 16:06

June 10, 2014

Telirati Analysis #12 Intel and Rockchip: Why Intel Isn't Inside Your Phone

SoCs are mostly the sameOne of the mysteries surrounding the question of why Intel hasn't gotten inside your phone is that systems-on-a-chip (SoCs) are all quite similar. Some CPU cores, a GPU, and i/o peripherals share a bus, all on one chip. If high-level architecture is so similar, and apparent barriers to entry low enough that numerous new entrants have flourished, what's up with Intel?
A combination of factorsIf new entrants can get in the game, and SoCs are difficult to differentiate, it is unlikely that you can find one factor, or one problem to solve, that will unlock the mobile market for Intel. Here are some of the factors cited over time as keeping Intel out of your phone:
Power management: ARM CPUs were designed for low power applications and ARM licensees have all the expertise in integrating systems that result in low power SoCs. Intel has largely overcome this barrier, though it took more product generations than it should have, and Intel may still trail NVidia in bringing big GPUs to low-power SoCs.
Windows and netbooks: Netbooks are relatively higher margin products, Intel knows all the netbook OEMs, and netbooks have bigger batteries and can tolerate higher power consumption. The Intel instruction set architecture and Microsoft's Windows technology strategy kept ARM-based competitors out of the mainstream Windows 
GPUs: Intel hasn't brought their own GPUs into low-power SoCs. Currently, Intel uses Imagination Technologies (IMGTec) PowerVR GPUs in Atom chips. There may be an IP licensing price mismatch between IMGTec and the kinds of OEMs who might first bring an Intel SoC to market in their device. There is no open source driver for PowerVR GPUs, which limits the experience Linux developers have with that GPU. That said, PowerVR is in Apple's mobile products and many other ARM-based SoCs. There is nothing apparently deficient about that GPU architecture. IMGTec recently acquired MIPS and looks like they will get in the CPU licensing game.
Performance: Benchmarks for Intel Atom CPUs have always been competitive with ARM CPUs, even though the fastest ARM CPUs, like Apple's A7, outperform the top Atom CPUs. So raw performance numbers can't be the whole answer. The rise of modestly priced, medium-performance products like the Motorola G further reduces the importance of ultimate performance. Toolchain issues are also unlikely to holding Intel back, so while one can conclude that performance is part of why Intel is not in your phone it is not likely to be a major part. Performance may become a larger factor in combination with GPU performance if NVidia is successful with the K1 chip. General-purpose GPU computing is another factorin the performance equation. Renderscript is a little-known and little-used technology that 
Price: The question of price by itself embodies the complexity of why Intel has no significant mobile wins. ARM licenses their designs, as do mobile GPU designers. That has proven to be a powerful business model. Samsung and Apple are vertically integrated, integrating their own SoCs. ARM SoCs are often fabbed by contract manufacturers, except for Samsung, which operates their own fabs and also makes Apple's chips. And despite the vertical integration at the top of the market, there are low-cost fabless ARM SoC vendors. While prices are obscured by the difficulty of measuring transfer pricing in large users of CPUS like Samsung and Apple, you can conclude that diversity of supply means intense competition and that Intel has a difficult time convincing OEMs that they will be the low-cost vendor over the long haul.
Intel and RockchipIt may seem hard to figure out why Intel entered into a relationship with Rockchip to produce SoCs. But it's a hard multidimensional problem they are trying to solve. Getting an experienced maker of low cost ARM-based SoCs that's got significant design wins involved is an approach that could solve more than one problem at a time. It's more alchemy than science, but Intel has much to gain and nothing in their history that would be get worse if this approach doesn't succeed.
One factor that isn't part of the answer is investment. Intel doesn't need Rockchip's money and Rockchip isn't taking any Intel money, expecting to make back their cost of development through selling this chip through Rockchip's own channels.
Both companies will sell the product, though it will be branded as an Intel product. This is significant because Rockchip has mobile handset customers and Intel doesn't. 
Another factor that can be eliminated from the answer is that Intel will license SoC CPU cores. This isn't an intellectual property license agreement in the same way that ARM licenses their designs, and it isn't being replicated with other SoC makers. Nor is it a foundry agreement. Rockchip is fabless.
IMGTec's venture into CPUs may also have motivated Intel to find a partner that can integrate GPUs other than PowerVR. Rockchip currently uses Vivante GPUs. But this does not necessarily augur a long-term relationship. Especially so if the next step on the road map is the integration of Intel's own GPUs into mobile chips.
Another indicator this may be just a bridging partnership is that the planned SoC will have a 3G radio. That may not be competitive is markets outside China and developing world markets, which are Rockchip's current target markets.
Not to say that Intel is exploiting Rockchip's customer base utterly without a road map for future collaboration, but it does appear that Intel sees this as an experimental way to acquire first customers in the mobile handset domain.
Strange bedfellows, and a limited relationship that leaves the future open for reconsideration. But this may be the way we get to see Intel-powered handsets in China. It may also be the way Intel learns to be competitive in a domain that has eluded them for complex reasons.
 •  0 comments  •  flag
Share on Twitter
Published on June 10, 2014 07:07

April 28, 2014

Telirati Analysis #11 Diagnosing and Fixing Google's Social Problems

Photo by Matthew Hester (CC BY-ND 2.0)
Google has become something of a slumlord. While Google+ has been accused of being a "ghost town," at least it looks pretty, even now after what feels like a long period of stagnation and un-addressed bugs. But Google+ isn't the most neglected neighborhood in Googleland.

Where is the Blogger blog?Various parts of Google, notably Search, use Blogger to convey news about new releases and their development road map. Blogger is key infrastructure for Google itself. But where is the Blogger blog? Like some other semi-abandoned properties, Blogger no longer has frequent updates about features.

If you want a history of Blogger and it's features, you'll have to rely on Wikipedia. Evidently there are ardent Blogger users who keep track of these things.

Why does this matter? There are a lot of abandoned places on the Internet. Blogger, however, is emblematic of the problems that precipitated the departure of Vic Gundotra from Google+. Blogger and Google+ have a very valuable natural symbiosis. Potentially, Blogger and Google+ could unify long-form and short-form writing on the Internet. But, in reality, the execution fell short. The Google+ comment thread system was grafted onto Blogger, which needed a replacement for its own weak and spam-riddled comment system. But this falls far short of actually solving the problem of creating a unified space for discovering posts and discussions of varying complexity and lifespan.

As we will see, this isn't the only missed opportunity.

Let's make a Group and discuss itYes, there is another social, post-centric, discussion thread tool at Google: Google Groups. It's easy to see why Google Groups is still used: You can get a concise listing of threads, organized by topic, with some particularly timeless threads "pinned" to the top of the list. Just the thing for discussing, among other things, features users want to see in Google products.

In 2013, the Groups announcement blog announced there wouldn't be any more announcement blog posts because they had become too rare. So we are left with a situation where I can't get a threaded view of a discussion on Google+ and I can't get "bell" notifications from Google Groups. Slummy.

2009 Called...Sites is another abandoned-looking amusement park in Googleland. Stagnant functionality. Outdated templates. No blog. Despite Google having properties like Wallet, there are no easy e-commerce options for Sites. It's just a middle-of-the-pack Web site builder with a wiki-ish flavor to it. Web site builders is a field in which there are less than a handful of first tier players. Therefore being "middle of the pack" means you're not in good company.

Braziiiiillll!!!We got this far, without mentioning Orkut, a social network operated by Google, with minimal integration with other Google properties (e.g. no "bell" in the strip at the top of the screen), that has a Turkish name and is big in Brazil and big-ish in India. With presumably limited resources, the look and feel of Orkut remains charmingly MySpace-like.

If Facebook is expanding internationally, how does an opportunity to integrate an international audience into Google+ keep floating out of reach?

YouTube is like Instagram for video, and should have been treated as gentlyYouTube is a social network. But here Google should have taken a tip from Facebook. YouTubers are mostly not Plussers, in the way that Instagram's hipsters are not Facebook schoolgirls. Facebook had the good sense to leave that situation alone. Instead, Google+ enforced strong identity on a trollish population. It's easy to see why that would be a thankless task, and a dumb move in an attention economy where "thanks" is the currency.

It's the product managementProduct management in technology companies is a distinct function that sometimes gets rolled up into program management, project management, or even engineering management.  But many of the above listed problems are the result of product management failures. The product manager is sometimes called the "product owner." He is meant to make the product competitive against specialist competitors, and balance the interests of "his" product against the sometimes abstract advantages of integrating with other products under the same roof.

In the egregious cases listed in this post, even the basic functions of competitive analysis seem to have been lost. That is either a product management failure or a senior management failure in deciding to keep products while starving them below a competitive level of resources. Lackadaisy or willful neglect, or both, are what turns some products into Internet "slums."

Let's take a property inventory:Let's do a little product management exercise and take an inventory of what Google owns, and the relevant characteristics. We'll leave Orkut out because it's largely redundant  in every characteristic to Google+.
ContentPersistenceUpdate streamDiscussion, commentsUser. group, managementGoogle+Short-ish formShort persistence - blink and it's goneBig, fast update stream, manageable with circlesReplies, but not branching threadsFlexible circle management integrated with Google address bookBloggerLong-formLongWeak. Can "follow" blogs, but it's a different "follow" than in Google+Weak, but optionally integrated with Google+NoneSitesMulti-pageLongNoneWiki-like comments and attachments are optionalSites can be private to a groupGroupsShort formMedium to long (pinned)Chronological view of posts, post-by-emailRich threaded branching discussionsGroup management not integrated with other Google propertiesYouTubeVideoMedium to longWeak Linear comments, now integrated with Google+ commentingCan follow users/channels; Users frquently maintained separate identities for YouTube

What does this table suggest? The main problem that jumps out at you from this table is that comment integration is weak integration. It doesn't deliver a lot of value and it doesn't solve the key problems for user-generated content. Meaningful, useful integration that improves the user experience must be deeper integration. All these properties have social characteristics, but comment integration is peripheral to social-centric services.

An MVC model for social propertiesWith Orkut, Google has six social properties, at least, before we get into things like social documents and social coding. With Google+, social became a pariah within Google for taking on the wrong integrations with the wrong products in the wrong order. Comment integration was weak, and with it came identity integration that was, in the case of YouTube a source of deep dissatisfaction. It's bad form, and just plain a bad idea, to go an mess with another product's community.

Integration should have started deeper, with the content database and update stream. Then with a gentle, optional, merging of identity, and adding the group-management features of Google+ Circles. Comment integration is then a minor issue. Each product can retain a distinctive set of views outside the chronological fast-flowing river of the update stream, thereby both merging and keeping distinctive the persistence characteristics of each property. You can think of that as multiple views into unified content, or a "model/view/controller" (MVC) approach to viewing the same model through multiple views.

The opportunityGoogle's biggest opportunity is to create the best user experience for user-generated content across Web sites/wikis, discussion group, blogs, videos, images, and the social update stream, and along with the content continuum, integrate (carefully!)the management of participants and their comments and collaborative participation. Almost everything has a social aspect, but Google somehow missed the deeper insight once Google+ latched on to comment integration as the feature it would take across properties.

One more thingOne more thing about Google+ and integration with other Google properties: Android. GMail is great on Android. The GMail Android interface is richer than the Web interface. Maps is magical on Android. Keep is simple but awesome on Android. And what's the big thing in social networks now? Mobile. Google+ is mediocre on Android. That's not how to make a product succeed at Google.
 •  0 comments  •  flag
Share on Twitter
Published on April 28, 2014 11:14

Telirati Blog #11 Diagnosing and Fixing Google's Social Problems

Photo by Matthew Hester (CC BY-ND 2.0)
Google has become something of a slumlord. While Google+ has been accused of being a "ghost town," at least it looks pretty, even now after what feels like a long period of stagnation and un-addressed bugs. But Google+ isn't the most neglected neighborhood in Googleland.

Where is the Blogger blog?Various parts of Google, notably Search, use Blogger to convey news about new releases and their development road map. Blogger is key infrastructure for Google itself. But where is the Blogger blog? Like some other semi-abandoned properties, Blogger no longer has frequent updates about features.

If you want a history of Blogger and it's features, you'll have to rely on Wikipedia. Evidently there are ardent Blogger users who keep track of these things.

Why does this matter? There are a lot of abandoned places on the Internet. Blogger, however, is emblematic of the problems that precipitated the departure of Vic Gundotra from Google+. Blogger and Google+ have a very valuable natural symbiosis. Potentially, Blogger and Google+ could unify long-form and short-form writing on the Internet. But, in reality, the execution fell short. The Google+ comment thread system was grafted onto Blogger, which needed a replacement for its own weak and spam-riddled comment system. But this falls far short of actually solving the problem of creating a unified space for discovering posts and discussions of varying complexity and lifespan.

As we will see, this isn't the only missed opportunity.

Let's make a Group and discuss itYes, there is another social, post-centric, discussion thread tool at Google: Google Groups. It's easy to see why Google Groups is still used: You can get a concise listing of threads, organized by topic, with some particularly timeless threads "pinned" to the top of the list. Just the thing for discussing, among other things, features users want to see in Google products.

In 2013, the Groups announcement blog announced there wouldn't be any more announcement blog posts because they had become too rare. So we are left with a situation where I can't get a threaded view of a discussion on Google+ and I can't get "bell" notifications from Google Groups. Slummy.

2009 Called...Sites is another abandoned-looking amusement park in Googleland. Stagnant functionality. Outdated templates. No blog. Despite Google having properties like Wallet, there are no easy e-commerce options for Sites. It's just a middle-of-the-pack Web site builder with a wiki-ish flavor to it. Web site builders is a field in which there are less than a handful of first tier players. Therefore being "middle of the pack" means you're not in good company.

Braziiiiillll!!!We got this far, without mentioning Orkut, a social network operated by Google, with minimal integration with other Google properties (e.g. no "bell" in the strip at the top of the screen), that has a Turkish name and is big in Brazil and big-ish in India. With presumably limited resources, the look and feel of Orkut remains charmingly MySpace-like.

If Facebook is expanding internationally, how does an opportunity to integrate an international audience into Google+ keep floating out of reach?

YouTube is like Instagram for video, and should have been treated as gentlyYouTube is a social network. But here Google should have taken a tip from Facebook. YouTubers are mostly not Plussers, in the way that Instagram's hipsters are not Facebook schoolgirls. Facebook had the good sense to leave that situation alone. Instead, Google+ enforced strong identity on a trollish population. It's easy to see why that would be a thankless task, and a dumb move in an attention economy where "thanks" is the currency.

It's the product managementProduct management in technology companies is a distinct function that sometimes gets rolled up into program management, project management, or even engineering management.  But many of the above listed problems are the result of product management failures. The product manager is sometimes called the "product owner." He is meant to make the product competitive against specialist competitors, and balance the interests of "his" product against the sometimes abstract advantages of integrating with other products under the same roof.

In the egregious cases listed in this post, even the basic functions of competitive analysis seem to have been lost. That is either a product management failure or a senior management failure in deciding to keep products while starving them below a competitive level of resources. Lackadaisy or willful neglect, or both, are what turns some products into Internet "slums."

Let's take a property inventory:Let's do a little product management exercise and take an inventory of what Google owns, and the relevant characteristics. We'll leave Orkut out because it's largely redundant  in every characteristic to Google+.
ContentPersistenceUpdate streamDiscussion, commentsUser. group, managementGoogle+Short-ish formShort persistence - blink and it's goneBig, fast update stream, manageable with circlesReplies, but not branching threadsFlexible circle management integrated with Google address bookBloggerLong-formLongWeak. Can "follow" blogs, but it's a different "follow" than in Google+Weak, but optionally integrated with Google+NoneSitesMulti-pageLongNoneWiki-like comments and attachments are optionalSites can be private to a groupGroupsShort formMedium to long (pinned)Chronological view of posts, post-by-emailRich threaded branching discussionsGroup management not integrated with other Google propertiesYouTubeVideoMedium to longWeak Linear comments, now integrated with Google+ commentingCan follow users/channels; Users frquently maintained separate identities for YouTube

What does this table suggest? The main problem that jumps out at you from this table is that comment integration is weak integration. It doesn't deliver a lot of value and it doesn't solve the key problems for user-generated content. Meaningful, useful integration that improves the user experience must be deeper integration. All these properties have social characteristics, but comment integration is peripheral to social-centric

An MVC model for social propertiesWith Orkut, Google has six social properties, at least, before we get into things like social documents and social coding. With Google+, social became a pariah within Google for taking on the wrong integrations with the wrong products in the wrong order. Comment integration was weak, and with it came identity integration that was, in the case of YouTube a source of deep dissatisfaction. It's bad form, and just plain a bad idea, to go an mess with another product's community.

Integration should have started deeper, with the content database and update stream. Then with a gentle, optional, merging of identity, and adding the group-management features of Google+ Circles. Comment integration is then a minor issue. Each product can retain a distinctive set of views outside the chronological fast-flowing river of the update stream, thereby both merging and keeping distinctive the persistence characteristics of each property. You can think of that as multiple views into unified content, or a "model/view/controller" (MVC) approach to viewing the same model through multiple views.

The opportunityGoogle's biggest opportunity is to create the best user experience for user-generated content across Web sites/wikis, discussion group, blogs, videos, images, and the social update stream, and along with the content continuum, integrate (carefully!)the management of participant and their comments and collaborative participation. Almost everything has a social aspect, but Google somehow missed the deeper insight once Google+ latched on to comment integration as the feature it would take across properties.

One more thingOne more thing about Google+ and integration with other Google properties: Android. GMail is great on Android. The GMail Android interface is richer than the Web interface. Maps is magical on Android. Keep is simple but awesome on Android. And what's the big thing in social networks now? Mobile. Google+ is mediocre on Android. That's not how to make a product succeed at Google.
 •  0 comments  •  flag
Share on Twitter
Published on April 28, 2014 11:14