The SOAP versus REST debate reminds me of the EJB versus POJO arguments years ago. When EJBs first appeared on the scene, the Java developer community embraced it with full vigor only to realize later that it was not the wisest thing to do. EJBs were over-complicated with convoluted, inter-twined component, security, configuration, transaction and application models. They were designed with very specialized, complex, enterprise-scale scenarios in mind - scenarios irrelevant to most applications. Frameworks such as Hibernate, Spring and JDO, and simpler design patters with POJOs showed the way towards a more elegant approach. As a result, the developer community today is far better off.
SOAP has the same pedigree as EJB. SOAP originated primarily to serve the need of .NET-to-Java integration within the enterprise firewall. The myriad of WS-* specs were geared towards complex security, reliability and transactions involved in inter-system communication within large enterprises. Like EJBs, SOAP was embraced quickly by the developer community since it represented the first real standard for interoperability since the failed promises of CORBA. With HTTP transport and XML-based payloads in typical implementations, SOAP seemed like the perfect solution for all inter-system communication. However, like with EJBs, the developer community has come to realize that perhaps we have been on the wrong track by adopting SOAP for everything. REST-full patterns have emerged that seem more appropriate for a vast majority of applications and are becoming more and more popular. The abuse of SOAP-based Web Services will hopefully continue to diminish. REST has done to SOAP what POJO frameworks did to EJBs.
One of the issues with REST is the lack of a formal standard. REST is simply an architectural pattern rather than a standard like SOAP. If you wanted to publish to the world a formal API to access your system, SOAP offers a well-understood, tooling-friendly WSDL format. REST has no such standard - until now. JSR 311 JAX-RS spec represents a first step in more formalized standards for REST which will go a long way towards its wider adoption.
Talking of standards, it was interesting to see an entire keynote from Microsoft at JavaOne 2009 (yes, the evil empire at JavaOne) addressing SOAP interoperability. After all these years of SOAP standards, we are still figuring out interoperability between .NET- and Java-based SOAP implementations! To me that says a lot about the complexity of the SOAP standards.
To be clear, SOAP does have its place. SOAP is useful in complex enterprise scenarios involving things such as:
- end-to-end rather than point-to-point security
- non-HTTP transports
- 2-phase commits across heterogeneous systems
- guaranteed reliability and SLAs in messaging
A good articulation of this is by David Chappell here.
It was heartening to see a great deal of interest in REST at JavaOne this year. Clearly the "REST-lers" have scored a point over the "SOAP-ists". This trend is likely to continue, and like at the end of the EJB saga, we developers will be far better off as a result.
Showing posts with label Software Architecture. Show all posts
Showing posts with label Software Architecture. Show all posts
Thursday, June 11, 2009
Friday, March 13, 2009
The Three Cs - Cloud, Collaboration, Client
If I were a venture capitalist looking to invest money in the software space, I'd look for innovative new companies that incorporate elements of "three Cs" - Cloud, Collaboration and Client. In my opinion, the three Cs represent the next big wave in the software space. They have the potential to significantly impact the way software is built and deployed, and the way users use software, interact with systems, and with each other. Let's examine the three Cs.
Cloud
Cloud can be defined as a collection of computing resources made publicly available for access through a standard Internet connection and using well-defined APIs. There are three aspects to the cloud - storage, applications and computing.
Cloud Storage - This is the basic and most common use of the cloud - a place to store electronic content. A classic example of cloud storage is online storage of photographs from your camera with sites such as Picasa Web. Another example is regular online backup of the contents of your computer using services such as Apple's iDisk. Amazon's S3 (Simple Storage Service) is an example of a general purpose cloud storage solution.
Cloud Applications - This is the next level of cloud usage - applications hosted on the cloud that can be used without any software installations on the user's local machine. Saleforce.com is perhaps the poster child for this with their successful hosted CRM (Customer Relationship Management) solution. The term SAAS (Software as a Service) is often used to describe such cloud applications. In the last couple of years we have also seen the emergence of application development frameworks such as Google's App Engine which offer a cloud-based platform for building your own applications that are then hosted on the cloud. The term PAAS (Platform as a Service) is often used to describe these offerings.
Cloud Computing - This is the third and perhaps the newest buzz in the cloud space - computing power available for rent in the cloud. You rent virtual machines online for whatever purposes you need, for however long you need, and pay according to usage. The term IAAS (Infrastructure as a Service) is a term that appropriately describes such services. Amazon's EC2 (Elastic Compute Cloud) is clearly the most prominent player in this space.
So is all the cloud excitement really just hype? Absolutely not. This thing is real and here to stay in a big way. Cloud's value proposition is quite simple and intuitive.
* With Storage, the value proposition is reduced cost, reliability, security, and universal access. I've been using an external hard-drive for backups at home which now roughly costs me the same as using Amazon's S3 (<10GB of data with infrequent transfer in and out of AWS, over a the 5 year lifespan of my drive). With S3, I don't have to worry about my drive failing or getting stolen (my 1TB drive does not have an easy password protection mechanism other than encrypting each file). And of course I cannot access my drive at home from any machine with an Internet connection.
* With Applications, the value proposition is TCO (Total Cost of Ownership), particularly for businesses. Traditional enterprise software has always been a TCO headache for companies that spend a huge amount of their resources installing, maintaining, configuring, upgrading, patching software and the associated hardware. Having someone do that for you cheaper taking advantage of economies of scale makes perfect economic sense. It also makes good business sense because it lets companies focus more on their core business.
* With Computing, the value proposition is cost, flexibility and reliability. Renting computing resources and paying as you go based on usage rather than having to buy, install and maintain hardware and systems software has both cost and reliability implications. I like to use the electric utility grid analogy here - buying electricity from the utility grid works out much cheaper and reliable than every home having its own electric power generator. The additional benefit is the flexibility of being able to scale up or down the computing resources you need as your loads fluctuate. Also, you can pick and choose from a variety of hardware and software configurations at any time. This is a boon for small startups that don't have to buy hardware and can rent just what they need or afford, scaling as they grow. In fact, it is possible to start a purely virtual company which does not own a single server box.
Some critics argue that all the elements of cloud have been around for sometime now and don't understand why there is all this hype suddenly as if a new technology revolution has taken place. While I agree partially with the critics, I think what has happened recently is that the cloud has reached a tipping point. IMO, there are 3 reasons for this tipping point. The first reason is bandwidth. High speed access to the cloud can be taken for granted now for a larger segment of businesses and homes. Unlike a few years ago, today I could probably get high-speed Internet access at a Motel 6 in rural Kansas. The second reason is virtualization technologies that have matured to make cost effective virtual machines on commodity hardware possible. The third and maybe the most important one is that the "big boys" are entering this space in a big way - Google with App Engine and Apps, Amazon with EC2, S3, SDB and SQS, Microsoft with Azure and OfficeLive, and IBM with its massive cloud initiative. These technology leaders are bound to drive innovation and adoption on a massive scale.
So everything about the cloud seems very rosy. Surely there must be some challenges. Of course, there are several.
* Security and Privacy - Security of content in the cloud is a concern for everyone especially for enterprise customers who may hold sensitive customer and business data. Ensuring that the data in the cloud can be accessed only by those authorized is absolutely critical. Also recent compliance requirements such as SOX force more processes around securing data for companies. While security concerns are genuine, there are good solutions. Today companies are storing their critical business data on the cloud e.g. the hundreds of companies that entrust their CRM data to Salesforce.com to be stored along-side data from their competitors (possibly in the same database instance!). Privacy concerns are important for individual consumers who want guarantees that their personal information will not be abused in any way. Ultimately it all comes down to trust, and as the cloud becomes more mainstream, it is likely companies and individuals will become more and more comfortable with most (if not all) of their data on the cloud.
* Vendor Lock-In - This is a major issue that is perhaps the biggest challenge to cloud adoption. With SAAS and PAAS, you are locked into a particular cloud vendor. What if you decide a year later to go elsewhere? There may not be an easy way to migrate to another cloud services provider. At a recent SDForum Cloud Services SIG talk, the term "cloud-neutrality" was mentioned several times. Until standards emerge and get adopted widely by cloud services providers, lock-in is going to a reality.
* Cloud Interoperability - This is going to be an issue for enterprise customers who move to using business applications on the cloud. Consider a company that uses Salesforce.com. Say the company acquires two companies - one using Workday and another using Netsuite. How does the company go about consolidating these three into one? Even a simple single sign-on across all three will be a major challenge leave alone data or business process integration. Web Services standards help to some extent assuming what you need is exposed as Web Services by the SAAS/PAAS vendor, but it does not solve the problem of rationalizing data semantics e.g. how do you rationalize the definition of Customer in one system with that in another? While this problem has always existed even with non-cloud applications from different vendors, with the cloud, the problem is compounded because the cloud applications are complete black-boxes to you and you have no access to the underlying data stores and app infrastructure which are hidden behind the SAAS/PAAS provider's proprietary systems. This has the potential to be a huge problem and perhaps a great opportunity for new companies providing "inter-cloud " migration/integration tools and services.
Collaboration
Once content is in the cloud, it is logical to think of how users can collaborate with each other on that content with the content remaining in the cloud. The last part is key - content that is collaborated on remains in the cloud and is not downloaded to the user's machine. So we are not talking of (say) one user editing a PowerPoint presentation, posting or e-mailing it, and another user pulling it down, making changes and posting or e-mailing another version. We are talking of real-time, concurrent editing of cloud content collaboratively by multiple users. Using PowerPoint as an example, new companies such as SlideRocket (now part of Adobe) and InstaColl offer such collaborative presentation software alternatives to Microsoft PowerPoint. Google offers simple word processing and spreadsheet solutions with Google Docs. Microsoft of course also offers such a service with OfficeLive but it still requires traditional Office applications installed on your machine and is therefore more of a cloud storage solution and less of a collaboration solution as described above.
For those who have struggled with managing and sharing versions of Word, Excel and PowerPoint files, I'm sure the notion of cloud-based collaboration without ever having to work with local versions of files is appealing. However, it remains to be seen if commercial-grade products can deliver a rigorous solution that handles the complexity of managing real-time concurrent updates, merge conflict resolutions, branching and versioning. This is strikingly similar to what revision control systems such as Clearcase and SVN do for source code; and we know the complexities involved with a large group of geographically dispersed developers working on a shared code base. But it is a surmountable problem and some winners are likely to emerge given the huge potential for products and solutions that improve the widely-used office productivity software.
Real-time collaboration with screen sharing, instant messaging, audio and video streams are powerful and familiar tools in collaboration which become even more powerful with the cloud because of cloud content being accessible from anywhere. You could, for example, launch a conference from any machine (or device) and collaborate on content residing in the cloud.
While screen sharing based collaboration is powerful, there is another even more powerful, emerging collaboration technique that I call "gesture replay". Unlike in screen sharing where collaboration is achieved by scraping the pixels off a presenter's screen and sending it across to other participants in the collaboration session, in gesture replay, the presenter's user gesture alone is sent across to all participants. The user gesture is then replayed at each participant's client resulting in the same state transition in each participant's application as the presenter's. Of course each participant has to be running the same client software and the collaboration is limited to only that client software and not the entire desktop. For example, all users connected to a Web site loading a Flash-based application can collaborate inside that application using gesture replay. With gesture replay, the user gesture event being propagated to all participants is very compact and bandwidth efficient because it only involves information about the event and not the application state resulting from execution of the event. This results in very efficient, fast, real-time responses for all participants. With gesture replay, you can get the responsiveness of a native instant messaging application in a screen sharing-type collaboration.
As noted above, gesture replay involves replaying the presenter's gesture in each participant's client. This obviously implies that the logic and knowledge to replay the gesture resides in each client which in turn implies that such clients are powerful applications and not simple Web pages. Further, it also implies that the clients run within powerful runtimes that can efficiently execute events in real-time. This brings us to the third "C" of the three Cs - Client.
But before we move on to the Client, it is worth noting that real-time collaboration with gesture replay has some scalability challenges with HTTP. Proprietary solutions exist today to solve the scalability problem. The upcoming Servlet 3.0 spec (JSR 315) with its suspend-able/resume-able request support will offer a standards-based solution which is likely to promote wider adoption of such collaboration techniques.
Client
I started programming on IBM mainframes using punch cards as input (man, was that painful!). There was no client. Then came dumb terminals. We gradually moved to minicomputers, workstations and PCs. The client-server era gave birth to widespread adoption of business software which had graphical user interfaces and were client-heavy. With the Internet revolution and the Web, the focus shifted back to the server with clients being thin Browser-based. The Web has of course truly revolutionized our lives. However, as we have striven to do more and more with the Browser which was oriented towards rendering static pages, we have struggled to provide users with engaging, rich, high-performance clients. Web 2.0 has moved us a step above where we used to be in this regard, but we are not there yet. We need the Web 2.5 revolution to get us to where we provide more intuitive, interactive, graphical, responsive interfaces to users.
With Web 2.5, the focus shifts back to the client while at the same time being able to maintain a Browser model - the best of both worlds. RIA (Rich Internet Application) technolgies such as Flash, Silverlight and JavaFX are ushering in the Web 2.5 (r)evolution. As this post describes, these RIA technologies provide very efficient and powerful client runtimes (such as the Flash Player), a uniform and consistent target to program to (without having to code for each Browser), and formalized client-side component and event model frameworks to build complex applications containing heavy-duty client-side logic. Such RIAs take advantage of the powerful machines users run their clients on, which traditional DHTM/AJAX applications have not done. A customer once asked us why his iPhone which is at best a 600MHz processor with 128MB RAM can provide such a cool interface for applications while our DHTML enterprise application running on a dual-core 2GHz box with 2GB RAM was so crummy!
The Web 2.5 (r)evolution is well underway. We are beginning to see more and more enterprise scale applications such as those from Workday that take advantage of the Web 2.5 model.
From a collaboration perspective, the Web 2.5 client model facilitates the kind of gesture replay collaboration described earlier. This is due to
a) the emergence of frameworks and tools from companies such as Adobe (for Flash/Flex), Micosoft (for Silverlight, WPF) and SUN (for JavaFX) which allow for users building heavy-duty applications that can handle the gesture event execution logic, and
b) efficient runtimes such as the ActionScript VM (Virtual Machine) for Flash or the Java VM for JavaFX - runtimes featuring JIT compilers, that allow for high performance gesture event execution on the client.
So there it is - the three Cs - Cloud, Collaboration and Client. As developers, we need to start orienting our skill sets towards these exciting new trends.
Cloud
Cloud can be defined as a collection of computing resources made publicly available for access through a standard Internet connection and using well-defined APIs. There are three aspects to the cloud - storage, applications and computing.
Cloud Storage - This is the basic and most common use of the cloud - a place to store electronic content. A classic example of cloud storage is online storage of photographs from your camera with sites such as Picasa Web. Another example is regular online backup of the contents of your computer using services such as Apple's iDisk. Amazon's S3 (Simple Storage Service) is an example of a general purpose cloud storage solution.
Cloud Applications - This is the next level of cloud usage - applications hosted on the cloud that can be used without any software installations on the user's local machine. Saleforce.com is perhaps the poster child for this with their successful hosted CRM (Customer Relationship Management) solution. The term SAAS (Software as a Service) is often used to describe such cloud applications. In the last couple of years we have also seen the emergence of application development frameworks such as Google's App Engine which offer a cloud-based platform for building your own applications that are then hosted on the cloud. The term PAAS (Platform as a Service) is often used to describe these offerings.
Cloud Computing - This is the third and perhaps the newest buzz in the cloud space - computing power available for rent in the cloud. You rent virtual machines online for whatever purposes you need, for however long you need, and pay according to usage. The term IAAS (Infrastructure as a Service) is a term that appropriately describes such services. Amazon's EC2 (Elastic Compute Cloud) is clearly the most prominent player in this space.
So is all the cloud excitement really just hype? Absolutely not. This thing is real and here to stay in a big way. Cloud's value proposition is quite simple and intuitive.
* With Storage, the value proposition is reduced cost, reliability, security, and universal access. I've been using an external hard-drive for backups at home which now roughly costs me the same as using Amazon's S3 (<10GB of data with infrequent transfer in and out of AWS, over a the 5 year lifespan of my drive). With S3, I don't have to worry about my drive failing or getting stolen (my 1TB drive does not have an easy password protection mechanism other than encrypting each file). And of course I cannot access my drive at home from any machine with an Internet connection.
* With Applications, the value proposition is TCO (Total Cost of Ownership), particularly for businesses. Traditional enterprise software has always been a TCO headache for companies that spend a huge amount of their resources installing, maintaining, configuring, upgrading, patching software and the associated hardware. Having someone do that for you cheaper taking advantage of economies of scale makes perfect economic sense. It also makes good business sense because it lets companies focus more on their core business.
* With Computing, the value proposition is cost, flexibility and reliability. Renting computing resources and paying as you go based on usage rather than having to buy, install and maintain hardware and systems software has both cost and reliability implications. I like to use the electric utility grid analogy here - buying electricity from the utility grid works out much cheaper and reliable than every home having its own electric power generator. The additional benefit is the flexibility of being able to scale up or down the computing resources you need as your loads fluctuate. Also, you can pick and choose from a variety of hardware and software configurations at any time. This is a boon for small startups that don't have to buy hardware and can rent just what they need or afford, scaling as they grow. In fact, it is possible to start a purely virtual company which does not own a single server box.
Some critics argue that all the elements of cloud have been around for sometime now and don't understand why there is all this hype suddenly as if a new technology revolution has taken place. While I agree partially with the critics, I think what has happened recently is that the cloud has reached a tipping point. IMO, there are 3 reasons for this tipping point. The first reason is bandwidth. High speed access to the cloud can be taken for granted now for a larger segment of businesses and homes. Unlike a few years ago, today I could probably get high-speed Internet access at a Motel 6 in rural Kansas. The second reason is virtualization technologies that have matured to make cost effective virtual machines on commodity hardware possible. The third and maybe the most important one is that the "big boys" are entering this space in a big way - Google with App Engine and Apps, Amazon with EC2, S3, SDB and SQS, Microsoft with Azure and OfficeLive, and IBM with its massive cloud initiative. These technology leaders are bound to drive innovation and adoption on a massive scale.
So everything about the cloud seems very rosy. Surely there must be some challenges. Of course, there are several.
* Security and Privacy - Security of content in the cloud is a concern for everyone especially for enterprise customers who may hold sensitive customer and business data. Ensuring that the data in the cloud can be accessed only by those authorized is absolutely critical. Also recent compliance requirements such as SOX force more processes around securing data for companies. While security concerns are genuine, there are good solutions. Today companies are storing their critical business data on the cloud e.g. the hundreds of companies that entrust their CRM data to Salesforce.com to be stored along-side data from their competitors (possibly in the same database instance!). Privacy concerns are important for individual consumers who want guarantees that their personal information will not be abused in any way. Ultimately it all comes down to trust, and as the cloud becomes more mainstream, it is likely companies and individuals will become more and more comfortable with most (if not all) of their data on the cloud.
* Vendor Lock-In - This is a major issue that is perhaps the biggest challenge to cloud adoption. With SAAS and PAAS, you are locked into a particular cloud vendor. What if you decide a year later to go elsewhere? There may not be an easy way to migrate to another cloud services provider. At a recent SDForum Cloud Services SIG talk, the term "cloud-neutrality" was mentioned several times. Until standards emerge and get adopted widely by cloud services providers, lock-in is going to a reality.
* Cloud Interoperability - This is going to be an issue for enterprise customers who move to using business applications on the cloud. Consider a company that uses Salesforce.com. Say the company acquires two companies - one using Workday and another using Netsuite. How does the company go about consolidating these three into one? Even a simple single sign-on across all three will be a major challenge leave alone data or business process integration. Web Services standards help to some extent assuming what you need is exposed as Web Services by the SAAS/PAAS vendor, but it does not solve the problem of rationalizing data semantics e.g. how do you rationalize the definition of Customer in one system with that in another? While this problem has always existed even with non-cloud applications from different vendors, with the cloud, the problem is compounded because the cloud applications are complete black-boxes to you and you have no access to the underlying data stores and app infrastructure which are hidden behind the SAAS/PAAS provider's proprietary systems. This has the potential to be a huge problem and perhaps a great opportunity for new companies providing "inter-cloud " migration/integration tools and services.
Collaboration
Once content is in the cloud, it is logical to think of how users can collaborate with each other on that content with the content remaining in the cloud. The last part is key - content that is collaborated on remains in the cloud and is not downloaded to the user's machine. So we are not talking of (say) one user editing a PowerPoint presentation, posting or e-mailing it, and another user pulling it down, making changes and posting or e-mailing another version. We are talking of real-time, concurrent editing of cloud content collaboratively by multiple users. Using PowerPoint as an example, new companies such as SlideRocket (now part of Adobe) and InstaColl offer such collaborative presentation software alternatives to Microsoft PowerPoint. Google offers simple word processing and spreadsheet solutions with Google Docs. Microsoft of course also offers such a service with OfficeLive but it still requires traditional Office applications installed on your machine and is therefore more of a cloud storage solution and less of a collaboration solution as described above.
For those who have struggled with managing and sharing versions of Word, Excel and PowerPoint files, I'm sure the notion of cloud-based collaboration without ever having to work with local versions of files is appealing. However, it remains to be seen if commercial-grade products can deliver a rigorous solution that handles the complexity of managing real-time concurrent updates, merge conflict resolutions, branching and versioning. This is strikingly similar to what revision control systems such as Clearcase and SVN do for source code; and we know the complexities involved with a large group of geographically dispersed developers working on a shared code base. But it is a surmountable problem and some winners are likely to emerge given the huge potential for products and solutions that improve the widely-used office productivity software.
Real-time collaboration with screen sharing, instant messaging, audio and video streams are powerful and familiar tools in collaboration which become even more powerful with the cloud because of cloud content being accessible from anywhere. You could, for example, launch a conference from any machine (or device) and collaborate on content residing in the cloud.
While screen sharing based collaboration is powerful, there is another even more powerful, emerging collaboration technique that I call "gesture replay". Unlike in screen sharing where collaboration is achieved by scraping the pixels off a presenter's screen and sending it across to other participants in the collaboration session, in gesture replay, the presenter's user gesture alone is sent across to all participants. The user gesture is then replayed at each participant's client resulting in the same state transition in each participant's application as the presenter's. Of course each participant has to be running the same client software and the collaboration is limited to only that client software and not the entire desktop. For example, all users connected to a Web site loading a Flash-based application can collaborate inside that application using gesture replay. With gesture replay, the user gesture event being propagated to all participants is very compact and bandwidth efficient because it only involves information about the event and not the application state resulting from execution of the event. This results in very efficient, fast, real-time responses for all participants. With gesture replay, you can get the responsiveness of a native instant messaging application in a screen sharing-type collaboration.
As noted above, gesture replay involves replaying the presenter's gesture in each participant's client. This obviously implies that the logic and knowledge to replay the gesture resides in each client which in turn implies that such clients are powerful applications and not simple Web pages. Further, it also implies that the clients run within powerful runtimes that can efficiently execute events in real-time. This brings us to the third "C" of the three Cs - Client.
But before we move on to the Client, it is worth noting that real-time collaboration with gesture replay has some scalability challenges with HTTP. Proprietary solutions exist today to solve the scalability problem. The upcoming Servlet 3.0 spec (JSR 315) with its suspend-able/resume-able request support will offer a standards-based solution which is likely to promote wider adoption of such collaboration techniques.
Client
I started programming on IBM mainframes using punch cards as input (man, was that painful!). There was no client. Then came dumb terminals. We gradually moved to minicomputers, workstations and PCs. The client-server era gave birth to widespread adoption of business software which had graphical user interfaces and were client-heavy. With the Internet revolution and the Web, the focus shifted back to the server with clients being thin Browser-based. The Web has of course truly revolutionized our lives. However, as we have striven to do more and more with the Browser which was oriented towards rendering static pages, we have struggled to provide users with engaging, rich, high-performance clients. Web 2.0 has moved us a step above where we used to be in this regard, but we are not there yet. We need the Web 2.5 revolution to get us to where we provide more intuitive, interactive, graphical, responsive interfaces to users.
With Web 2.5, the focus shifts back to the client while at the same time being able to maintain a Browser model - the best of both worlds. RIA (Rich Internet Application) technolgies such as Flash, Silverlight and JavaFX are ushering in the Web 2.5 (r)evolution. As this post describes, these RIA technologies provide very efficient and powerful client runtimes (such as the Flash Player), a uniform and consistent target to program to (without having to code for each Browser), and formalized client-side component and event model frameworks to build complex applications containing heavy-duty client-side logic. Such RIAs take advantage of the powerful machines users run their clients on, which traditional DHTM/AJAX applications have not done. A customer once asked us why his iPhone which is at best a 600MHz processor with 128MB RAM can provide such a cool interface for applications while our DHTML enterprise application running on a dual-core 2GHz box with 2GB RAM was so crummy!
The Web 2.5 (r)evolution is well underway. We are beginning to see more and more enterprise scale applications such as those from Workday that take advantage of the Web 2.5 model.
From a collaboration perspective, the Web 2.5 client model facilitates the kind of gesture replay collaboration described earlier. This is due to
a) the emergence of frameworks and tools from companies such as Adobe (for Flash/Flex), Micosoft (for Silverlight, WPF) and SUN (for JavaFX) which allow for users building heavy-duty applications that can handle the gesture event execution logic, and
b) efficient runtimes such as the ActionScript VM (Virtual Machine) for Flash or the Java VM for JavaFX - runtimes featuring JIT compilers, that allow for high performance gesture event execution on the client.
So there it is - the three Cs - Cloud, Collaboration and Client. As developers, we need to start orienting our skill sets towards these exciting new trends.
Sunday, February 10, 2008
M13
M13 Application Development Paradigms for large-scale enterprise applications:
- All business functionality is thought of in terms of one or more services. An application includes a set of services and their implementers/providers.
- A Service does not automatically imply Web Service.
- A Service is defined by a Java interface and its implementers are Java classes. A Service can have multiple implementers.
- Service definitions and implementers are independent of how the services are provisioned (Web Service, Java API, EJBs etc.), and are independent of the UI Channel (Browser DHTML, Browser Flash, Adobe AIR, MS WPF, Java Swing etc.).
- Service provisioning is declarative meta-data driven.
- Service implementers can be switched with configuration entry changes and without any code change. Services can optionally be versioned and it is possible to have multiple versions of a service active in the system at the same time.
- For remote clients, a client-side service API is available for clients to code against.
- Services produce and consume strongly typed objects and not XML. Optimized binary formats are preferred over text-based formats for data transfer across the wire
- UI Components bind to strongly-typed, well-defined data models produced directly by the services without any intermediate transformations.
- A common canonical form for clients across UI Channels is impractical and infeasible for any real-life application. Clients are written separately for each UI Channel in its own technology but they share the same services layer.
- All application data can be broken up into 2 clear and distinct categories - meta-data and data.
- Application structure and flow is meta-data driven.
- All meta-data is efficiently query-able and stored in a relational data store. Meta-data is versionable and upgradable, and it is possible to have multiple versions of a meta-data element active in the system at the same time.
Sunday, December 30, 2007
Web Services Abuse
Web Services is arguably one of the most significant advance in enterprise software systems in the last 5 years. Web Services have provided a mechanism to unlock and expose data and business logic buried within legacy systems in a uniform and standardized fashion. This is significant because most enterprises use a multitude of different software systems for various operational and strategic functions. These systems tend to be very different in their architectures, platforms, programming languages, interfaces and data stores. Web Services has made it is possible to have a single standard mechanism to access all systems.
Business processes in enterprises often involve accessing more than one system and require information flow among different systems. With Web Services-based standardized programmable access to all systems, we now have an opportunity to create so-called "composite applications" which allow seamless transfer of data and invocation of business logic across multiple systems.
SOAP is an important standard for Web Services. While the simpler REST-based Web Services are popular, SOAP is important in Web Services where security and reliability are important. For both SOAP- and REST-based Web Services, XML is the de facto standard for the format in which data is produced and consumed by services. This text-based format with a standardized and well understood structure is one of the reasons for the easy adoption of Web Services.
While Web Services is a great choice for enterprise system integration, it is also widely used where it probably shouldn't be. The primary scenario where Web Services is not the best choice is when it is used to integrate sub-systems within a system built by the same vendor, and in particular when there is a large amount of data transfer among these sub-systems. Often sub-systems owned by different development sub-groups are built independently without attempting to standardize on common infrastructure and inter sub-system service contracts. This is typically a result of development groups growing unmanageably large (see A Few Good Men) and sub-groups inherited from mergers and acquisitions. Each sub-system tends to adopt its own architecture, frameworks, metadata and data formats, and access mechanisms, ending up looking very different even though they may all be part of the same product suite sold by the same vendor. When it comes time for these sub-systems to talk to each other, the challenge is easily addressed with the magic words - Web Services. Each sub-system builds a Web Services layer and now they can all talk to each other fine. Problem solved. In reality though, there is a great deal of inefficiency in this approach. Often these sub-systems live in the same runtime and communication among them does not involve crossing process boundaries. In these cases, what should be simple co-located service API calls exchanging objects between sub-systems, becomes an inefficient process with unnecessary penalty of protocols such as SOAP and the overhead of XML processing. XML is by nature verbose and parsing of XML is expensive even with modern parsers. When the volume of data is large, the performance overhead of object->XML->Object transformations can be very significant. Data exchanges in binary format are typically orders of magnitude more efficient that text-based formats. This study illustrates graphically a benchmark that support this argument.
The need for loose coupling among sub-systems is often cited as a reason for integrating them through Web Services. The specious argument goes like this - if two sub-systems expose APIs and share object models, any change to one sub-system will require re-compilation of the other which is undesirable. However, what is often overlooked is that even with Web Services, if the format of the XML data produced by a service changes, the consumer of the XML has to re-work the business logic code requiring re-compilation and re-deployment.
The right paradigm for designing systems is to adopt the M13 services paradigm. All business functionality is thought of in terms of services. Services do not necessarily mean Web Services. One defines Java service interfaces that produce and consume serializable Java objects, and builds their implementations. This should be independent of how the services are exposed and invoked. An external declarative provisioning mechanism is available that dictates the best mode of service consumption - plain Java APIs, Web Services, EJBs etc. The underlying infrastructure insulates the author of the service from the provisioning and lets him/her focus on the service contract and business logic. It also allows for employing the optimal service invocation mechanism depending on the scenario. Of course, this requires all sub-systems to conform to a common service authoring paradigm such as M13.
Web Services are great for exchanging small targeted data between disparate systems. It may be the wrong choice when you have full control over systems on either end of the wire. In those cases, consider more efficient alternatives.
Business processes in enterprises often involve accessing more than one system and require information flow among different systems. With Web Services-based standardized programmable access to all systems, we now have an opportunity to create so-called "composite applications" which allow seamless transfer of data and invocation of business logic across multiple systems.
SOAP is an important standard for Web Services. While the simpler REST-based Web Services are popular, SOAP is important in Web Services where security and reliability are important. For both SOAP- and REST-based Web Services, XML is the de facto standard for the format in which data is produced and consumed by services. This text-based format with a standardized and well understood structure is one of the reasons for the easy adoption of Web Services.
While Web Services is a great choice for enterprise system integration, it is also widely used where it probably shouldn't be. The primary scenario where Web Services is not the best choice is when it is used to integrate sub-systems within a system built by the same vendor, and in particular when there is a large amount of data transfer among these sub-systems. Often sub-systems owned by different development sub-groups are built independently without attempting to standardize on common infrastructure and inter sub-system service contracts. This is typically a result of development groups growing unmanageably large (see A Few Good Men) and sub-groups inherited from mergers and acquisitions. Each sub-system tends to adopt its own architecture, frameworks, metadata and data formats, and access mechanisms, ending up looking very different even though they may all be part of the same product suite sold by the same vendor. When it comes time for these sub-systems to talk to each other, the challenge is easily addressed with the magic words - Web Services. Each sub-system builds a Web Services layer and now they can all talk to each other fine. Problem solved. In reality though, there is a great deal of inefficiency in this approach. Often these sub-systems live in the same runtime and communication among them does not involve crossing process boundaries. In these cases, what should be simple co-located service API calls exchanging objects between sub-systems, becomes an inefficient process with unnecessary penalty of protocols such as SOAP and the overhead of XML processing. XML is by nature verbose and parsing of XML is expensive even with modern parsers. When the volume of data is large, the performance overhead of object->XML->Object transformations can be very significant. Data exchanges in binary format are typically orders of magnitude more efficient that text-based formats. This study illustrates graphically a benchmark that support this argument.
The need for loose coupling among sub-systems is often cited as a reason for integrating them through Web Services. The specious argument goes like this - if two sub-systems expose APIs and share object models, any change to one sub-system will require re-compilation of the other which is undesirable. However, what is often overlooked is that even with Web Services, if the format of the XML data produced by a service changes, the consumer of the XML has to re-work the business logic code requiring re-compilation and re-deployment.
The right paradigm for designing systems is to adopt the M13 services paradigm. All business functionality is thought of in terms of services. Services do not necessarily mean Web Services. One defines Java service interfaces that produce and consume serializable Java objects, and builds their implementations. This should be independent of how the services are exposed and invoked. An external declarative provisioning mechanism is available that dictates the best mode of service consumption - plain Java APIs, Web Services, EJBs etc. The underlying infrastructure insulates the author of the service from the provisioning and lets him/her focus on the service contract and business logic. It also allows for employing the optimal service invocation mechanism depending on the scenario. Of course, this requires all sub-systems to conform to a common service authoring paradigm such as M13.
Web Services are great for exchanging small targeted data between disparate systems. It may be the wrong choice when you have full control over systems on either end of the wire. In those cases, consider more efficient alternatives.
Sunday, August 19, 2007
Web 2.5
I've always considered myself a "server guy" - working on server-side infrastructure and tooling for enterprise-scale application development, with Java as the programming language of choice in recent years. Client-side UI development, Web UIs in particular, always seemed less challenging. Writing HTML markup and scripting was for "designers" whose focus was on aesthetics rather than heavy duty coding. The heavy duty coding was done by the so-called server guys who provided nicely encapsulated business logic "beans" that designers could consume easily through simple scripting and without much knowledge of programming constructs. Currently popular Web UI development frameworks such as Java Server Faces (JSF) have promoted such a separation of concerns. JSF is fundamentally server oriented (the user interface code runs on the server) and provides separation of the roles of "page authors" writing markup and "application/component developers" writing server-side Java code.
Then came the Web 2.0 revolution. Rich Internet Applications (RIAs), the principal component of Web 2.0, are everywhere. Applications based on AJAX, the umbrella term for technologies associated with building Web 2.0 RIAs, are becoming increasingly sophisticated. The appearance a couple of years ago of revolutionary applications such as Google Maps perhaps triggered interest in more traditionally server-side developers to start working with Web 2.0 AJAX client applications. A larger community of developers taking on the challenge of building Web 2.0 RIAs has resulted in more innovation and wider adoption. In addition, it has changed much of our traditional thinking on application development for the Web.
The fundamental paradigm shift with Web 2.0 RIAs is the moving of more processing away from the server and on to the client. Rich interactivity in RIAs is provided by more client-side execution of business logic and by portions of the application being able to asynchronously communicate with the server and update their state. This paradigm shift has not been a result of any new technology breakthrough, rather a result of factors such as more developers discovering the exciting possibilities of DHTML and JavaScript, and the increased penetration of broadband internet connectivity among consumers. So as more processing moved from the server to the Browser client, what has it meant for the way we build these Web 2.0 RIAs? Well... a lot of pain! The pain is related to three principal factors - client-side component and event models, browser compatibility, and enhanced graphical capabilities.
When software modules reach a certain level of complexity, they require formalized models and frameworks for developers to work with to be effective. With JSF, for the first time we had a formalized component and event model for building Web UIs that we never had before. However, JSF's model is a server-side model. There is no formal, standardized client-side model defining component and event contracts, with frameworks for managing component lifecycle and event handling in the client. As a result most RIAs work without any formal client-side models. Coding complex client-side business logic in RIAs typically involves JavaScript code manipulating an XML/HTML DOM tree rather than invoking client component APIs. With multiple elements of an application asynchronously interacting with the server, having an element of the application coordinate with, and react to, events triggered by other elements often involves intimate knowledge of the internals of other elements. All this results in code that is difficult for developers to write, debug and maintain beyond certain simple applications. For a server-side developer in particular, coding with loosely typed scripting against a XML DOM tree is disconcerting compared to working with statically type-checked, fully object oriented Java code using strongly typed objects and formal APIs.
The second pain point is all too well known - getting a Web application to work consistently across Browsers. With JavaScript implementations that are still different in different Browsers, and RIAs heavily using JavaScript, managing the quirks of each Browser implementation is a major challenge requiring significant development and maintenance effort. This is particularly unsettling for server-side Java developers who take the "write once run anywhere" paradigm for granted.
The last pain point is related to building better graphical capabilities in user interfaces. RIA UIs are increasingly becoming less text oriented and more graphical. Popular Finance applications provide sophisticated graphical data visualizations, with greater ability for users to interact with the data. Even traditional "shopping cart applications" employ more graphical capabilities in helping consumers visualize and interact with products they buy. This enhanced interaction involves things such as sliders for moving the window of data to visualize, 3-D views of products and charts, rotation of a views for alternate perspectives, real-time data push to clients with animated update, use of audio and video etc. It is safe to assume that this trend will continue to grow as users get accustomed to rich graphical capabilities in the interfaces they use. The challenge in today's Web 2.0 RIAs is related to the difficulty in building such capabilities into applications without sacrificing performance. The Browser compatibility issue mentioned before is perhaps even more of a problem when attempting to do vector drawing and animation in the Browser.
Several commercial and open-source AJAX frameworks exist today that attempt to address the pain points described above. While many have done a reasonable job, two fundamental problems remain. One is the fact that these frameworks typically are not directly compatible with each other. Due to the lack of any client-side standards, plugging in AJAX widgets from multiple vendors within an application and having them interact with each other is not an easy task. The second and more fundamental problem relates to the inherent limitations of the JavaScript runtime in the Browser. AJAX frameworks that have attempted to create a "thick client" in the Browser with complete component/event models and client-side state have suffered from performance issues when trying to scale from basic Web pages to larger full-blown applications. It is unlikely that this fundamental limitation is going to go away in the immediate future.
So are we then doomed to just living with these limitations of building Web 2.0 RIAs? Is there a better solution? Yes, there is. It appeared to me as a flash while prototyping with Adobe's Flex framework for building applications that run in the Flash player in Browsers. The pain points mentioned above don't exist when building in this environment. With Flex components, one works with a formal client-side UI component and event model, data model contracts, and MVC-based patterns. With the object-oriented ActionScript 3.0 language, one can work with strongly typed objects, static type checking, interfaces and implementation classes, formal API contracts, polymorphism and exception handling which feels natural for a Java developer. Service calls to the server from the client are by definition asynchronous and don't require any special coding. The AMF3 message format used in the communication between the Flash Player and a Java server provides an efficient binary mode of exchanging information without being forced to using verbose and inefficient text-based formats. With remoting and data service facilities, plugging into a Java back end is a breeze. The Flash Player frees the developer from having to code for compatibility with all Browsers. The ubiquitous nature of the player in consumer and business machines provides a standardized, pre-installed runtime. But above all, the most significant factor is the superior runtime - the Virtual Machine in the Flash Player that executes bytecode generated by compiling ActionScript code. The new ActionScript Virtul Machine (AVM2) in Flash Player 9 with its JIT compiler provides compelling performance. It is possible now to build "thick" client applications running in the Flash Player that can provide satisfactory runtime performance. Flash has graduated from being a channel for "intro screen" multi-media content rendering to a viable medium for full-blown enterprise applications. The graphical capabilities within Flash are of course well known - after all that has traditionally been the sweet spot of Flash. The fact that popular Finance sites offering stock charts render Flash versions is testimony to the superior capabilities of this channel for such graphical content.
I find building with the Flex framework for the Flash Player a significant step forward in the evolution of Web 2.0 RIA development. Microsoft's Silverlight (formerly WPF/E) may provide a similar channel in the near future although we're unlikely to see any easy integration with Java back ends. JavaFX may facilitate similar capabilities for building RIAs sometime in the future (re-incarnation of applets). I call this evolution to the next step Web 2.5. Web 2.5 is not a revolutionary new breakthrough (that's why it is not Web 3.0!), rather an evolutionary step up. Web 2.5 is building the same Web 2.0 style applications without all the pain. Web 2.5 is large complex applications being able to provide more intuitive, interactive, graphical, responsive interfaces to users. Web 2.5 is building "thick" Browser-based client applications that can perform. And with Web 2.5, even us server guys can whip out sizzling Web UIs!
Subscribe to:
Posts (Atom)
