Testing Your Web site
by Eric Kaufman
A good Web site is vital to a company's success, especially with the emergence of e-commerce and the trend from packaged applications to Web-based applications. If you are involved in any stage of a Web site's development, you are responsible for contributing to the quality of the site.
To ensure your visitors have a high-quality experience, and more importantly, that they return to your site, you need a well-planned and thorough testing effort addressing different browsers, competing technologies, and variances of the Internet.
Remember: Your users demand higher quality than ever, and it only takes one click to abandon your site and find another. To make sure your users don't do that, you'll want to focus on the following issues:
· Browser Compatibility
· Operating System Compatibility
compatibility is a huge issue. There are many different browsers and, of
course, multiple versions of each browser. Most market share belongs to
Netscape's Navigator, Microsoft's Internet Explorer (IE), and America OnLine (
Browsers do not always follow the same standards, so technologies and functions that work in one browser may not necessarily work in another; this means you must design carefully and test against many different browsers.
How do browsers differ?
First, the newer versions of the browsers support newer technologies, such as Cascading Style Sheets. Second, Netscape and IE do not implement the same technologies in the same manner. Because of these differences, a Web page can look different and behave differently when viewed with different browsers.
Chances are that a site is developed using one browser much more than the others. Testing across a variety of browsers will reveal issues the developer may be unaware of. For example, a button titled "Submit Order" looks good when viewed in IE, but it might be too small when viewed with Netscape. The result may be incomplete text.
Another differentiating technology is ActiveX. Microsoft's IE handles ActiveX controls by default, but Netscape Navigator requires a plug-in. Netscape users must be aware that the plug-in is required and should be given instructions on how to install it.
Be sure to use clean machines when you start testing and make sure no plug-ins are installed. If the plug-ins are already installed, you might miss a defect that has a dependency on the plug-in. There should always be some test cases that involve using a browser as it is first installed, with no extra components.
Which browsers should you test with?
There are many users with a wide range of configurations. Some home users use an older version of a browser because they do not want to spend the time downloading a newer version over their slow dial up connection. And, while "techies" will download a new version as soon as it is released, non-technical users might not care about using the latest version of a browser.
testing with the 3.x, 4.x, and 5.x (if applicable) versions of Netscape, IE,
Operating System Compatibility
Testing must be done across different operating systems because the operating system can affect how things are displayed. Your Web site should be tested in the following environments:
· Windows 95 Gold
· Windows 95 OSR 2.0
· Windows 95 OSR 2.1
· Windows 95 OSR 2.5
· Windows 98
· Windows 98 Second Edition
· NT 4.0
· NT 4.0 w/ SP1
· NT 4.0 w/ SP2
· NT 4.0 w/ SP3
· NT 4.0 w/ SP4
· NT 4.0 w/ SP5
· Windows 2000
· Mac 7.5.5
· Mac 8.6
You must know your target audience. For example, if scientists will be viewing your site, you'll need to add Linux or Unix to this list.
When testing across the different operating systems, also test across the different browsers. It is helpful to create a matrix to show which combinations have been covered.
A good Web site is more than just something to look at, it is functional and interactive. The functionality of a Web site must be tested just as the functionality of packaged software. There is a trend to sell Web-based software instead of packaged software, so this type of testing has become more important.
Chances are that your site will have forms with text boxes for input, buttons, checkboxes, radio buttons, drop down lists, and other controls. These must be tested as they would be in a packaged application, but do not be surprised if you run into many issues. Web development involves scripting languages that are not as powerful as, say, C++, so the functionality might not be up to par on the first build. Pay particular attention to data validation and exception handling. For example:
· Is text accepted in a number field?
· Are wildcards allowed?
· Is a blank field acceptable?
· Are long strings accepted?
· Is the maximum length long enough to allow reasonable input?
· Are the initial values of checkboxes and radio buttons correct?
· Can only one radio button in a group be checked at one time?
· Do check boxes trigger the desired event?
Testing should be done via the browser to exercise functionality of any back-end database.
Typical tests include:
· Adding duplicate information
· Tests of boundary conditions
· Size of resulting data (use a wildcard to generate large results)
Security can be an issue too. If a user is required to logon with a password, test this functionality thoroughly. Make sure that a user cannot just type in the URL of the final destination to get around the password protection. Thoroughly test the functionality for changing the password. For additional security and encryption, check to see if the Secure Sockets Layer (SSL) protocol is being used. The URL should start with https when using SSL. Be sure to keep up with the latest security issues!
There are also functional issues with the Web page itself and the controls on it. Some things to consider:
· Do scroll bars appear when a list gets long?
· Can a control or frame be resized without causing undesired affects?
· If a user checks a check box, goes back one page, and then forward one page, is the check box still checked?
· If the browser is resized, does the content still display properly?
· Does the page display correctly when refreshed?
· Is there validation of required fields?
· Should certain controls be disabled or greyed out under certain conditions?
If a defect can't be reproduced, it may be because the page is cached and the defect only occurs under initial conditions where the page comes from the server. To check for this, clear the browser's cache often.
Frames can be good and bad. They look good and help organize a page, but there can be problems bookmarking pages in frames and if poorly designed, frames can lead to confusing navigation. Some issues to consider include:
· Do frames resize automatically and appropriately? Is the user able to manipulate frame size?
· Does a scrollbar appear if required?
· What is actually recognized by the Bookmark or Favorites feature?
· Are all pages which are included in the frameset reliable?
· Can a search engine find content within the frames?
· Do the frame borders look good?
· Are there any "Refresh" issues?
Animation is popular and can bring a page to life. Check for:
· Smooth animation
· Flashing during animation
Amount of animation (too much can be distracting to
If there are audio or video components on your page, test that the attributes of the plug-in are correct. This includes volume, replay, and auto-start.
· Does the audio/video clip start and stop at the correct places?
· Is the audio/video quality acceptable?
Do not forget to test printing of your Web pages by printing on a variety of popular printers. Printing can be unpredictable, particularly with frames. Keep an eye on what is printed, the readability of content, and the speed of the print job.
Navigation of any Web site should be easy, logical, and intuitive. A user should be able to navigate in all directions and not get lost or be unable to return to a previous page. All links must be functioning and must get the user to the desired page.
Links can be text or graphs, and some image maps might have multiple links. Check that all links are valid, change color when clicked, and open the expected page. Be sure to frequently test links to other URLs that might become out of date.
Besides links, there are the browser's forward and back buttons, the history list, the drop down list, and possibly forward and back buttons on the Web page itself. All these methods of navigation must be tested for functionality and usability.
If you have forms that require user input, test the tab order and know what should happen after tabbing off the last field.
The look of your Web page has a considerable impact on the user's experience. A page must look good and the look should be consistent throughout the site. Once again, it is critical to test across many browser and operating system combinations because the page can look different in each scenario.
Another concern is the screen resolution and color depth. A page might look good at a resolution of 600 x 800, but parts of it might get cut off at 640 x 480. Different color depths should be used on the test machines also. Colors might vary unpredictably if a browser-safe palette is not used.
Testers should look for:
· Color of links
· Broken images
· Low color contrast
· Spacing in tables
· Text wrapping issues
· Alignment, formatting, and size of text
· Alignment of controls such as radio buttons and check boxes
If your Web site is not user friendly and intuitive, users will quickly go somewhere else - probably to one of your competitors. To reduce the number of deserters, start thorough usability testing early in the development cycle when changes in design are easier to accommodate. Compare your site to other similar sites or "best of breed" sites. Some tests to consider are:
· Is the site easy to navigate?
· Is the look and feel of the site consistent?
· Can the user find information easily and intuitively?
· Is there a table of contents?
· Are menus standard and in the proper location?
· Is there a home button or link on every page?
· Is the user required to scroll to see the whole page?
· Are all targets, where a user must click, big enough for ease of use?
· Are there too many pop up windows?
· Is data persistent if a form is filled out and then the user goes back a page, or will the user have to refill all of the fields when returning to the page?
· If there is a file to download, is the size of the file displayed to the user?
· If a plug-in or component such as a viewer or audio player is required, is the user given a link and instructions to download it? For example if you have links to pdf files, is a link provided to where the user can download Adobe Acrobat?
· Is there contact information for your organization, including an e-mail link?
Remember to test for accessibility issues, such as making sure there is alternative text with images for people who use devices that convert text to speech.
Visitors to your site expect fast results and do not want to wait. While testing, you must be aware of performance and know what can affect performance. Of course, Internet conditions vary from moment to moment and this affects performance, but certain issues are directly related to the design of the site.
Graphics should be small enough that they do not cause prolonged download times. If the design is such that images fade-in using interlaced GIFs or progressive JPEGs, test to make sure the image is completely painted.
Measure how long it takes for pages to load. For example, how long does it take from clicking a Submit button until a new page is downloaded and the word "done" appears in the browser's status bar? This should be done many times and on several different days so that an average time can be calculated, which takes varying Internet conditions into account. An automation tool, such as Rational Software's Visual Test, can be used for this.
Testing should be done using different connection methods. Use a variety of T1 and modem connections so that office and home environments are sufficiently simulated.
Web site testing has become a critical function of software QA departments. Businesses are now judged in large part, by their Web sites; understandably, users demand that these sites be of the highest quality. With the incredibly fast speed at which Internet technologies are growing, testers must continually learn these technologies and know which browsers support them.
Web will become even more important as there is a shift toward
e-commerce, and product software becomes Web-based software. The testing of Web sites will become more complex and more important than ever.
Test Cycles are Frequent with Web sites and Web Applications
Testers must keep abreast of emerging Internet technologies, and must know the issues that accompany each technology and recognize which browsers support which technologies. The constant development of new technologies also means that Web sites must continually be changed and improved. This in turn means that there will be many development and test cycles. This requires that you are aware of the changes that are made to your site and have good configuration management and change management control. Start testing as early in the development process as possible. Ideally, this means you would be able to review the specifications before any coding has begun.
of the shift toward e-commerce and Web-based software, testing Web sites is
more complex and more important than ever. By following the suggestions
outlined in this article, you'll be well on your way to keeping your site
visitors coming back.
FEATURE ARTICLE - Testing Web Applications
Priming the Team
Andrea MacIntosh, QA Labs Inc.
***Web applications vs. web pages***
What do we mean by web application? There is an incredible range of sophistication in web applications from a simple company website with some order fulfillment to sites like Yahoo or Amazon. One way to look at the web application architecture is to take the model of a traditional business transaction application and to replace the user front end by the website. A customer acquires goods and/or services from your company in exchange for money. Mechanisms are in place to facilitate that transaction between client and company. But instead of a sales rep, a clerk, or a cashier, you have a browser pointing at a website. The company is never closed! Customers can serve themselves!
Think of a vending machine. It fills orders based on input from users, verifies transfer of funds, and has a basic user interface. Now add some complexity. Make the UI a browser-based solution that must run in multiple browsers on multiple operating systems instead of a touchpad, oh, and have the machine fill orders directly from a warehouse while tracking (real-time) inventory. And to top that off, people won't be putting coins in the machine, but swiping their credit cards in a reader - so you'll need to have each transaction approved by the credit card companies. Hey, and another great idea would be to assign each customer a user name and a
***The development life cycle and its impact on testing***
Most of us have been exposed to a few software development life cycle models, such as the Spiral model, the Waterfall model, and so forth. The typical software project includes phases such as planning, requirements gathering, analysis and design, implementation (coding), integration, testing, release and maintenance. Your team needs to know how thesephases match up for a web application project.
The phases are somewhat similar, but without the sometimes lengthy timeframes previously seen in the industry. Web applications are software and, as such, fall under the same rules as all software development projects: you need requirements, design, implementation, and testing as a bare minimum. And if you want to limit risk, you need sound planning and management like in any other software project. Even speed is similar in the sense that marketing always wanted the product yesterday. Only now "yesterday" is even earlier.
The best way to reduce the risks when testing a web application is to add formal test planning and analysis to the beginning of the project life cycle. Every project has testing at the end of the project life cycle. When the development schedule slips, testing time is almost always reduced to make release or "go-live" dates. Adding test planning to the beginning of the project life cycle will allow testers to prioritize their testing efforts based on risk, schedule constraints, and testers' abilities and aptitudes. This will be crucial to managing the test effort when the time crunch hits prior to going live.
***Five ways to prepare now***
Testing a web page with relatively static content and few to no forms will take very little time. Testing a web application will require much more sophisticated testing strategies and thus more time. Due to the nature of web development, your team probably won't get more time and maybe even less than for a traditional development project. You can save time at the end of the project by using "downtime" in the beginning to prepare your test team for the task ahead.
a. Learn more about the environment in which you'll be working.
Testers should familiarize themselves with subtle browser, operating system, web server, and database differences. The more they know about scripting (ASP, XML, HTML, etc.), databases (Oracle, SQL, etc.), web servers (IIS, Apache, etc.), and the data transfer behind the UI, the more effective they are. Testers simply can't just test the functionality by exercising the UI (in this case, the browser). They will miss all the other kinds of testing required for web applications (such as performance, security, database integrity). Remember, crackers don't use browsers to crack sites; they use scripts.
b. Find or create appropriate test tools.
The lack of mature test tools makes automation difficult. Remember when Java first hit the scene? Developers and project managers alike wanted to use this new technology. Testers suddenly had their workloads doubled, tripled, or more, simply because of the number of configurations and the lack of any mature testing and automation tools available. More test tools are available now, but it will still take time to source an appropriate tool, learn its ins and outs, and even customize to your environment. If a tool is not available, you'll want to find out sooner rather than later and build some test apps of your own.
c. Create a matrix of operating systems vs. browser versions.
Grossly large amounts of browser and operating system compatibility testing are required. If you create a matrix of operating systems vs. browser versions, you'll have a method of attacking all the variations.
d. Define your development and test environment with version control and other configuration management efforts.
If you are testing web applications without defining your environment, you'll face issues such as:
- How do you roll back code changes when you have no previous build?
- How do new functionality and defect fixes get migrated into each build?
- Does the term "build" mean anything in the web-space?
Test personnel cannot revert back to a "known state" if the source code is not being archived or not being labeled or branched in the version control repository. Not having a previous release to revert to for testing purposes makes isolating and analyzing defects more difficult, as the environment continually becomes more complex. If you set up a test friendly environment, you won't have to face the problems arising from the above questions.
e. Set up a separate test server.
A common (and dangerous) practice of web testers is migrating defect fixes and new functionality to a live server prior to testing, and testing on live servers. Your test team shouldn't bring down your site; they should bring down a separate test server.
These five steps should help prepare your team for the challenging task ahead. In a future article, I will outline a series of specific questions you should be asking when beginning your test planning, and how the answers to these questions can be used to define and focus your testing strategy.
FEATURE ARTICLE - Testing Web Applications Part 2: Planning
In "Testing Web Applications Part 1: Priming the Team," I discussed some of the reasons why testing a web application is different than testing something like a desktop application. In this article, I am going to outline some key points to help you keep focused and hopefully allow you to avoid common pitfalls when planning your testing effort.
When to Plan
Day 1. Period. As soon as the marketing requirements are gathered and have stabilized/been signed-off, test planning should begin in earnest. This rule of thumb should be applied to all software development projects, but for the web application project this rule must be applied. The timeframe for these kinds of projects is so short and they are under such intense pressure to go live at a specified date, that test planning must begin as soon as possible.
How to Plan
I've listed a condensed set of items to think about when planning to test a web application. Often this type of information is not made available to testers in any written form...or even in any oral form! Most test teams get stuck with an incomplete picture when starting their test planning activities. The items below are designed to help you fill in the gaps, where information is often miscommunicated or not communicated at all. Think ahead to two weeks before release when the product manager says to the test team: "You've tested on IE 4.5 on Mac 9.0, right?" This is one way to ensure you can answer the question.
This is useful if only to eliminate the non-purpose elements from your test planning. For example, if the site's primary purpose were one of information exchange, not e-commerce, then the test planning would have less detail or focus on the e-commerce functionality of the site.
Knowing this will help you focus on the type of user and the most common functions they will perform at the site, as well as their expectations! Useful User Scenarios can then easily be created to help define your testing scope and focus.
Perhaps reliability and security are the key areas on which to focus your testing; perhaps it is speed and performance. Asking these kinds of questions will help you focus on the most important areas. If performance and load testing are criteria for your web application, and you've never done this kind of testing before, get help from people who are more technically savvy in this area. A good place to start is with in-house development resources, maybe they can create some scripts that test the web server directly and that can simulate many users.
Then count them. This will allow you to generate ball-park estimates of the number of test cases required to test the functionality of the web application. A quick rule of thumb is to have a minimum o four test cases for each distinct functional requirement: a valid case, an invalid case and two boundary cases (upper and lower). A closer inspection of the functional requirements themselves will reveal areas that will require more test cases than the minimum to achieve adequate functional coverage.
For example: 100 functional requirements x 4 test cases per requirement = 400 test cases. This is a minimum. You will still have to go through the requirements and their associated test cases to ensure the level of test planning is adequate for all the requirements. Having test cases that are traceable back to the originating requirements will make this task much simpler.
Since this area is what will multiply your testing effort the most quickly, it is a good idea to get a complete understanding of the scope desired by project management. This is one area where test planning can reveal the huge Impact adding an extra browser or operating system will have on the test schedule. This is also the one area where prudent negotiation, pre-planning, and risk management can keep your test schedule under control. The clearer your understanding of the testing required on one operating system/browser combination, the more easily you can communicate the costs of additional testing for other platforms and/or browsers.
For example: 400 test cases x 4 environments = 1,600 total test cases. Each additional test environment (i.e., operating system/browser combination) adds 400 test cases to your test effort. Imagine if it takes 1 person-week (5 working days) to execute 100 test cases. Each additional test environment would add 4 person-weeks of testing.
Try not to be too verbose or overly formal: this document is a high-level view for the test team to use throughout the project. It is not designed for use by the rest of the project team, nor as a replacement for other project documentation. Hopefully, much of this information will already be written in some document(s) to which you can refer. Once that is done, have it reviewed by the project stakeholders for accuracy.
I suggest creating them in this order: valid functional test case; invalid functional test case; boundary functional test cases, User Scenario test cases. Thus, you would create a set of valid functional test cases before your boundary functional test cases. Likewise, you would create User Scenarios by linking together your valid functional test cases (notice that the User Scenarios consist of functional test cases, and thus the "ball-park number" mentioned in step 4 above is still valid). If your timeframe is very short, create the User Scenarios before the boundary functional test cases (these will give you more bang for your buck); otherwise, create them afterwards to ensure you test all boundary conditions. Prioritize them in order of importance and frequency. The test cases you will run the most frequently or that test critical functionality would be of a higher priority than those you would run infrequently or only once.
In "Testing Web Applications Part 3," I will address the act of testing this web application, including some ideas for tools and tricks to simplify the tasks that lay ahead.
FEATURE ARTICLE - Testing Web Applications Part 3: The Attack!
In the previous article, I suggested some ways to plan testing a web application. For this last article in my three-part series on web testing, I will address the act of testing this web application, including some ideas for tools and tricks to simplify the tasks that lay ahead.
Test from Back-to-Front, Not Front-to-Back
Many testers focus on testing a web application through the browser. However, this can result in spending extra cycles testing the browser, not the web application itself. Of course, you still need to test within the browser, but don't let it become the focus of your testing. This is particularly important when performing load and performance testing. If your web application is like most, you'll have a web server that connects to one or more application servers that are then connected to a database. Therefore, performance and load slow-downs could occur at the web server, the application server(s), or the database itself.
Scripting Is Good
Use Risk-Based Testing
This is still one of the best techniques available to testers today. At the beginning of the project, the test team should meet regularly to generate a "Top 10 Risk List," which lists all the risks to testing (specifically) and the project (in general). Then, as the project progresses, meet regularly and update this list. Use this list to help target your testing. For each risk identified, plan ways to reduce or minimize the impact of the risk occurring. Many of the possible solutions will affect your testing strategy. That's OK - this is when you need to embrace change to minimize your risk of failure. As new problems crop up, you will have to adapt your plans and strategies to deal with them effectively. So don't get locked into "tunnel vision," where you blindly follow the original test strategy and swear to the heavens (and your development team) that it's the *only* strategy that will work. Risks change throughout a project, so must your strategies change to deal with them.
Remember That UI Is Not the Same as Usability
Just because your web application looks good, doesn't mean it is usable (or that it exhibits good usability). The reality is most websites and web applications display marginal to poor usability. Many web application UI designers come from a graphic design or desktop publishing background. This is useful for text-based websites -- what the Internet was originally designed for; but, web applications need a greater focus on Computer-Human Interface design.
The five key areas to remember when thinking about usability are movement, perception, language, memory, and thinking. Three system properties to apply to each of these areas include:
Some things to test for:
Prioritize Testing Environments
If your list of supported browser/operating systems is hovering or is well over the double-digit mark, you'll need to pick a few environments for testing (unless you have a dozen or more testers, or have figured out how to clone humans successfully). Pick the latest Internet Explorer and Netscape versions, then pick *one* older version of each. That's four browser versions. Assign one of each browser type to Windows and the other to Macintosh. Example: IE 5 on Windows 2000 and Netscape 4.7 on Macintosh 9.0, plus Netscape 4.0x on Windows 98 and IE 4.0x on Macintosh 8.6. That's four configurations, with enough mix'n'match to achieve adequate coverage. Of course, you'll still have to test your high-priority (must pass) test cases on the other permutations (particularly security test cases), but this strategy seems to provide the largest gain for the effort expended.
These are just a few hints for testing web applications. I hope they'll save you some time and much grief as you and your team work to get your application out the door. However, these tips will not replace experience. The more work you do in this area, the more experience you gather, the more readily you will be able to plan and execute testing a web application.