Inside Look: Browser Isolation and Improved Stability

The Increasingly Complicated Web

The Internet has transformed in the past decade. Websites are becoming increasingly sophisticated in their design and complicated by the technologies they use. This increasing complexity presents new challenges for web scraping companies. The technologies many web scraping companies rely on are no longer robust or efficient enough to handle the complex nature of these new websites.

At Mozenda, we took a forward-looking approach to this challenge by building technology that scrapes web pages the same way a user navigates them. This means that if you can browse a web page using a modern web browser, you can scrape it with Mozenda, regardless of how the web changes in the future. And our Mozenda Cloud allows you to do this without purchasing and maintaining any hardware or software, or working with your IT department to provision the necessary resources. We handle all of this for you and do it cost effectively!

With our March release, we’ve enhanced Mozenda with new technology to improve stability and performance. We call this technology Browser Isolation.

Using Browser Isolation in the Agent Builder

We originally introduced Browser Isolation in our November release and provided an option to enable Browser Isolation in the Agent Builder. Since then, we’ve added numerous improvements that make using Browser Isolation the best way to use the Agent Builder for almost every website.

To enable Browser Isolation in the Agent Builder, click File, then Advanced Features, and then check the box titled Browser Processes. You can read more about this in our previous blog post.

What Exactly is Browser Isolation?

The Agent Builder, represented by gears in the following images, creates a Mozenda Browser for each page defined in your Agent and uses this browser to perform actions on the page. Website processing, such as Javascript execution and AJAX requests, also occur in the Mozenda Browser. Prior to Browser Isolation, every Mozenda Browser was created and ran inside the Agent Builder application.

With Browser Isolation, these Mozenda Browsers are created outside of the Agent Builder application in their own compartmentalized, hidden applications called processes.

You can see these individual processes in Windows Task Manager on the Process tab. They show up as Mozenda.Browser.exe. Also notice the Mozenda.ClientAgentBuilder.exe process. This is the Agent Builder application. In Windows, the terms application and process can be used interchangeably. We’ll refer to these as application processes for the remainder of this post to avoid confusion.

One way to look at how Browser Isolation works is to use a well-known analogy. Without Browser Isolation, all eggs (the Mozenda Browsers) go into one basket (the Agent Builder). With Browser Isolation, each egg gets put into its own basket. With web scraping, there are significant benefits to putting each egg into its own basket.

The Benefits of Browser Isolation

The idea behind Browser Isolation is not new. Browser Isolation is our implementation of an already-existing, great idea.

The developers of the Chrome browser realized there were significant benefits to taking this approach. They wrote documents and blogged about their architecture. Now, all modern browsers follow their lead by including similar technology.

In one of their documents, the Chrome developers explain:

It’s nearly impossible to build a rendering engine that never crashes or hangs. It’s also nearly impossible to build a rendering engine that is perfectly secure…All it takes is one browser or plug-in bug to bring down the entire browser and all of the currently running tabs.

Browsers don’t usually crash when used by users because users navigate websites slowly and for short periods of time. The effects of mistakes in website code do not manifest themselves well under these conditions.

In web scraping, however, a browser navigates quickly and for long periods of time. Small problems in website code, over time, turn into big problems, such as slow performance or crashes.

Improved Stability Through Browser Recovery

Isolating each Mozenda Browser into its own process allows the Agent Builder to gracefully recover the Mozenda Browser when a misbehaving website crashes it. Instead of crashing the Agent Builder, the website crashes only the specific Mozenda Browser executing the website code at the time of the crash. When this happens, the Agent Builder simply creates a new Mozenda Browser and continues processing the Agent where it left off without the user ever knowing the crash happened.

Improved Memory Management

Some websites execute JavaScript code that doesn’t entirely clean up its memory. In the past, these memory leaks sometimes caused the Agent Builder to become less responsive or crash when defining or testing large or long running websites. With Browser Isolation, the Agent Builder effectively manages this problem by shutting down and replacing Mozenda Browsers before the website JavaScript uses too much memory. All of this is done behind the scenes without the user ever knowing it’s happening.

Improved Performance and User Experience

Browser Isolation also improves performance and the user experience. Tasks that were previously performed in the Agent Builder, such as automated action testing, data previewing, page highlighting, and browser rendering, were moved to the Mozenda Browser where they happen concurrently. The result is a more responsive Agent Builder experience.

Overcoming Technical Challenges

During development, although we encountered many technical challenges, two stand out as having been particularly important to solve:

1. Passing information between the Agent Builder and Mozenda Browsers
2. Sharing browser session information

Inter-Process Communication

In Windows, each application process runs as a unique entity. Normally, they do not have access to each other’s information.

Because each Mozenda Browser runs in a separate application process, we created a way for the Agent Builder and Mozenda Browsers to communicate. They do this by sending small messages to each other. This is often called inter-process communication.

When you use the DOM Viewer, HTML Viewer, HTML Element Inspector, and other tools, the Agent Builder communicates with the appropriate Mozenda Browser to highlight the page, capture information from the page, and so forth. In some cases, the amount of communication taking place is significant. Because of this, unintended consequences of the messaging system surfaced, which initially resulted in a less-than-ideal user experience.

We solved this by moving these tools into the Mozenda Browser application process to eliminate the need for the tools to communicate with the Mozenda Browser using messages.

Sharing Browser Session Information

The Mozenda Browser uses Internet Explorer to display web pages and execute JavaScript (read more about this).

When Internet Explorer connects to a website, a low-level module in Windows called WinInet creates something called a Session (represented by the red S in the following images). This Session holds information pertinent to Internet Explorer’s interaction with the website, such as cookie information.

When Internet Explorer requests information from a website, the Session information is sent to the website so the website can uniquely identify the browser, and consequently, the person using the browser, as a unique visitor. This is how a website remembers who you are when you browse from page to page.

A separate Session is created and maintained for each application. Prior to Browser Isolation, this was not a problem because all Mozenda Browsers ran within the Agent Builder application. They all shared the same Session.

With Browser Isolation, each Mozenda Browser is a separate application process, which means each uses a separate Session. This is problematic because each Mozenda Browser would appear to be a unique visitor to the website even though they would be performing actions for the same Agent.

We solved this problem by replicating the Session behavior ourselves in the Agent Builder.

When a Mozenda Browser needs to perform an action on a page, it requests the latest Session information from the Agent Builder, performs the action, then reports back to the Agent Builder any changes that occurred in the Session during the execution of the action. The Agent Builder thus acts as a broker between the Mozenda Browsers so they share the same Session information.

Working Together

The Agent Builder and Mozenda Browser application processes work together to show you what you see when using the Agent Builder. Each maintains a set of specific responsibilities.

Agent Builder Responsibilities

– Maintain the Agent definition
– Provide UI elements for defining and testing the Agent
– Manage the Mozenda Browser application processes
– Maintain web browser Session information
– Display the appropriate Mozenda Browser and related tool windows
– Communicate with the Mozenda Cloud

Mozenda Browser Responsibilities

– Host the Internet Explorer browser control
– Perform actions on a web page
– Manage the DOM View, HTML Viewer, HTML Element Inspector, and other tools

The Agent Builder user interface is partly constructed of elements from the Mozenda Browser. Below is a screenshot of the Agent Builder showing the UI elements managed by the Mozenda Browsers: the main browser window and the tools window. These are outlined in red. When defining an Agent, the Agent Builder creates a new Mozenda Browser for each page defined in the Agent. When you select an action on a page, the Agent Builder displays the corresponding Mozenda Browser and tools for that page.

What Lies Ahead

We believe the Agent Builder released in March is the best Agent Builder yet. Our unique web scraping technology that navigates and performs actions on websites just like a user, along with our new Browser Isolation technology, provides a consistent, stable, and flexible platform that is easy to use while accommodating increased website complexity and sophistication.

This is great, but there’s more to be done! This year, we’ll introduce new features and usability enhancements that will continue to solidify our position as the leader in web data extraction.

As always, we’d love to hear from you. Call 801.995.4550 or email us at support@mozenda.com.