Access and Scrape More Data With Cookie Stores

Mozenda is always working to bring you greater access to the ever-expanding internet. To deliver on that promise, we are excited to announce our latest feature: Cookie Stores.

With this new feature, all accounts can now save cookie sessions to enable easier access to the data you want to collect.

Potential use cases include:

• Logging into websites that use two-factor authentication
• Logging into websites that use CAPTCHA validation
• Collecting personalized data such as shopping carts and order histories

*Please note that individual website parameters still apply when using the Cookie Stores feature. For example, if you wanted to scrape auto insurance quotes and you are were using data stored in the Cookie Store to provide the car and driver information the website may only authenticate cookies that are less than 30 minutes old. The lifecycle of a cookie is set by the website that stores them in your browser, and these restrictions will still govern your scraping abilities.

The new Cookie Stores feature gives you the ability to access websites without having to re-enter credentials each time you visit. With a cookie store, you can pass through a website’s authentication features such as username, password, two-factor authentication, and CAPTCHAs without recreating these steps each time you run your Agent.

Why Cookie Stores is Valuable
Cookies are small files saved to your browser by websites as you visit them. These files act as an identification card or history of what you have done on the individual site. Next time you visit the site your browser will send back the stored cookie information to the website, enabling the website to “remember” you and provide a better browsing experience. Cookies can be used to save your session, preferences, progress in filling out forms, personalized information, and other information to improve your experience each time you return to the site.

Mozenda’s new Cookie Stores feature allows you to save your cookies for later use in building and running your Agents. As the website recognizes your cookies and restores your session information and preferences, your Agent design can be simplified. For some websites this can be used to collect data from the website without having to reauthenticate (entering passwords, completing two-factor authentication, solving CAPTCHAs, etc).

Here’s a look at Cookie Stores in the Agent Builder.

How to Use Cookie Stores
Creating a Cookie Store in Mozenda is simple. Begin by opening the Agent Builder in the browse mode (this is before you have started a new agent or opened an existing agent). Next, navigate to the website and enter the required information such as user name, password, verification code, two-factor authentication, or CAPTCHA. Once the page is set up, click the “Save Cookies” button. This will bring up a window prompting you to…

• Name the Cookie Store
• Add an optional description
• Click “Save”

Now that the Cookie Store is saved, it can be accessed by any Agent in the account, and there is no limit to the number of Cookie Stores that can be saved to an account. On corporate accounts, any user with permission to create Agents can access all of the cookies within their corporate account. This provides all users access to data locked behind authentication gates without sharing sensitive details such as specific user names and passwords.

Previously saved Cookie Stores can also be updated from the save dialog by overwriting an existing Cookie Store. All Cookie Stores can be accessed and managed through the “Manage Cookies” functionality.

When you’re ready to use a cookie in your agent, click “Load Cookies” to add it as a step within your Agent. You’ll have the option to choose the desired Cookie Store from the “Manage Cookies” popup.

Example Use Case
One example of this new tool’s functionality is the creation of a Cookie Store to log into a shopping account and scrape order history. The gif below gives you a look at how to save the required cookies and then add the Load Cookies step to your agent. The user’s shopping history is then accessible for scraping.

For step-by-step instructions on using the new Cookie Stores feature, please go to visit our Help Center article.

“I am personally really excited about Cookie Stores,” said Chris Curtis, Mozenda System Architect. “It enables you to get information from websites you simply couldn’t scrape before. We’re providing better data access, and we’re doing it within the bounds each website has set without requiring users to store passwords or other authentication data within their agents.”