What Security Criteria Should You Evaluate When Buying an A/B Testing Tool?

Well, if you are a part of the experimentation ecosystem, you already know that Google Optimize is bidding us goodbye. Time to look for a few Google Optimize alternatives!

Yes, you need to free up your schedule, sit in for a ton of demos, and take trials.

You know what to look for, but there’s so much out there that it’s hard to figure out which features are most important.

When looking for an A/B testing tool, among other things, you should also evaluate the security of the platform. And in the context of experimentation, security means three things:

1. Preventing leaking of data
2. Preventing manipulation attempts
3. Encrypting all communication channels between your systems and the A/B testing tool provider’s system.

To help you make the right choice, in this article we’re going to share a list of security questions to consider when choosing an A/B testing platform.

Is the A/B Testing Platform Compliant With Any Security Standards and Certifications?

Over the past ten years, more and more enterprises have started requiring security and compliance certifications from SaaS and other service providers before awarding contracts.

This has increased the demand for certifications such as SOC 2, ISO 27001, etc.

By obtaining these accreditations, organizations can standardize their cybersecurity precautions to protect data and preserve their brand integrity. Some of the most popular ones for A/B testing are:

• ISO 27001: This is a standard for information security. Your A/B testing platform must comply with best practices in information security and regulations in order to be certified.
• SOC2: The Service Organization Control 2 (SOC2) certification ensures that your A/B testing tool has a comprehensive set of internal controls in place.
• PCI-DSS: This standard ensures that credit card information is maintained in a secure environment if your A/B testing platform processes, stores, or transmits it.

Check if the A/B testing platform you intend to use meets any of the above criteria, or better, all of them.

Note: Convert’s support team can answer any questions you may have, including security-related inquiries. Contact support@convert.com if you wish to report a problem or find the answer you need.

How Often Does the A/B Testing Platform Perform Penetration Tests?

Penetration testing is a process that allows an organization or individual to evaluate their security systems by simulating real-world attacks. Penetration tests or pen tests should be performed regularly and at different intervals depending on the type of data being stored and its sensitivity.

When you conduct A/B tests, a pen test could protect you from

• Unprotected access points
• Weak passwords
• SQL injection attacks
• Code injection
• Cross-site scripting
• Data breaches
• Phishing attacks

Does the A/B Testing Tool Offer a Bug Bounty Program?

A bug bounty program rewards ethical hackers who find and tell the application’s developer of vulnerabilities or issues. This allows businesses to increase the security of their systems by leveraging the hacker community.

Since bounty programs attract a diverse group of hackers with varying skill sets, companies using bounty programs are at an advantage over those who use less experienced security teams.

That’s why businesses often use bounty programs in addition to normal penetration testing to assess the security of their applications throughout the development process.

Does the A/B Testing Platform Encrypt Data Both in Transit and at Rest?

Encryption is a security best practice that protects data in transit and at rest, preventing unauthorized access to the information.

Most A/B testing tools encrypt your data at the highest level possible (SSL/TLS). This means that while it’s being transmitted over the internet, it’s protected by encryption technology that keeps prying eyes from seeing what you’re doing with your customer data.

If you’re using an A/B testing tool that doesn’t offer encryption as part of its service or only offers limited protection for sensitive information, it’s time to look elsewhere!

Does the A/B Testing Platform Support Single Sign-On?

With Single Sign-On (SSO), users can log in to multiple connected but independent software systems with a single ID and password. By sharing the authentication token between multiple applications, SSO authentication allows users to remember fewer passwords, which improves security.

This makes it an important security feature to look for in an A/B testing platform.

Who Will Have Access to Your Data?

A/B testing platforms handling PII data should adhere to the principle of least privilege, which states that employees should only have access to the data they need to do their duties. By doing this, only a portion of each vendor’s staff will be allowed access to the technology stack’s storage areas or transit zones, reducing the risk of both malicious actors and user breaches.

What Type of Information Is Logged? How Long Is It Available?

When choosing an A/B testing platform, you should also consider the tool’s logging capabilities.

• What information is logged, and how long are logs available?
• How long is data stored in the cloud?
• How long is data stored on its servers?
• How long is data stored on your client’s devices (if applicable)?

It is important to know how long you have to retrieve information from these sources when you need to conduct an audit or investigation.

Is There an Incident Response Plan in Place?

When you’re evaluating an A/B testing tool, you need to consider how the company handles security incidents. It is especially important if you plan to use the software in a production environment and want to be sure that your data is protected. For example,

• Do they have an incident response plan?
• What steps would they take if their system was compromised?
• How will they communicate with customers about the breach?
• Who will be responsible for handling these situations?
• How often does their team meet up internally and externally (e.g., with law enforcement) to discuss potential threats or best practices surrounding them?
• If there’s ever a breach or other issue involving user data, what kind of information would need to be released publicly so that users can take action accordingly?

What Happens to Your Data if You Choose to Stop Taking A/B Testing Services?

Compromises in your A/B testing platform compromise your own systems as well. If your data is stored on a vendor’s system, you are at risk.

When you stop using an A/B testing tool, make sure all of your data has been deleted from their databases. Some providers treat this final phase as an afterthought, but it’s imperative that they delete all of your data. The last thing you want is massive data repositories or breadcrumbs scattered all over the vendor ecosystem. This increases exposure and overall risk.

How Do You Audit User Activity on Your A/B Testing Platform?

It is not always possible to see what activity is taking place within an A/B testing tool. Some tools don’t allow you to see critical information, like

• Who made a change to a particular Split URL test
• Who added a GA integration to a particular deployment
• Who deleted some critical code, preventing an experience from running

A log of user activity is crucial to securing a platform as it ensures users are accountable for their actions and can be used as evidence as needed in the event of a security breach.

Does the A/B Testing Vendor Choose Third-Party Vendors Carefully?

Make sure that the vendor you’re considering has a good track record and is reputable.

• Check for tool security issues or if the vendor works with third parties that could threaten your data.
• Read reviews from previous customers. Find out if there are any common complaints about poor performance or worse, data breaches due to poor security practices on the vendor’s part.

Does the A/B Testing Tool Have the Visitors’ Consent To Process the Personal Data Collected?

Choosing an A/B testing platform should also include asking the following questions:

• Did the vendor recommend updating your privacy policy?
• Has the visitor’s consent been obtained for the tool to process their personal data?
• Is the privacy policy easy to find on your website, and does it explain how you collect and use visitor data, including personal information (e.g., names, email addresses)?
• Does it include information about how visitors can withdraw their consent at any time?
• Do you have a procedure in place for obtaining consent from visitors who want to receive marketing messages via email or SMS?

What Data Does the A/B Testing Platform Collect? (e.g. Traffic, Clicks, Page Views, Demographic Data, etc.)

Testing data collected from your website(s) is transmitted to the A/B testing vendor’s servers and stored there. Traffic, visitors, IP addresses, and location are some of the most common data collected by A/B testing tools. These can be used to make decisions about whether a test is successful or not.

You can use A/B testing tools to determine how different versions of your website perform in real-world environments. In order to do so, your tool needs to collect visitor behavior data, including

• Traffic information: How many visitors came to your site, where they came from (referrals), and which pages they viewed on your site.
• Visitor information: The device that each visitor uses (iPad vs iPhone vs desktop), the operating system on the device, their geographic location (city and country), what browser they use, and other similar characteristics.
• IP address information: An IP address uniquely identifies every computer connected to the internet at any given time. Because these tools keep track of every visitor who comes to your site, they can tell if someone has visited more than once or if someone visited multiple times within a short period of time
• User behavior data: This can help you understand how your users react to a new design or layout. You can also see how long they spend on a particular page and which elements attract their attention more than others.
• Number of conversions. You can, for example, use an A/B test on a landing page for your product to see how each variation performs in terms of conversions in comparison with the original.

What Tracking Technology Does the A/B Testing Platform Use? (e.g. Fingerprinting, Local Storage, Web Storage, Cookie, Pixel, etc.)

Several methods are used by A/B testing platforms to track users across devices.

Some companies rely on fingerprinting, which is the process of identifying a device by its unique fingerprint. This method is the most accurate, but it’s also more invasive and requires more work from the user.

Cookies are another common method for tracking users, though they’re far less accurate than fingerprinting because they rely on a lot of data being sent back and forth between the browser and the server every time a user visits a page. Cookies can also be disabled by users who want to avoid being tracked.

Other methods include local storage, which stores data locally on your computer, or mobile advertising IDs (MAIDs), which identify mobile devices based on unique identifiers.

Always ask your A/B testing tool what method it uses.

We hope this article helped you understand why security is important in A/B testing tools, and how to evaluate your options. Taking the time to thoroughly review these criteria will help ensure that you ultimately select a platform that is secure and reliable. Here is a quick comparison of the most important security features in a few popular A/B testing tools: