<aside> ℹ️ Instructions

  1. There are two sections in this test, both about improving Faire’s retailer verification process. Section 1 contains two short answer questions, while Section 2 is comprised of five prompts guiding you through solving a key customer pain point. We recommend you spend 45 minutes on Section 1, and 2 hours on Section 2.
  2. Please read the “Context” sections carefully, as they contain important qualitative context that is relevant for answering the prompts.
  3. In addition to writing answers to the prompts, please link your raw data analysis from Section 2 to your answer or attach it to the email you send back to us. </aside>

Context

At Faire, all retailers must go through a verification process before receiving orders on Faire. This ensures that our retailers are eligible to buy on Faire (e.g., intend to resell the products they buy), and mitigates risk from bad actors. Our goal with verification is to verify retailers as quickly and accurately as possible.

We put retailers into our verification flow after they place their first order on Faire. First, we try to automatically verify retailers via a data-driven model. For all retailers who are not automatically confirmed, we send them a form requesting information about themselves and their shop that they must complete. After they submit the form, we review their information and follow up with the retailer as needed to clarify the information they submitted, before making a decision to either confirm or reject the retailer.

If a retailer is rejected or does not complete the verification form in 30 days, their order will be canceled. If they are confirmed, their order will be confirmed and sent to the brand for fulfillment, and the retailer will be considered a “confirmed retailer.” During the verification process, the retailer’s order is on hold and will not be shown to the brand.

Screenshot 2022-12-01 at 11.21.46 PM.png

Section 1: Short Answer (45 minutes)

Please aim to spend no more than 5-7 minutes on each question, and keep your responses concise. We recommend writing no more than 1-2 paragraphs per question.

  1. In our verification form, retailers are often required to upload photos of their shop. We got feedback that retailers were confused regarding what types of photos we were looking for, so we ran an A/B test where for 50% of retailers required to upload photos of their shop, we clarified instructions for what kinds of photos to upload, and separated photo uploading into its own step in the form. The other 50% of retailers saw our existing experience. Our hypothesis was that having a clearer file uploading process would result in more retailers completing the verification form, and our goal was to increase the number of confirmed retailers.

    The screenshots below illustrate the new and old experiences for an online retailer.

    Old Experience

    Screenshot 2022-12-03 at 6.54.59 PM.png

    New Experience

    The new experience includes updated photo uploading instructions, and separates the photo uploading question into its own step in the verification form.

    The new experience includes updated photo uploading instructions, and separates the photo uploading question into its own step in the verification form.

    We saw the following impact on metrics for retailers who were shown the new experience, compared to retailers who saw the old experience, using 90% confidence intervals to determine statistical significance:

                    A statistically significant +3.0% increase in **verification form submission**
    
                    A not statistically significant +1.0% increase in **confirmed retailers**
    
                    A statistically significant +25% increase in **confirmed GMV**
    
    1. Did we accomplish our goal with the changes? Why or why not?
    2. If you were to continue working on improving verification, what implications would this test result have on what you tackle next?
    3. This test was run on retailers who accounted for 2% of Faire’s overall GMV while the test was running. How would you compare this feature’s impact on GMV with another feature that impacted all retailers on Faire and drove a statistically significant +0.25% increase in GMV?
  2. We’ve heard that retailers feel like our verification process is a hidden hoop, since it happens after they provide payment information and place their first order. We’re thinking of improving our verification experience by moving the process further upfunnel before they place their first order, and sending retailers the verification form shortly after signup. Retailers would not be required to submit the form until after they place their first order, and a verification decision would still only be made after retailers place their first order.

    Screenshot 2022-12-01 at 11.30.08 PM.png

    1. Our primary goal is to drive an increase in confirmed retailers (i.e., the number of retailers who have placed an order and been confirmed). What do we need to assume about how this will impact retailer behavior for us to accomplish our goal via this change?
    2. The number of confirmed retailers is too noisy to use as a primary metric for measuring the impact of this feature. Another PM suggests using verification form submission rate as a proxy metric, defined as the percentage of retailers who are sent the verification form that complete it. Is this an effective proxy? Why or why not?
    3. What other metric, or combination of metrics, can we use to proxy the number of confirmed retailers?

Section 2: Improving Verification (2 hours)

We’ll now switch gears — this section focuses on how the retailer verification process can cause pain for brands who refer retailers to Faire. The prompts below will guide you through solving this customer problem. Please read the “Context” and “Data” sections carefully, as they contain relevant information to the problem.

Context

Our retailer verification process can cause pain for brands when they refer retailers to Faire. We’ve gotten feedback from brands who are frustrated that the retailers they refer are either getting rejected or stuck in the verification process for a long time. This is problematic for brands, not only because they don’t receive orders from the retailers they referred until verification is complete, but also because it makes them lose credibility and potentially burns bridges with retailers — brands spend time and effort convincing their retailers to sign up and place orders through Faire, only for them to have a frustrating verification experience.

We currently don’t give brands any visibility (e.g., in-product / email communications) while referred retailers are in the IDV process. Brands typically realize referred retailers are stuck in verification when retailers reach out to them, which prompts the brands to reach out to us.

We think improving this experience for brands can help us with our overall goal of verifying retailers as quickly and accurately as possible — imagine you’re the PM in charge of this initiative.

Data

See this link for sample data on our verification process — please make a copy for your own use.