Faire operates as a wholesale marketplace facilitating connections between independent artisans, brands, and local retailers. The platform boasts an extensive network of retailers spanning 50,000 cities, fostering 7 million connections globally with over 100,000 brands. This expansive global platform is underpinned by a robust product and engineering team comprising several hundred engineers dedicated to driving innovation and scalability.
To sustain its rapidly expanding global business network, Faire manages a multitude of backend services and continues to add more. The conventional method of running these services locally via Docker Compose became increasingly unviable due to resource limitations and the complexity of diagnosing issues. At the same time, a different approach was used in the Continuous Integration (CI) process for end-to-end testing, creating disparities between the staging and production environments. Maintaining three disparate deployment methods for the same set of services not only imposed a substantial maintenance overhead but also resulted in a suboptimal engineering experience, exemplified by the common challenge of "working locally but failing in production."
Faire recognized the need to address the difficulties posed by the local Docker Compose setup and opted for a standardized method of deploying services on Kubernetes for development and testing. Given that Kubernetes is adopted in the staging and production environments, this approach would also enhance observability and ensure closer alignment between the development/testing systems and the staging/production ones. Clearly, a system built around on-demand environments that mimic production is necessary for efficient end-to-end testing.
It became clear that eliminating the challenges of the local docker-compose setup in favor of a standardized way of deploying our services into Kubernetes for dev/test purposes would be ideal.
However, to adopt such a Kubernetes-based approach, several challenges needed to be addressed. First, most engineers who need to work with the testing system daily lack the specialized DevOps expertise required for Kubernetes and Terraform. They have little motivation to acquire these skills due to the steep learning curve and the lack of relevance to their primary responsibilities. Combined with the necessity for safeguards to prevent a single engineer's error from disrupting the entire testing system, there is a pressing need for an intuitive and user-friendly platform built on top of Kubernetes to cater to end-users.
In addition, there is a potential concern regarding resource costs associated with managing numerous copies of end-to-end (E2E) testing environments. While an individual E2E testing environment may not require substantial resources, the cumulative expense can escalate rapidly when maintaining hundreds of such environments to accommodate every engineer's needs. Coupled with the tendency of engineers to forget to manually release resources, automatic lifecycle management became an imperative requirement for the system.
We considered building a platform to manage these sandbox environments ourself, but also looked at third party options, and eventually settled on Crafting as our preferred solution.
After careful consideration of multiple options, including building internally or using third-party systems, the Faire team chose Crafting as the preferred solution. Specifically, the platform engineering team at Faire leverages two key features Crafting offers: a) lifecycle management of end-to-end testing sandboxes on the testing Kubernetes cluster, and b) an intuitive and customizable UI for a superior developer experience.
Using the Crafting system, the Faire team created a testing template that encapsulates the end-to-end environment, running numerous services within a dedicated Kubernetes namespace. Engineers can then conveniently launch these environments on demand. Each pull request with a tag indicating the need for E2E testing triggers a git hook to create such an environment and run corresponding E2E tests on it. When the tests finish, the environment is recycled quickly and automatically, promptly freeing up resources without additional intervention from engineers.
An essential aspect of this approach is the reuse of config files for launching services in E2E Kubernetes namespaces, drawing directly from the production config. Compared to creating a new set of custom configurations or relying on Docker Compose, this approach not only reduces onboarding effort but also simplifies ongoing maintenance, preventing divergence across multiple environments. The capability to test in a truly production-like, end-to-end environment significantly increases overall reliability by reducing "surprises" when code is deployed to production.
Another key advantage of this approach lies in the capability to let engineers view logs easily and debug in place. Debugging issues found in integration testing has been a long-standing pain point for engineers, as it typically lacks good observability into the environment. At Faire, the on-demand Kubernetes environments are not only integrated with robust logging support but can also be retained in their current state for engineers to debug in case of test failures. Crafting also provides tooling that allows engineers to access pods in the connected testing Kubernetes cluster, enabling them to inspect the exact environments to validate assumptions—a feature that is invaluable for catching configuration issues.
Finally, Faire engineers can connect their frontend to an E2E sandbox for local frontend development. Instead of waiting for the corresponding backend changes to be merged and deployed to staging or production, engineers can develop the frontend by pointing their API to an E2E sandbox. This has significantly accelerated the iteration process for modifications that affect both the frontend and backend, leading to more efficient development cycles.
Faire solved the issue of standardizing dev/test environments and launching on-demand end-to-end testing namespaces with a solution based on Crafting. The system now runs at scale with around 1,000 daily launches of such E2E sandboxes, verifying code changes from an engineering team of hundreds.
Crafting has been a step change in our developer experience at Faire.
The improved user experience is celebrated by engineers. When the platform engineering team presented the new E2E sandbox solution in an internal tech talk, the engineers were excited and shared their enthusiasm in channel messages. Within one minute, the message board was filled with comments like, “Wow,” “This can’t be real! Too good to be true,” “OMG this is awesome,” and “I’ve never run BE locally. And now I never will.” Powerful and convenient tooling has encouraged engineers to test more and catch issues earlier in the development cycle.
Additionally, this powerful solution keeps resource costs low at Faire. Thanks to automatic lifecycle management, sandboxes are recycled quickly, and the corresponding Kubernetes namespaces are cleared promptly when tests finish, unless test failures require engineers to investigate the environment. This efficient process enables a high volume of tests to run through the system without an outsized node pool.
Furthermore, Faire minimizes the burden of maintaining the system by leveraging Crafting’s managed self-hosting solution. In this setup, the Crafting system is hosted on the customer side but is continuously monitored and managed by the Crafting team, which handles all updates and configuration changes to fit customer needs. “Crafting's managed self-hosting solution keeps our maintenance burden low and they are very responsive to both feature requests and to help with any troubleshooting needed,” noted Ben Poland, Staff Platform Engineer at Faire. “The integration process was very smooth and flexible, allowing us to adapt our existing tools and processes to get things working faster than I expected."
