Spotify is a household name for music lovers, but developers admire it for its internal tooling, especially Backstage—a platform for building developer portals. Ilya Lyamkin, a Senior Software Engineer at Spotify, shared his insights into how these tools optimize workflows, the impact these have on the company, and advice for professional growth.
Ilya, can you share your professional journey and how you ended up at Spotify? What attracted you to the company?
Before joining Spotify, I worked at both large companies and startups. One notable project was building a ride-hailing service similar to Uber in Europe, where I gained experience managing a diverse engineering team and multitasking. My time in startups gave me a chance to handle not only coding but also customer relations, backlog management, and testing.
Spotify stood out to me for its unique development culture and the challenges that arise from working on products which serve hundreds of millions of users. When I saw an opening, I knew I could both learn and contribute.
The hiring process was quite rigorous—it typically takes one or two months and includes calls with recruiters and up to five interviews testing technical skills, design knowledge, and cultural fit. Spotify places significant importance on ensuring new hires align with the company's values and collaborative spirit.
You've been part of several exciting projects at Spotify. Let's start with Band Manager, which you helped architect and launch. How did it impact team management and workflows?
Band Manager is a tool that simplifies access to resources within Spotify. Previously, employees had to submit IT requests and wait hours for approval. We realized automating this process could significantly speed up business operations, and with Band Manager, employees now gain access within seconds. It's also fully transparent, allowing the security team to monitor access.
Another key project was improving organizational transparency. Spotify has over 8,000 employees, and it was difficult to keep track of who did what. Workday provided limited information, showing data only for individual employees without revealing team structures. To solve this, we built an internal site that visualizes the company's structure. It became one of the most popular internal resources, helping employees navigate the organization and reach the right person with requests.
How important is collaboration at Spotify, especially between engineers, designers, and data specialists? Is there something unique about the company's culture?
Collaboration is the cornerstone of all work at Spotify. Our teams are cross-functional, meaning each team has all the necessary members to complete a project, including product managers, designers, and data scientists. This autonomy allows teams to work more efficiently and independently, launching new services faster.
Another key part of Spotify's culture is innovation. Every year, we hold a company-wide hackathon where employees focus solely on developing new ideas. It's a chance to experiment without distractions and to bring fresh solutions to life.
You contributed significantly to the Cost Engineering Ecosystem, which helps Spotify optimize cloud service costs. Could you elaborate on the technologies and approaches you used?
When I joined, the platform lacked visibility into cloud spending. Engineers couldn't easily track how much they were spending on services, and reports had to be requested separately. I redesigned the platform to include dynamic dashboards with intuitive charts, filters, and search options, allowing developers to easily monitor and reduce costs.
We also built complex pipelines to merge data from multiple sources into one dataset accessible to all employees. We use SQL for querying data and break down complex requests using DBT technology for better performance.
Backstage, another project you contributed to, is now used by over a thousand companies. How has this open-source project impacted the global IT community and Spotify itself?
Backstage is a developer portal framework that helps engineers manage internal services. It serves as a centralized site where all the necessary information, like service documentation and CI/CD pipelines, is available. This is crucial for organizing development in such large companies, ensuring everyone operates from a single, reliable source.
Initially, we built it for internal use, but after open-sourcing it, many companies showed interest in using Backstage to structure their development processes. This move sparked an entire ecosystem—there are now entire startups dedicated to building internal portals.
For Spotify, Backstage has become one of our most recognizable projects, enhancing our brand among developers. It also helps us attract top talent, who often join not just for the music services but for the opportunity to work on cutting-edge developer tools.
I personally contributed to the Cost Insights plugin, which helps companies analyze the operational costs of their services. It's a valuable tool for understanding when a service's resource consumption increases and identifying the causes of this.
What was your biggest technical challenge at Spotify, and how did you overcome it?
One of the most challenging aspects has been working with massive amounts of data. Spotify serves hundreds of millions of users, generating terabytes of data that need constant analysis. We use BigQuery for this, but it's expensive, and visualizing the data clearly is always a challenge.
For example, in the Cost Engineering Ecosystem project, we had to figure out how to display cost breakdowns across hundreds of teams. The initial graphs were too complex and unreadable to serve this purpose. To solve this, I created a system that displays only the top 100 domains while aggregating the rest into a single line. This allowed users to see both detailed and high-level cost breakdowns, with the option to explore further as needed.
Finally, how do you stay professionally competent, and what resources would you recommend for self-education?
Staying current is critical in IT since the field is always evolving. I constantly read tech blogs and watch online or in-person conferences to keep up with the latest trends and technologies.
I recommend Hacker News as a great portal for tech research and news. I also watch frontend and backend development conferences and keep up with advancements in Java.
In addition, Spotify encourages ongoing learning with internal courses taught by experienced developers. For instance, I took an in-depth course on machine learning, and I've also personally organized courses on front-end development.
Beyond formal learning, daily interactions with colleagues are incredibly valuable. We embrace what we call "T-Shapeness," meaning each person is an expert in one area but has foundational knowledge in several others. This fosters collaboration and knowledge-sharing across teams.
After work or on weekends, I experiment with new technologies and try to apply them in practice. When we start new projects, we always consider whether newer tools could make our work faster, more flexible, or cost-effective.
