UrbanSim
Project highlights
About the client
About the client
Paul Waddel is a leading expert in urban analytics, data science, decision support systems, and urban simulation with over 10,000 citations to his published work. Formerly a professor and chair at the University of California, Berkeley, he researched urban microsimulation and created the UrbanSim modeling methodology that received numerous grants from the National Science Foundation. To bring the results of his research to the real world, he created UrbanSim, a SaaS solution for city development, now applied by urban planning organizations and agencies in various US cities like Detroit, Houston, Phoenix, Salt Lake City, San Francisco, and Seattle, as well as across Europe, Asia, and Africa.
Powered by statistical models, UrbanSim allows construction workers and urban analysts to efficiently plan urban development and infrastructure projects, from building households to locating public services. Using open-source statistical data, the platform calculates the consequences of urban changes, letting city municipalities create better life conditions for their citizens – longer life with a greener environment, better access to shops and services, fewer traffic jams.
The UrbanSim solution has gained popularity but over time Paul Waddel found it hard to scale the platform further. They decided to strengthen their team with senior software engineers who could provide a fresh perspective on their product. That’s why they addressed Flyaps.
Read on to learn how we helped UrbanSim move from an outdated legacy system to a scalable solution with microservices architecture and ML-ready infrastructure.
CLIENT REQUEST
UrbanSim needed to augment their team with senior software developers to rebuild their existing urban planning platform. They reached out to Flyaps for engineers with experience in machine learning and statistical modeling.
Paul Waddel and his team wanted to develop UrbanSim further, enhancing it with faster statistical modeling and AI algorithms but the outdated platform didn’t allow for scaling.
Building new features took too much time, recurring bugs continued to appear and each change impacted the entire system, augmenting technical debt and making the development process slow and costly.
So the client decided to rebuild the platform from scratch.
We already had experience rebuilding complex legacy platforms so we shared our insights for the new system architecture and provided top backend, frontend, DevOps, and ML engineers who could rebuild the existing product.
problem
To move from a legacy to a new scalable platform, we had to solve several problems:
To tackle technical debt, replace outdated technology, and reduce high maintenance costs, UrbanSim needed a strong team of senior developers.
We not only rebuilt the system using modern technology but also enhanced software development efficiency by establishing a robust DevOps pipeline and preparing the platform for the incorporation and maintenance of AI-based features.
approach
To help our client make UrbanSim a future-proof solution, we had to rebuild the platform from scratch, migrating from monolith to microservices, and preparing the infrastructure for MLOps.
Rebuilding every module of a complex solution like UrbanSim demanded thorough preparation. We started by analyzing the system to define the roadmap with the stakeholders.
The platform's primary innovation lay in its differentiable statistical models, a groundbreaking achievement by UrbanSim that enabled urbanists to compute scenarios within minutes instead of hours.
In the revamped system, our task was to integrate this functionality at the code level. Additionally, we introduced new features, including zoning for predicting outcomes in specific areas and improved visualization to enable users to compare various scenarios on a single screen.
solution
To help UrbanSim fully implement their vision without legacy limitations, we chose a future-proof tech stack and built a new product.
Developing such a complex system from scratch wasn’t easy but this decision allowed us to achieve significant improvements:
In tandem with UrbanSim, we delivered a breakthrough urban planning solution. Here’s a short overview of the key technical challenges and how we solved them.
The original solution was constructed using a monolithic architecture, which affected system flexibility and made scalability difficult. To address this issue, we chose a microservices architecture, breaking down the platform into modules to enable independent scaling of each component. Here are the core modules we've built:
To reduce dependencies among services, we adopted an event-driven architecture style employing a publish-subscribe model. Also known as a message-driven architecture, this style facilitates the real-time management of substantial data volumes with minimal time lag. The pub-sub model enables the exchange of messages without direct point-to-point connections, facilitating the real-time processing of diverse events throughout the entire ecosystem.
All microservices operate autonomously within a Kubernetes cluster which enables the deployment and scaling of containers independently. As a result, development teams can work independently on specific services, streamlining the development process and facilitating easy scaling of resources.
UrbanSim relies on differentiable statistical models to create urban scenarios. These predictive models involve just-in-time compilation into machine code and are trained on historical datasets, applying statistical analysis for accurate predictions.
While each new scenario previously required a considerable amount of time for calculation, the UrbanSim team achieved a breakthrough in statistical models, reducing the scenario calculation time from hours to mere minutes.
To incorporate these advancements, we optimized the code and seamlessly integrated the models into the system.
The performance challenge was the next one to tackle. In an application with numerous detailed maps, each with several layers, it was essential to optimize performance and improve the application speed as much as possible.
To speed up UI loading times, we leveraged GIS (Geographic Information System), a technology that uses maps for analyzing and interpreting spatial data. It improves performance through techniques such as spatial indexing, caching, and optimized rendering, ensuring quick retrieval and analysis of geographic information.
Addressing excessive expenses related to cloud services was another issue we had to solve.
Google Kubernetes Engine (GKE) allows two options – Standard mode and Autopilot. In the Standard mode, you need to manually configure and manage resources. Autopilot, on the other hand, automates resource provisioning based on application needs and dynamically adjusts resources, preventing over-provisioning.
The UrbanSim platform was initially hosted on a standard cluster, making manual resource control challenging and resulting in disproportionately expensive simulation creation. Apart from that, the cloud infrastructure wasn't optimized for cost-effectiveness.
To address these issues and reduce cloud costs, we implemented the following measures:
As a result, we successfully reduced cloud costs by 40%, resulting in substantial monthly savings amounting to thousands of dollars. The new product proved to be 10 times more cost-effective to host compared to the legacy system, thanks to the optimized infrastructure.
Finally, we had to streamline the development processes. The system struggled because of slow development time and inefficient processes, primarily stemming from the failure to adopt the latest DevOps practices.
To fix it, we introduced an automated DevOps workflow, integrating Terraform, an infrastructure-as-code software tool, and continuous integration and deployment (CI/CD) practices. We also optimized the use of Kubernetes, an open-source platform for the deployment, scaling, and management of containerized applications. With it, we managed to efficiently distribute workloads across multiple servers, ensuring seamless operation and easy scalability.
The UrbanSim platform manages multiple model families and versions. It requires daily model training, leading the team to actively explore model tuning for improved prediction quality. The training of statistical models demands a substantial amount of computational power. But in the majority of scenarios, opting for GPU over CPU for training can result in acceleration speeds that are 6 to 10 times faster, thanks to the benefits of parallelization. GPUs are equipped with thousands of cores, while CPUs' cores typically range from two to 64.
To achieve faster and more efficient statistical model training, we set up a system that allows the use of a GPU within the Kubernetes cluster on demand. By switching to GPU for training, we drastically reduced the calculation time, while the cost largely remained unchanged. Plus, thanks to Kubernetes, the training process became fully automated, so any regular user can kick it off from the UI.
By closely collaborating with UrbanSim, we delivered a complex solution for urban planning built on microservices architecture, cloud technology, and a modern tech stack.
The UrbanSim team can now not only incorporate new functionality without any problems but also scale their development processes effectively. This includes establishing development streams for independent work and tailoring the product to meet the unique needs of various clients.
Scalable and customizable, the platform opened limitless growth possibilities for UrbanSim and already hosts three of their clients with others joining soon.
We continue our partnership, further evolving the product.
Result
City municipalities in San Francisco, El Paso, and Nashville are already using the new platform, and more clients from the US, Canada, and Australia are about to adopt it soon.
The system can be scaled and customized for clients in diverse locations through the use of independent development streams. It's also highly extendable and ready for AI and ML.
Maintenance and cloud costs have been reduced by almost half through correct DevOps implementation and optimal resource allocation, allowing further scaling with no tech debt.
Flyaps is a senior software development team. We provide top-notch engineers and DevOps experts to help our clients create exceptional software products. If you're interested in collaboration, please reach out to us.
Let’s discuss your project
