Technical Challenges for Simulation and Training in Megacities

Jon Watkins and Chuck Campbell

Introduction

Megacities, urban areas with populations over 10 million people, are of growing importance to the military, and thus are of growing importance to training.  Training needs vary widely based upon maneuver or fighting in open desert, urban environments on the streets, and building interiors (room clearing).  Training applications span aggregate commander and staff constructive trainers to individual combatant virtual simulation. It is critical for Modeling and Simulation (M&S) applications to represent those environments and situations which are inherently unusual or difficult to train live.  An example of this is urban areas, which are difficult to recreate realistically for training.  It is impractical to impossible to fully recreate megacities (like we do with MOUT sites) or to get access to real-world cities for live training.  As a result, simulation becomes critical for both training and analysis use cases (i.e. analysis to assess if a particular weapons system or offensive plan might work better than another).

Figure 1: (L) Metro entrance in Canada, showing an example of the layers of features and activity at a single location; (R) Underground Mall in Taipei, showing the importance of advanced underground modeling.

As the threats to the U.S. change, megacities become increasingly relevant and important for preparing for the future.  Violent extremist groups regularly innovate in their approaches to terrorism, creating new threats to U.S. troops.  They operate out of a wide variety of locales around the world, to include large urban areas and megacities.  In addition, a conventional war with a major power like China, North Korea, or Russia would inevitably involve urban environments.  Modeling and simulation must be able to represent these megacities so that U.S. forces can more efficiently prepare to combat threats arising within the megacities of our world. Accurately representing megacities and dense urban environments provide particular challenges to the simulation and training community.  This paper discusses those challenges from a Synthetic Natural Environment (SNE) standpoint.

Within modeling and simulation systems, higher level models operate on, around, and within the synthetic natural environment.  The SNE is the foundation of M&S capabilities, and must be designed and developed to meet new and far-reaching requirements.  If the SNE representation lacks a critical data element required for a model to function, then training may be impeded.  For example, if the material characteristics of walls are critical for calculating damage, building collapse, mouseholes, or weapon penetration, then the SNE must contain that data or a reasonable approximation of it.   Current SNE capabilities can be applied to the challenges with representing megacities in limited ways, but must advance exponentially to meet 2025+ challenges.

The two major challenge focus areas presented here are data representation and the environment model.  While data collection and processing is a major concern for simulation of megacities, we do not focus on these issues here, since collection and processing is also an issue for operational situations as well, and thus is not unique to simulation and training.  We do, however, reference collection and processing where we see M&S-specific issues.

Background

The authors have practical, real-world experience with SNE as it is designed, produced, and used in M&S today, and we are familiar with both strengths and weaknesses of the current state of the art.  We have a long standing history of pushing SNE technology towards the future, having contributed to the design and development of the SNEs on many of the Army’s simulation and training systems over the past 20 years, including WARSIM, CCTT, OneSAF, SE Core, and others.  The authors are currently leading research efforts for the M&S community looking at complex urban environments, including megacities in general and underground representations in particular.  This work is being conducted primarily for the U.S. Army Research, Development and Engineering Command (RDECOM) Army Research Lab (ARL) Human Research and Engineering Directorate (HRED) Advanced Training & Simulation Division (ATSD).

While the overarching scope of our work is to investigate the future direction of SNE for simulation in general, we are looking at megacities as a key area requiring improvement.  This paper summarizes the high level concerns with megacity representation as we see it (relative to today’s state of the art in fielded M&S systems).

Challenges

Inevitably, SNE requirements for the future must be driven by functional needs, since ultimately any SNE implementation exists to provide the context and surrounding environment for a larger system.  For example, one training use case may consider it critical to represent power outages if power lines are damaged while others may consider this irrelevant.  However, we can still extract several larger themes of concern with accurate portrayal of megacities, given the inevitable desire for increased fidelity and realism.

Data Representation

A new approach is needed for data representation.  Current data representation approaches do not scale to the extent needed for the ultra-dense urban environment, as is seen with a megacity.  Megacities will naturally require a wider breadth and higher density of data than is currently used in M&S applications.  In addition, to keep up with the increasing availability of new data and notifications of changes to existing data, mechanisms are needed to quickly update and modify data.  Technology advancements enable rapid, continuous bursts of new information from the field, which should be integrated into the data to improve the representation of the environment within simulation.

Several challenges arise when considering megacities.

Increased density.  In terms of the environment, this means an increase in non-specific environment features such as clutter; items like benches, trash, signs, etc.  Buildings will be large, close together, and will often require both interior and sub-subterranean representation.  Transportation networks will be increasingly complex, requiring representation of roadways, railways, water ways, and subways, many of these transitioning frequently from above ground to below ground.

Figure 2: (L) A simple roadway interchange; basic traffic laws are obeyed by simulated vehicles; (R) A complex highway interchange in Boston, MA; will require advanced representation and models (Google Maps).

Increased flexibility.  Megacities have many characteristics in common, but training in one city is not directly applicable to training in another city.  Simulations must be able to represent the key characteristics of various megacities so that training can occur in geo-specific locations around the globe.  This diversity requires increased flexibility in how SNE data is represented. Every mission brings new and unfamiliar challenges. Just as the military must adapt quickly to overcome adversity, training simulation must respond accordingly to remain effective. Megacity representation requires a fluid SNE specification able to rapidly change to provide training for future, often unpredictable threats. 

Vast relational challenge.  Increases in urban density compound the complexity of relationships between infrastructure components.  Critical infrastructure such as transportation networks become progressively complex as the interactions between environment features (roads, tunnels, railways, traffic signals, etc.) and moving actors (vehicles, pedestrians, etc.) become more frequent, with a greater number of variables and possible outcomes.  Other infrastructure representations are in their infancy, such as computer and cellular networks, and challenges must be overcome to accurately represent these as well.

Streaming information at appropriate levels.  Given the ubiquity of sensors and sensing platforms, live, up to the minute data is often available, and should be leveraged to maintain the environment representation accurately.  This challenge has not yet been tackled in the modeling and simulation community to a level that is applicable for megacities. Data filtering becomes increasingly important with the rise of augmented reality systems.

Multiresolution representation. Megacity training objectives will be wide and varied. Some applications will need very detailed information while others will only need an abstract representation. For example, one trainer may need to simulate disabled power to a section of the city, whereas another may need to disable the power to a particular building. The effects of urban modeling should correlate at every level.  In the power disruption example, the power grid is represented by power being distributed from power plants, across lines, through substations, transformers and other equipment.  While this example focuses on electricity, the concepts apply to many SNE representational areas, including transportation/supply lines (including shipping and air), potable water, structural materials, communication, medical facilities, shelters, weather, and more. The SNE must be equipped to be able to manage the timely provision of high-resolution data in specific areas when it is required.

Environment Model

Megacities are inherently intricate in terms of how people interact with them, including complex effects on entities, secondary effects (such as damage to power lines causing lights to go off), implied or expected behaviors (such as traffic lanes or areas where civilians are likely to congregate), and multi-dimensional context (such as airport versus hospital/school versus police station).  At least some of these complex effects must be captured in simulation in order for training to be effective and realistic.

Improved fidelity.  In order to accurately portray the increasing complexity discussed above in relation to infrastructure, flexibility, and increasingly available data, environment models need to be more advanced, more capable, and of a higher fidelity.  For example, models must take into account the difference between types of clutter.  Clutter may create obstacles and change line of sight, but higher fidelity models should distinguish between a movable, non-blocking obstacle (trash bag) and a stationary, blocking obstacle (dumpster).  As another example, advanced vehicle models will be able to use road lane and traffic signal data, and obey basic traffic laws of the city in which they are operating.

Cascading effects.  Many environment models in M&S today effectively simulate the single function for which they are designed. As interactions between various models within the simulation increase, so do possible outcomes, issues, effects, etc.  Infrastructure failure such as a water main break is one example of a cascading effect in an urban environment. Often what follows are flooded streets and subways, localized loss of electricity, increased vehicle traffic, etc. In addition, political, economic, and social models will need to be incorporated in order to support training for combating terrorist activities.

Weather realism.  While weather is not specific to megacities, Army simulations currently have minimal representation and modeling of weather.  Weather affects real world operations, and similarly should affect training, but it has not yet been a focus of Army simulations. Two critical components are required to achieve weather realism in modeling and simulation: accurate and complete weather models, and usage of the resulting weather data by other models in the simulation.  Accurate weather models and weather data are available, although not often leveraged in M&S, and other models rarely consider weather conditions.  Weather affects much of what happens in the real world (fog, haze, smog, precipitation, etc. all effect movement, visibility, etc.), and weather realism in simulation is a complex challenge. Localized weather effects can induce the need for immediate deployment of first responders to help those in need and quarantine unsafe areas.  Considerations for weather in megacities are often unique, and may require additional data and models (e.g. the uncharacteristically high amount of snow in Boston, MA in 2015 presented new challenges with snow removal - there was nowhere to dispose of the excess snow from plowed roadways).

Figure 3: Rain is accumulating in the dug hole, and changing the composition of the soil.  Dignitas added weather modeling and effects to a Construction Equipment Virtual Trainer (CEVT).

Current State of the Art

While the simulation and training community has made great strides in some areas related to dense urban environments, the community lacks the current capability to represent megacities at a sufficient level.

Current SNE capabilities exist in some key areas, which can be applied and leveraged to represent dense urban environments.  For instance, high fidelity representation of building interiors has been achieved and applied in simulation training for tasks such as room clearing or attacks within a building.  In addition, advances have been seen in visualization technologies, where visualizing urban and high density environments is no longer a struggle, and specialized, expensive hardware is no longer a necessity.  Computers have gotten faster, smaller, and more capable over the years, while visualization algorithms have improved, making it possible to visualize complex 3D scenes on inexpensive hardware.  Representation of high density features has been achieved in support of computer generated forces as well.  Simulation systems such as OneSAF, Close Combat Tactical Trainer (CCTT), and JCATS offer support for limited urban areas and building interiors.  Although some strides have been made, many key elements of urban environments are notably missing from modern simulation systems. 

The authors see the modeling and simulation community as largely unprepared to accurately represent the physical environment in a simulation context.  Innovation to address this challenge must come through targeted research to provide a foundation on which programs of record can base future requirements.  Visual systems required for virtual simulation are generally in a better position to represent complex urban environments, although much work remains in the detailed geometry, topology, and physical modeling of the environment required to provide supporting services.

Figure 4: The Dignitas Veritas 2D/3D Viewer is used by LVC-IA and SE Core and runs on inexpensive laptops or desktop computers, with no specialized hardware requirements.

Below we discuss a selection of the issues that exist with the current state of the art.

Specialized Attribution Requirements.  The modern battlefield is characterized by unparalleled access to data from a wide range of sensors.  Open environments, such as those seen from air or space or easily observed from the ground, can be visualized and sensed in real-time.  However, obtaining realistic and geospecific data for megacities is more complicated.  Techniques that scan environments can be hindered by physical blockages (such as visibility into buildings or underground), as well as lack the ability to capture the intricacies of the man-made environment (such as such as material characteristics, access points, mobility information, etc.).  Accurate simulation of a megacity synthetic environment will require access to these types of information or the ability to realistically infer or derive such data. These content requirements, of course, will be driven by training and functional requirements.  If a training use case requires accurate representation of underground fire or smoke propagation, then issues like airflow, venting, etc., become important.  If structural collapse or simulation of mouseholes is important, then structural attribution becomes more important.

Leveraging Commercial Data and Technology.  In times of limited budgets, it is critical that the Army adapt to development of systems and technology that can more readily leverage commercial solutions.  Current Army M&S solutions are stove-piped, built through specialized development with only limited reuse of open and commercial solutions.  While specialized development may remain necessary in many areas, reuse can be increased.

Traffic and Crowd Modeling.  The density and variety of human activity is a critical component in urban operations. Even in wartime situations, there may be civilians hiding out in cities, fleeing the hot zones, or congregating in the city’s outskirts.  When there is little warning of impending conflict, civilians may be the critical factor in operations.  Military operations may need to be carried out with regular civilian activities occurring, as in Baghdad.  Simulation of civilians requires specialized data in SNE, including things like traffic information (lanes, stop lights), designation of areas of heavy traffic (e.g. transportation hubs or malls would have more civilian traffic than a warehouse), etc. Public gathering locations such as stadiums, parks, hospitals, and churches can serve as terrorist targets or opportunities for civil unrest.

Underground and Interiors.  While the Army has extensive recent experience in urban environments and building clearing, regional differences can make a tremendous difference.  Clearing a high rise building requires different approaches than clearing a small building in a walled compound.  Similarly, underground environments represent special risks due to factors such as collapse, access, and even air flow (for fires or smoke propagation).

Civilian Infrastructure.  Megacities will have complex and widely varying types of civilian infrastructure including power networks, lights, water lines, specialized buildings (hospital, power plant), etc.  This infrastructure can matter in physical ways (damage to gas pipes could be physically dangerous, knocking out power will disable lights) as well as in conceptual ways (approaches will vary for a hospital versus school versus police station). Large, open areas such as stadiums, playgrounds, and parking lots can have military significance in providing locations for displaced civilians, holding areas for detainees, or logistics support.

Physics Effects and Interactions.  Complex physical interactions will be critical in representing urban environments, including structural collapse, weapon penetration of walls, cratering of roads, collapse of tunnels, etc.  In more open, rural environments, these effects could often be minimized or avoided except at key chokepoints.  Megacities are large aggregations of critical chokepoints.

Urban Clutter.  In most Army simulations, the training environment is largely sterile and lacks dense clutter like dumpsters, parked cars, street side market stalls, utility poles, etc.  Urban clutter is especially important for high-fidelity training where objects can reduce visibility, provide cover and concealment, and hamper movement.

Figure 5: Complex civilian infrastructure must be modeled and considered in megacity operations.

Emerging Technology

Keeping an eye on emerging technologies across the industry ensures re-use candidates can be identified and used as appropriate. 

There is an emerging trend of crowd-sourced data and increased timeliness of geospatial and infrastructure data available from multiple sources.  For example, navigational data on smart phones and GPS systems is updated frequently with up to date traffic data to indicate whether traffic is flowing smoothly or is backed up.  Some navigational apps also include crowd sourced data, allowing users to input accidents, road hazards, and detours.  However, such data sources must be carefully considered since they could easily provide a pathway for misinformation to enter into military systems. Critical areas and categories of data will remain difficult to obtain trust-worthy sources for, especially in foreign countries or network denied areas.

Figure 6: Google maps and GPS systems show live traffic status via colored lines (red for heavy traffic, green for clear) and icons for construction zones and accidents.

Game engines continue to evolve, but their focus is often in areas that are not important in military applications.  Game engines often excel at providing highly realistic visual environments and compelling physical interactions, but the focus is generally on visual effect and gameplay rather than realism. Realistic effects are of utmost importance to military simulations to avoid negative training.  Similarly, realistic-looking effects in game engines are often achieved through extensive manual effort from artists and scenario developers.  This approach is cost-prohibitive if representing large areas or when rapid turnaround is needed from source to training.

Figure 7: Realistic video game visuals require extensive manual designs and illustrations.

Cloud-based, thin-client solutions are a growing trend that could help facilitate training at point of need. As a download repository, cloud technology has no impact on SNE representation. But the notion of streaming SNE database information adds an additional layer of complexity to the solutions needed for megacity representation. To help combat delays caused by network latency, a number of techniques are available, including optimized data structures, compression, composition (configure what data is needed per application), predictive caching, and procedural generation of SNE features. Investigation and experimentation is needed to determine which technique or combination of techniques deliver best “bang for the buck” results.

Future Focus of Science and Technology Investments

We believe concentrated investment in government development is needed to meet military needs.  Crowd-source and commercial technology is widely available, but doesn’t fully meet military needs and is susceptible to injection of misinformation.  We also need pathways to connect simulation and training applications over to classified data or networks, and operational data to enhance training and realism.

The authors’ current research efforts are investigating methods to represent geometry and topology in separate ways, yet providing some assurance of correlation between them.  Megacities will require complex geometries which will make it extremely difficult to automatically derive key behavioral aspects needed for automated opponents, such as logical firing and over watch positions.  Current Army simulation systems have structured yet comparatively simplistic SNE data to reason on, often making broad assumptions (such as not worrying about overhead clearances).  In megacities, these assumptions will not hold up.  Our research will also consider how to use a mixture of automation, machine learning, and very targeted human in the loop attribution (for heavily used training areas) to support behavioral aspects.

Defining what types of data are needed for megacity simulation is critical for success. This will include not just obvious information…such as locations of buildings, bridges, and roads…but potentially far more complex data such as material characteristics, support structures (load bearing walls for collapse), topological information (turn lanes, stop lights, paths where civilians often travel on foot), and more.  Since elements of this data will be difficult to collect from standard sources (e.g. interiors and below ground geometry), it will be important for such information to be derivable in geotypical, realistic ways. Our research will focus on what is needed for runtime functional needs rather than considering methods for collection.

Megacity simulation will require different data and services for various training use cases.  For example, an immersive simulation for individual soldiers will require a different type of detail and information than a strategic bombing simulation.  However, to ensure communications and interoperability between simulations, these varying representations would ideally be derived from and relate back to a common overarching representation which can handle multiple resolutions or levels of detail. Our research will investigate techniques for how to represent varying levels of detail in a correlated and realistic fashion, while also supporting highly selective data use wherein a given simulation system only loads data that is relevant and useful to its use case.

The authors believe it important that future simulations support just in time data delivery and flexible paradigms for computing platforms and environments.  A simulation run one day in the cloud, might be run on network denied devices the next.  In addition, we expect frequent updates to geospatial, topological, and human behavioral data from a wide range of sources in the future. This concept requires the ability to distribute the latest data rapidly, consistently, and selectively so as to get maximum benefit out of limited bandwidth for edge devices.  Just in time data processing and delivery will enable different data use cases, such as cloud-based simulation using data centers or highly select data sets for resource- and bandwidth-limited devices.

As data sources, content, and updates increase in complexity, the mechanisms and processes for verifying consistency, relationships, and interdependencies must grow.  Seemingly simple changes in basic geometry for feature data can have complex secondary effects on other data.  For example, if data becomes available for changing the function of a building (from office building to embassy, for example), it can impact secondary data, including crowd modeling flows, behavioral cues, and more.  As we build up content requirements and update mechanisms in early years, the later years of our research will look at ways to reconcile all of this data and maintain consistency where it matters.

Work is needed even in areas where the Army simulation community has extensive previous experience, such as calculation of cover, conceal, or hide locations, basic movement planning, and line of sight.  These algorithms will require a different level of complexity with different tradeoffs when working in complex megacity environments.

Conclusion

As the U.S. military adapts to operating in megacities, it will be critical to enable realistic training through simulation so that warfighters are able to refine and practice their procedures.  This is necessary not just for training and experience, but also to assess effectiveness of new weapons systems and platforms.  The simulation and training community faces a wide range of challenges to enabling this training, especially with regards to realistic representation of dense urban environments.  The authors believe focused investment is needed to ensure the simulation and training community can meet the technical challenges required to enable analysis and training in megacities.  We welcome feedback from and collaboration with other Army agencies and contractors working in this domain.